Module for remote in-memory Python package/module loading through HTTP/S

Overview

httpimport

Python's missing feature!

The feature has been suggested in Python Mailing List

Remote, in-memory Python package/module importing through HTTP/S

PyPI - Python Version PyPI version Build Status Coverage Badge

CPython 2.7 CPython 3.4 CPython 3.7 Pypy 2.7 Pypy 3.6 Jython 2.7.1

A feature that Python2/3 misses and has become popular in other languages is the remote loading of packages/modules.

httpimport lets Python2/3 packages and modules to be imported directly in Python interpreter's process memory, through remote URIs, and more...

Examples

Load a simple package/module through HTTP/S

>>> with httpimport.remote_repo(['package1','package2','package3'], 'http://my-codes.example.com/python_packages'):
... 	import package1
...

Load directly from a GitHub/BitBucket/GitLab repo

  • Load a python file from a github-gist (using this gist):
import httpimport

url = "https://gist.githubusercontent.com/operatorequals/ee5049677e7bbc97af2941d1d3f04ace/raw/e55fa867d3fb350f70b2897bb415f410027dd7e4"
with httpimport.remote_repo(["hello"], url):
    import hello
hello.hello()
>>> with httpimport.github_repo('operatorequals', 'covertutils', branch = 'master'):
...     import covertutils
... # Also works with 'bitbucket_repo' and 'gitlab_repo'

Load a package/module from HTTP/S directory directly to a variable

>>> module_object = httpimport.load('package1', 'http://my-codes.example.com/python_packages')
>>> module_object
<module 'package1' from 'http://my-codes.example.com/python_packages/package1/__init__.py'>

Load a package/module that depends on other packages/modules in different HTTP/S directories

>>> # A depends on B and B depends on C (A, B, C : Python modules/packages in different domains):
>>> # A exists in "repo_a.my-codes.example.com" |
>>> # B exists in "repo_b.my-codes.example.com" | <-- Different domains
>>> # C exists in "repo_c.my-codes.example.com" |
>>> with httpimport.remote_repo(['C'], 'http://repo_c.my-codes.example.com/python_packages'):
...  with httpimport.remote_repo(['B'], 'http://repo_b.my-codes.example.com/python_packages'):
...   with httpimport.remote_repo(['A'], 'http://repo_a.my-codes.example.com/python_packages'):
...   import A
... # Asks for A, Searches for B, Asks for B, Searches for C, Asks for C --> Resolves --> Imports A
>>>

Load Python packages from archives served through HTTP/S

>>> # with httpimport.remote_repo(['test_package'], 'http://example.com/packages.tar'):
>>> # with httpimport.remote_repo(['test_package'], 'http://example.com/packages.tar.bz2'):
>>> # with httpimport.remote_repo(['test_package'], 'http://example.com/packages.tar.gz'):
>>> # with httpimport.remote_repo(['test_package'], 'http://example.com/packages.tar.xz'): <-- Python3 Only
>>> with httpimport.remote_repo(['test_package'], 'http://example.com/packages.zip'):
... 	import test_package
...
>>>

Serving a package through HTTP/S

$ ls -lR
test_web_directory/:                                                         
total 16                                                                     
drwxrwxr-x. 4 user user 4096 Sep  9 20:54 test_package                       
[...]                  
                                                                             
test_web_directory/test_package:                                             
total 20                                                                     
drwxrwxr-x. 2 user user 4096 Sep  9 20:54 a                                  
drwxrwxr-x. 2 user user 4096 Sep  9 20:54 b                                  
-rw-rw-r--. 1 user user   33 Sep  9 20:54 __init__.py                        
-rw-rw-r--. 1 user user  160 Sep  9 20:54 module1.py                         
-rw-rw-r--. 1 user user  160 Sep  9 20:54 module2.py                         
                                                                             
test_web_directory/test_package/a:                                           
total 4                                                                      
-rw-rw-r--. 1 user user  0 Sep  9 20:54 __init__.py                          
-rw-rw-r--. 1 user user 41 Sep  9 20:54 mod.py                               
                                                                             
test_web_directory/test_package/b:                                           
total 4
-rw-rw-r--. 1 user user  0 Sep  9 20:54 __init__.py
-rw-rw-r--. 1 user user 41 Sep  9 20:54 mod.py

$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...

Usage

Importing Remotely

add_remote_repo() and remove_remote_repo()

These 2 functions will add and remove to the default sys.meta_path custom HttpImporter objects, given the URL they will look for packages/modules and a list of packages/modules its one can serve.

>>> import test_package### Contexts

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named test_package
>>>
>>> from httpimport import add_remote_repo, remove_remote_repo
>>> # In the given URL the 'test_package/' is available
>>> add_remote_repo(['test_package'], 'http://localhost:8000/') #  
>>> import test_package
>>>
>>> remove_remote_repo('http://localhost:8000/')
>>> import test_package.module1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named module1

The load() function (as of 0.5.10)

The load() function was added to make module loading possible without Namespace pollution. It is used to programmatically load a module in a variable, and call its objects directly from that variable.

>>> import httpimport
>>> pack1 = httpimport.load('test_package','http://localhost:8000/')
>>> pack1
<module 'test_package' from 'http://localhost:8000//test_package/__init__.py'>
>>>

Contexts

The remote_repo() context

Adding and removing remote repos can be a pain, especially if there are packages that are available in more than one repos. So the with keyword does the trick again:

>>> from httpimport import remote_repo
>>>
>>> with remote_repo(['test_package'], 'http://localhost:8000/') :
...     from test_package import module1
...
>>>
>>> from test_package import module2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name module2

>>> module1.dummy_str
'Constant Loaded'
>>> module1.dummy_func
<function dummy_func at 0x7f7a8a170410>

The Github Use Case!

The dedicated github_repo() context:
>>> from httpimport import github_repo
>>> with github_repo( 'operatorequals', 'covertutils', ) :
...     import covertutils
...
>>> covertutils.__author__
'John Torakis - operatorequals'
>>>
What about branches?
>>> from httpimport import github_repo
>>> with github_repo( 'operatorequals', 'covertutils', branch='py3_compatibility' ) :
...     import covertutils
...
>>> covertutils.__author__
'John Torakis - operatorequals'
>>>
And ad-hoc commits too?

What if you need to stick to a fixed -known to work- commit?

>>> from httpimport import github_repo
>>> with github_repo( 'operatorequals', 'covertutils', commit='cf3f78c77c437edf2c291bd5b4ed27e0a93e6a77' ) :
...     import covertutils
...
>>> covertutils.__author__
'John Torakis - operatorequals'
>>>

The newer sibling bitbucket_repo() (as of 0.5.9)

>>> with bitbucket_repo('atlassian', 'python-bitbucket', module='pybitbucket'):
...     import pybitbucket
...
>>>

Another sibling gitlab_repo() (as of 0.5.17)

>>> with gitlab_repo('harinathreddyk', 'python-gitlab', module='gitlab'):
...     from gitlab import const
...
>>>
The domain parameter for gitlab_repo()

You can point to your own installation of GitLab by using the domain parameter:

>>> with gitlab_repo('self', 'myproject', module='test_package', domain='127.0.0.1:8080'):
...     import test_package
...
>>>

This covers the posibility of using httpimport to target local development environments, which is a strong use case for httpimport.

Import remote (encrypted) ZIP files (as of 0.5.18)

After version 0.5.18 the add_remote_repo and the load functions, as well as the remote_repo context got the zip and zip_pwd parameters. By pointing to a HTTP/S URL containing a ZIP file, it is possible to remotely load modules/packages included in it, without downloading the ZIP file to disk!

>>> with httpimport.remote_repo(
...     ['test_package'], base_url='http://localhost:8000/test_package.zip',
...     ):
...    import test_package
...
>>>

Using a ZIP password (zip_pwd parameter)

>>> with httpimport.remote_repo(
...     ['test_package'], base_url='http://localhost:8000/test_package.enc.zip',
...     zip_pwd=b'[email protected]!'
...     ):
...    import test_package
...
>>>

Life suddenly got simpler for Python module testing!!!

Imagine the breeze of testing Pull Requests and packages that you aren't sure they are worth your download.

Recursive Dependencies

If package A requires module B and A exists in http://example.com/a_repo/, while B exists in http://example.com/b_repo/, then A can be imported using the following technique:

>>> from httpimport import remote_repo
>>> with remote_repo(['B'],"http://example.com/b_repo/") :
...     with remote_repo(['A'],"http://example.com/a_repo/") :
...             import A
... 
[!] 'B' not found in HTTP repository. Moving to next Finder.
>>> 
>>> A
<module 'A' from 'http://example.com/a_repo/A/__init__.py'>
>>> B
<module 'B' from 'http://example.com/a_repo/B.py'>
>>> 

Any combination of packages and modules can be imported this way!

The [!] Warning was emitted by the HttpImporter object created for A, as it couldn't locate B, and passed control to the next Finder object, that happened to be the HttpImporter object created for B!

Debugging...

>>> from httpimport import *
>>>
>>> import logging
>>> logging.getLogger('httpimport').setLevel(logging.DEBUG)
>>>
>>> with github_repo('operatorequals','covertutils') :
...     import covertutils
...
FINDER=================
[!] Searching covertutils
[!] Path is None
[@] Checking if connection is HTTPS secure >
[@] Checking if in declared remote module names >
[@] Checking if built-in >
[@] Checking if it is name repetition >
[*]Module/Package 'covertutils' can be loaded!
LOADER=================
[+] Loading covertutils
[+] Trying to import as package from: 'https://raw.githubusercontent.com/operatorequals/covertutils/master//covertutils/__init__.py'
[+] Importing 'covertutils'
[+] Ready to execute 'covertutils' code
[+] 'covertutils' imported succesfully!
>>>

Beware: Huge Security Implications!

Using the httpimport with HTTP URLs is highly discouraged outside the localhost interface!

As HTTP traffic isn't encrypted and/or integrity checked (unlike HTTPS), it is trivial for a remote attacker to intercept the HTTP responses (via an ARP MiTM probably), and add arbitrary Python code to the downloaded packages/modules. This will directly result in Remote Code Execution to your current user's context! In other words, you get totally F*ed...

Preventing the disaster (setting httpimport.INSECURE flag):

>>> import httpimport
>>>
>>> # Importing from plain HTTP ...
>>> httpimport.load('test_module', 'http://localhost:8000//')
[!] Using non HTTPS URLs ('http://localhost:8000//') can be a security hazard!
[-] 'httpimport.INSECURE' is not set! Aborting...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "httpimport.py", line 302, in load
    raise ImportError("Module '%s' cannot be imported from URL: '%s'" % (module_name, url) )
ImportError: Module 'test_module' cannot be imported from URL: 'http://localhost:8000/'
>>> # ... Throws Error!
>>>
>>> # Importing from plain HTTP has to be DELIBERATELY enabled!
>>> httpimport.INSECURE = True
>>> httpimport.load('test_module', 'http://localhost:8000//')
[!] Using non HTTPS URLs ('http://localhost:8000//') can be a security hazard!
<module 'test_module' from 'http://localhost:8000//test_module.py'>
>>> # Succeeded!

You have been warned! Use HTTPS URLs with httpimport!

Minification

This project has started to suggest stager code for HTTP/S RATs made with covertutils. The Documentation for minifying and using httpimport for such purposes can be found here.

Further minification can be achieved by python-minifier, also available in PyPI. So a minified version can be obtained as follows:

pip install python-minifer    # the "pyminify" command
curl https://raw.githubusercontent.com/operatorequals/httpimport/master/httpimport.py | sed 's#log.*#pass#g' | grep -v "import pass" | pyminify - > httpimport_min.py

size reduction:

# Original Size Count
$ curl https://raw.githubusercontent.com/operatorequals/httpimport/0.7.1/httpimport.py |  wc 
[...]
504    1914   18876
# Minified Size Count
$ curl https://raw.githubusercontent.com/operatorequals/httpimport/0.7.1/httpimport.py | sed 's#log.*#pass#g' | grep -v "import pass" | pyminify - | wc 
[...]
177     936   12141

Contributors

  • ldsink - The RELOAD flag and Bug Fixes
  • lavvy - the load() function
  • superloach - Deprecation of imp module in Python3 in favour of importlib
  • yanliakos - Bug Fix
Owner
John Torakis
It is all about what puzzles we prefer to delve into
John Torakis
Project 2 for Microsoft Azure on WUT

azure-proj2 Project 2 for Microsoft Azure on WUT Table of contents Team Tematyka projektu Architektura Opis rozwiązania Demo dzałania The Team Krzyszt

1 Dec 07, 2021
GitHub saver for stargazers, forks, repos

GitHub backup repositories Save your repos and list of stargazers & list of forks for them. Pure python3 and git with no dependencies to install. GitH

Alexander Kapitanov 23 Aug 21, 2022
Senior Comprehensive Project For Python

Senior Comprehensive Project Author: Grey Hutchinson My project, which I nicknamed “Murmur”, was to create a research tool that would use neural netwo

1 May 29, 2022
Abilian Core: an enterprise application development platform based on the Flask micro-framework, the SQLAlchemy ORM

About Abilian Core is an enterprise application development platform based on the Flask micro-framework, the SQLAlchemy ORM, good intentions and best

Abilian open source projects 47 Apr 14, 2022
Very simple encoding scheme that will encode data as a series of OwOs or UwUs.

OwO Encoder Very simple encoding scheme that will encode data as a series of OwOs or UwUs. The encoder is a simple state machine. Still needs a decode

1 Nov 15, 2021
LAPS module for CrackMapExec

Crackmapexec-LAPS LAPS module for CrackMapExec Make sure to point to the DC Specify the full domain name Be careful the rid 500 might not be "Administ

28 Oct 05, 2022
Syarat.ID Source Code - Syarat.ID is a content aggregator website

Syarat.ID is a content aggregator website that gathering all informations with the specific keyword: "syarat" from the internet.

Syarat.ID 2 Oct 15, 2021
Personal Chat Assistance

Python-Programming Personal Chat Assistance {% import "bootstrap/wtf.html" as wtf %} titleEVT/title script src="https://code.jquery.com/jquery-3.

PRASH_SMVIT 2 Nov 14, 2021
Reso is a low-level circuit design language and simulator, inspired by things like Redstone, Conway's Game of Life, and Wireworld.

Reso Reso is a low-level circuit design language and simulator, inspired by things like Redstone, Conway's Game of Life, and Wireworld. What is Reso?

Lynn 287 Nov 26, 2022
Collections of python projects

nppy, mostly contains projects written in Python. Some projects are very simple while some are a bit lenghty and difficult(for beginners) Requirements

ghanteyyy 75 Dec 20, 2022
Goal: Enable awesome tooling for Bazel users of the C language family.

Hedron's Compile Commands Extractor for Bazel — User Interface What is this project trying to do for me? First, provide Bazel users cross-platform aut

Hedron Vision 290 Dec 26, 2022
It is a Blender Tool which can convert the Object Data Attributes in face corner to the UVs or Vertex Color.

Blender_ObjectDataAttributesConvertTool It is a Blender Tool which can convert the Object Data Attributes in face corner to the UVs or Vertex Color. D

Takeshi Chō 2 Jan 08, 2022
Python Project For Beginner

Basic-Vitrual-AI-Assistant Python Project For Beginner Hey There, I had manipulated Selenium WebDriver to make this assistant. I hope, It will be help

Maruf Billah 13 Dec 12, 2022
A python script that fetches the grades of a student from a WAEC result in pdf format.

About waec-result-analyzer A python script that fetches the grades of a student from a WAEC result in pdf format. Built for federal government college

Oshodi Kolapo 2 Dec 04, 2021
A Python program that generates a maze that solves itself using DFS

Maze Generator And Solver Program Purpose: Generates a maze that then solves itself Language: Python and Pygame Algorithm: Randomized DFS / Floodfill

Joshua Liu 1 Jul 25, 2022
Python Projects is an Open Source to enhance your python skills

Welcome! 👋🏽 Python Project is Open Source to enhance your python skills. You're free to contribute. 🤓 You just need to give us your scripts written

Tristán 6 Nov 28, 2022
Syntax highlighting for yarn.lock and bun.lockb files

Yarn.lock Syntax Highlighting Syntax highlighting for yarn.lock and bun.lockb files Installation Plugin is not publushed yet on Package Control, to in

Alexander Kuznetsov 4 Jul 06, 2022
Script to work around some quirks of the blender obj importer

ObjFix 1.0 (WIP) Script to work around some quirks of the blender obj importer Installation Download this repo In Blender, press "Edit" on the top-bar

Red_3D 4 Nov 20, 2021
Data Structures and Algorithms Python - Practice data structures and algorithms in python with few small projects

Data Structures and Algorithms All the essential resources and template code nee

Hesham 13 Dec 01, 2022