Module for remote in-memory Python package/module loading through HTTP/S

Overview

httpimport

Python's missing feature!

The feature has been suggested in Python Mailing List

Remote, in-memory Python package/module importing through HTTP/S

PyPI - Python Version PyPI version Build Status Coverage Badge

CPython 2.7 CPython 3.4 CPython 3.7 Pypy 2.7 Pypy 3.6 Jython 2.7.1

A feature that Python2/3 misses and has become popular in other languages is the remote loading of packages/modules.

httpimport lets Python2/3 packages and modules to be imported directly in Python interpreter's process memory, through remote URIs, and more...

Examples

Load a simple package/module through HTTP/S

>>> with httpimport.remote_repo(['package1','package2','package3'], 'http://my-codes.example.com/python_packages'):
... 	import package1
...

Load directly from a GitHub/BitBucket/GitLab repo

  • Load a python file from a github-gist (using this gist):
import httpimport

url = "https://gist.githubusercontent.com/operatorequals/ee5049677e7bbc97af2941d1d3f04ace/raw/e55fa867d3fb350f70b2897bb415f410027dd7e4"
with httpimport.remote_repo(["hello"], url):
    import hello
hello.hello()
>>> with httpimport.github_repo('operatorequals', 'covertutils', branch = 'master'):
...     import covertutils
... # Also works with 'bitbucket_repo' and 'gitlab_repo'

Load a package/module from HTTP/S directory directly to a variable

>>> module_object = httpimport.load('package1', 'http://my-codes.example.com/python_packages')
>>> module_object
<module 'package1' from 'http://my-codes.example.com/python_packages/package1/__init__.py'>

Load a package/module that depends on other packages/modules in different HTTP/S directories

>>> # A depends on B and B depends on C (A, B, C : Python modules/packages in different domains):
>>> # A exists in "repo_a.my-codes.example.com" |
>>> # B exists in "repo_b.my-codes.example.com" | <-- Different domains
>>> # C exists in "repo_c.my-codes.example.com" |
>>> with httpimport.remote_repo(['C'], 'http://repo_c.my-codes.example.com/python_packages'):
...  with httpimport.remote_repo(['B'], 'http://repo_b.my-codes.example.com/python_packages'):
...   with httpimport.remote_repo(['A'], 'http://repo_a.my-codes.example.com/python_packages'):
...   import A
... # Asks for A, Searches for B, Asks for B, Searches for C, Asks for C --> Resolves --> Imports A
>>>

Load Python packages from archives served through HTTP/S

>>> # with httpimport.remote_repo(['test_package'], 'http://example.com/packages.tar'):
>>> # with httpimport.remote_repo(['test_package'], 'http://example.com/packages.tar.bz2'):
>>> # with httpimport.remote_repo(['test_package'], 'http://example.com/packages.tar.gz'):
>>> # with httpimport.remote_repo(['test_package'], 'http://example.com/packages.tar.xz'): <-- Python3 Only
>>> with httpimport.remote_repo(['test_package'], 'http://example.com/packages.zip'):
... 	import test_package
...
>>>

Serving a package through HTTP/S

$ ls -lR
test_web_directory/:                                                         
total 16                                                                     
drwxrwxr-x. 4 user user 4096 Sep  9 20:54 test_package                       
[...]                  
                                                                             
test_web_directory/test_package:                                             
total 20                                                                     
drwxrwxr-x. 2 user user 4096 Sep  9 20:54 a                                  
drwxrwxr-x. 2 user user 4096 Sep  9 20:54 b                                  
-rw-rw-r--. 1 user user   33 Sep  9 20:54 __init__.py                        
-rw-rw-r--. 1 user user  160 Sep  9 20:54 module1.py                         
-rw-rw-r--. 1 user user  160 Sep  9 20:54 module2.py                         
                                                                             
test_web_directory/test_package/a:                                           
total 4                                                                      
-rw-rw-r--. 1 user user  0 Sep  9 20:54 __init__.py                          
-rw-rw-r--. 1 user user 41 Sep  9 20:54 mod.py                               
                                                                             
test_web_directory/test_package/b:                                           
total 4
-rw-rw-r--. 1 user user  0 Sep  9 20:54 __init__.py
-rw-rw-r--. 1 user user 41 Sep  9 20:54 mod.py

$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...

Usage

Importing Remotely

add_remote_repo() and remove_remote_repo()

These 2 functions will add and remove to the default sys.meta_path custom HttpImporter objects, given the URL they will look for packages/modules and a list of packages/modules its one can serve.

>>> import test_package### Contexts

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named test_package
>>>
>>> from httpimport import add_remote_repo, remove_remote_repo
>>> # In the given URL the 'test_package/' is available
>>> add_remote_repo(['test_package'], 'http://localhost:8000/') #  
>>> import test_package
>>>
>>> remove_remote_repo('http://localhost:8000/')
>>> import test_package.module1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named module1

The load() function (as of 0.5.10)

The load() function was added to make module loading possible without Namespace pollution. It is used to programmatically load a module in a variable, and call its objects directly from that variable.

>>> import httpimport
>>> pack1 = httpimport.load('test_package','http://localhost:8000/')
>>> pack1
<module 'test_package' from 'http://localhost:8000//test_package/__init__.py'>
>>>

Contexts

The remote_repo() context

Adding and removing remote repos can be a pain, especially if there are packages that are available in more than one repos. So the with keyword does the trick again:

>>> from httpimport import remote_repo
>>>
>>> with remote_repo(['test_package'], 'http://localhost:8000/') :
...     from test_package import module1
...
>>>
>>> from test_package import module2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name module2

>>> module1.dummy_str
'Constant Loaded'
>>> module1.dummy_func
<function dummy_func at 0x7f7a8a170410>

The Github Use Case!

The dedicated github_repo() context:
>>> from httpimport import github_repo
>>> with github_repo( 'operatorequals', 'covertutils', ) :
...     import covertutils
...
>>> covertutils.__author__
'John Torakis - operatorequals'
>>>
What about branches?
>>> from httpimport import github_repo
>>> with github_repo( 'operatorequals', 'covertutils', branch='py3_compatibility' ) :
...     import covertutils
...
>>> covertutils.__author__
'John Torakis - operatorequals'
>>>
And ad-hoc commits too?

What if you need to stick to a fixed -known to work- commit?

>>> from httpimport import github_repo
>>> with github_repo( 'operatorequals', 'covertutils', commit='cf3f78c77c437edf2c291bd5b4ed27e0a93e6a77' ) :
...     import covertutils
...
>>> covertutils.__author__
'John Torakis - operatorequals'
>>>

The newer sibling bitbucket_repo() (as of 0.5.9)

>>> with bitbucket_repo('atlassian', 'python-bitbucket', module='pybitbucket'):
...     import pybitbucket
...
>>>

Another sibling gitlab_repo() (as of 0.5.17)

>>> with gitlab_repo('harinathreddyk', 'python-gitlab', module='gitlab'):
...     from gitlab import const
...
>>>
The domain parameter for gitlab_repo()

You can point to your own installation of GitLab by using the domain parameter:

>>> with gitlab_repo('self', 'myproject', module='test_package', domain='127.0.0.1:8080'):
...     import test_package
...
>>>

This covers the posibility of using httpimport to target local development environments, which is a strong use case for httpimport.

Import remote (encrypted) ZIP files (as of 0.5.18)

After version 0.5.18 the add_remote_repo and the load functions, as well as the remote_repo context got the zip and zip_pwd parameters. By pointing to a HTTP/S URL containing a ZIP file, it is possible to remotely load modules/packages included in it, without downloading the ZIP file to disk!

>>> with httpimport.remote_repo(
...     ['test_package'], base_url='http://localhost:8000/test_package.zip',
...     ):
...    import test_package
...
>>>

Using a ZIP password (zip_pwd parameter)

>>> with httpimport.remote_repo(
...     ['test_package'], base_url='http://localhost:8000/test_package.enc.zip',
...     zip_pwd=b'[email protected]!'
...     ):
...    import test_package
...
>>>

Life suddenly got simpler for Python module testing!!!

Imagine the breeze of testing Pull Requests and packages that you aren't sure they are worth your download.

Recursive Dependencies

If package A requires module B and A exists in http://example.com/a_repo/, while B exists in http://example.com/b_repo/, then A can be imported using the following technique:

>>> from httpimport import remote_repo
>>> with remote_repo(['B'],"http://example.com/b_repo/") :
...     with remote_repo(['A'],"http://example.com/a_repo/") :
...             import A
... 
[!] 'B' not found in HTTP repository. Moving to next Finder.
>>> 
>>> A
<module 'A' from 'http://example.com/a_repo/A/__init__.py'>
>>> B
<module 'B' from 'http://example.com/a_repo/B.py'>
>>> 

Any combination of packages and modules can be imported this way!

The [!] Warning was emitted by the HttpImporter object created for A, as it couldn't locate B, and passed control to the next Finder object, that happened to be the HttpImporter object created for B!

Debugging...

>>> from httpimport import *
>>>
>>> import logging
>>> logging.getLogger('httpimport').setLevel(logging.DEBUG)
>>>
>>> with github_repo('operatorequals','covertutils') :
...     import covertutils
...
FINDER=================
[!] Searching covertutils
[!] Path is None
[@] Checking if connection is HTTPS secure >
[@] Checking if in declared remote module names >
[@] Checking if built-in >
[@] Checking if it is name repetition >
[*]Module/Package 'covertutils' can be loaded!
LOADER=================
[+] Loading covertutils
[+] Trying to import as package from: 'https://raw.githubusercontent.com/operatorequals/covertutils/master//covertutils/__init__.py'
[+] Importing 'covertutils'
[+] Ready to execute 'covertutils' code
[+] 'covertutils' imported succesfully!
>>>

Beware: Huge Security Implications!

Using the httpimport with HTTP URLs is highly discouraged outside the localhost interface!

As HTTP traffic isn't encrypted and/or integrity checked (unlike HTTPS), it is trivial for a remote attacker to intercept the HTTP responses (via an ARP MiTM probably), and add arbitrary Python code to the downloaded packages/modules. This will directly result in Remote Code Execution to your current user's context! In other words, you get totally F*ed...

Preventing the disaster (setting httpimport.INSECURE flag):

>>> import httpimport
>>>
>>> # Importing from plain HTTP ...
>>> httpimport.load('test_module', 'http://localhost:8000//')
[!] Using non HTTPS URLs ('http://localhost:8000//') can be a security hazard!
[-] 'httpimport.INSECURE' is not set! Aborting...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "httpimport.py", line 302, in load
    raise ImportError("Module '%s' cannot be imported from URL: '%s'" % (module_name, url) )
ImportError: Module 'test_module' cannot be imported from URL: 'http://localhost:8000/'
>>> # ... Throws Error!
>>>
>>> # Importing from plain HTTP has to be DELIBERATELY enabled!
>>> httpimport.INSECURE = True
>>> httpimport.load('test_module', 'http://localhost:8000//')
[!] Using non HTTPS URLs ('http://localhost:8000//') can be a security hazard!
<module 'test_module' from 'http://localhost:8000//test_module.py'>
>>> # Succeeded!

You have been warned! Use HTTPS URLs with httpimport!

Minification

This project has started to suggest stager code for HTTP/S RATs made with covertutils. The Documentation for minifying and using httpimport for such purposes can be found here.

Further minification can be achieved by python-minifier, also available in PyPI. So a minified version can be obtained as follows:

pip install python-minifer    # the "pyminify" command
curl https://raw.githubusercontent.com/operatorequals/httpimport/master/httpimport.py | sed 's#log.*#pass#g' | grep -v "import pass" | pyminify - > httpimport_min.py

size reduction:

# Original Size Count
$ curl https://raw.githubusercontent.com/operatorequals/httpimport/0.7.1/httpimport.py |  wc 
[...]
504    1914   18876
# Minified Size Count
$ curl https://raw.githubusercontent.com/operatorequals/httpimport/0.7.1/httpimport.py | sed 's#log.*#pass#g' | grep -v "import pass" | pyminify - | wc 
[...]
177     936   12141

Contributors

  • ldsink - The RELOAD flag and Bug Fixes
  • lavvy - the load() function
  • superloach - Deprecation of imp module in Python3 in favour of importlib
  • yanliakos - Bug Fix
Owner
John Torakis
It is all about what puzzles we prefer to delve into
John Torakis
Inverted-pendulum-with-fuzzy-control - Inverted pendulum with fuzzy control

Fuzzy Inverted Pendulum Basically, this project consists of an inverted pendulum

Mahan Ahmadvand 1 Aug 25, 2022
Exam assignment for Laboratory of Bioinformatics 2

Exam assignment for Laboratory of Bioinformatics 2 (Alma Mater University of Bologna, Master in Bioinformatics)

2 Oct 22, 2022
This collection is to provide an easier way to interact with Juniper

Ansible Collection - cremsburg.apstra Overview The goal of this collection is to provide an easier way to interact with Juniper's Apstra solution. Whi

Calvin Remsburg 1 Jan 18, 2022
Python package for handling and analyzing PSRFITS files

PyPulse A pure-Python package for handling and analyzing PSRFITS files. Read the documentation here. This is an alternate code base from PSRCHIVE. Req

Michael Lam 15 Nov 30, 2022
A simple BrainF**k compiler written in Python

bf-comp A simple BrainF**k compiler written in Python. What else were you looking for?

1 Jan 09, 2022
Personal Finance Forecaster - An AI tool for forecasting personal expenses

Personal Finance Forecaster - An AI tool for forecasting personal expenses

2 Mar 09, 2022
Return-Parity-MDP - Towards Return Parity in Markov Decision Processes

Towards Return Parity in Markov Decision Processes Code for the AISTATS 2022 pap

Jianfeng Chi 3 Nov 27, 2022
DC619/DC858 Mainframe Environment/Lab

DC619 Training LPAR The file DC619 - Mainframe Overflows Hands On.pdf contains the labs and walks through how to perform them. Use docker You can use

Soldier of FORTRAN 9 Jun 27, 2022
This is a program for Carbon Emission calculator.

Summary This is a program for Carbon Emission calculator. Usage This will calculate the carbon emission by each person on various factors. Contributor

Ankit Rane 2 Feb 18, 2022
Simple dotfile pre-processor with a per-file configuration

ix (eeks) Simple dotfile pre-processor with a per-file configuration Summary (TL;DR) ix.py is all you need config is an ini file. files to be processe

Poly 12 Dec 16, 2021
K2HASH Python library - NoSQL Key Value Store(KVS) library

k2hash_python Overview k2hash_python is an official python driver for k2hash. Install Firstly you must install the k2hash shared library: curl -o- htt

Yahoo! JAPAN 3 Oct 19, 2022
Pacman - A suite of tools for manipulating debian packages

Overview Repository is a suite of tools for manipulating debian packages. At a h

Pardis Pashakhanloo 1 Feb 24, 2022
A site that went kinda viral that lets you put Bernie Sanders in places

Bernie In Places An app that accidentally went viral! Read the story in WIRED here Install First, create a python virtual environment, and install all

310 Aug 22, 2022
p5 is a Python package based on the core ideas of Processing.

p5 p5 is a Python library that provides high level drawing functionality to help you quickly create simulations and interactive art using Python. It c

p5py 645 Jan 04, 2023
A Python Web Application for Checking vaccine slots by pincodes and auto slot booking.

The Dashboard is developed using Bokeh and python 3.5+. This dashboard is useful for you if you are looking for something which will help you to book the vaccine slot once slots become available. Oth

Suraj Deshmukh 10 Jan 23, 2022
Some Python scripts that fx(hash) users might find useful.

fx_hash_utils Some Python scripts that fx(hash) users might find useful. get_images This script downloads all the static images of the tokens generate

30 Oct 05, 2022
Quick script for automatically extracting syscall numbers for an OS

Syscalls-Extractor Quick script for automatically extracting syscall numbers for an OS $ python3 .\syscalls-extractor.py --help usage: syscalls-extrac

m0rv4i 54 Feb 10, 2022
Subnet calculator script using python

subnetCalculator Subnet calculator script using python3 Interactive Version Define the subnet variable interactively Use: subnetDict = subnetCalculato

1 Feb 15, 2022
Converts a base copy of Pokemon BDSP's masterdatas into a more readable and editable Pokemon Showdown Format.

Showdown-BDSP-Converter Converts a base copy of Pokemon BDSP's masterdatas into a more readable and editable Pokemon Showdown Format. Download the lat

Alden Mo 2 Jan 02, 2022
An easy FASTA object handler, reader, writer and translator for small to medium size projects without dependencies.

miniFASTA An easy FASTA object handler, reader, writer and translator for small to medium size projects without dependencies. Installation Using pip /

Jules Kreuer 3 Jun 30, 2022