A library which implements low-level functions that relate to packaging and distribution of Python

Related tags

Distributiondistlib
Overview

AppVeyor

https://coveralls.io/repos/github/vsajip/distlib/badge.svg?branch=master

What is it?

Distlib is a library which implements low-level functions that relate to packaging and distribution of Python software. It is intended to be used as the basis for third-party packaging tools. The documentation is available at

https://distlib.readthedocs.io/

Main features

Distlib currently offers the following features:

  • The package distlib.database, which implements a database of installed distributions, as defined by PEP 376, and distribution dependency graph logic. Support is also provided for non-installed distributions (i.e. distributions registered with metadata on an index like PyPI), including the ability to scan for dependencies and building dependency graphs.
  • The package distlib.index, which implements an interface to perform operations on an index, such as registering a project, uploading a distribution or uploading documentation. Support is included for verifying SSL connections (with domain matching) and signing/verifying packages using GnuPG.
  • The package distlib.metadata, which implements distribution metadata as defined by PEP 426, PEP 345, PEP 314 and PEP 241.
  • The package distlib.markers, which implements environment markers as defined by PEP 426.
  • The package distlib.manifest, which implements lists of files used in packaging source distributions.
  • The package distlib.locators, which allows finding distributions, whether on PyPI (XML-RPC or via the "simple" interface), local directories or some other source.
  • The package distlib.resources, which allows access to data files stored in Python packages, both in the file system and in .zip files.
  • The package distlib.scripts, which allows installing of scripts with adjustment of shebang lines and support for native Windows executable launchers.
  • The package distlib.version, which implements version specifiers as defined by PEP 440 / PEP 426, but also support for working with "legacy" versions (setuptools/distribute) and semantic versions.
  • The package distlib.wheel, which provides support for building and installing from the Wheel format for binary distributions (see PEP 427).
  • The package distlib.util, which contains miscellaneous functions and classes which are useful in packaging, but which do not fit neatly into one of the other packages in distlib.* The package implements enhanced globbing functionality such as the ability to use ** in patterns to specify recursing into subdirectories.

Python version and platform compatibility

Distlib is intended to be used on any Python version >= 2.7 and is tested on Python versions 2.7 and 3.3-3.6 on Linux, Windows, and Mac OS X (not all versions are tested on all platforms, but are expected to work correctly).

Project status

The project has reached a mature status in its development: there is a test suite and it has been exercised on Windows, Ubuntu and Mac OS X. The project is used by well-known projects such as pip and caniusepython3.

Code of Conduct

Everyone interacting in the distlib project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the PyPA Code of Conduct.

Comments
  • 0.3.5: processes of the

    0.3.5: processes of the "w" Windows launcher executables can't be killed anymore

    Describe the bug

    After the launcher files were updated in 8d2acb51bae32a0237f2d2419b0636bce807b6f7 (0.3.5), processes spawned from the "w" launchers can't be killed anymore. I am facing this issue when trying to spawn and kill the processes from a NodeJS context. Calling childprocess.kill() via NodeJS's child_process module returns success, but the python process keeps running.

    This is a major problem for me, because my NodeJS application (GUI application via NW.js / Electron) is running a python CLI application built with the distlib launcher executables, and it needs to use the "w" launchers, because otherwise a console window pops up, which is bad for the user experience.

    0.3.3 is the most recent release which is working fine.

    0.3.4 had the immediate 3221225477 / 0xC0000005 exit code error, and I already had to ignore this version when building standalone installers of my python application. Fortunately, pip had reverted back to distlib 0.3.3 until recently, so installing via pip was no problem either, but now it has merged the distlib 0.3.5 release, which is once again bad, and the issue needs to be fixed. Related (old 0.3.4 issue): https://github.com/takluyver/pynsist/issues/243#issuecomment-1192054549

    Apparently, these are the recent launcher changes: https://bitbucket.org/vinay.sajip/simple_launcher/commits/

    To Reproduce

    I've created a reproduction repo (already did this for 0.3.4). Please see the build config and build+run scripts for the 0.3.5 issue: https://github.com/bastimeyer/distlib-bug/tree/53f520efec1a6173b7ca037d26c89a6dec8ca633

    The resulting GH action logs: https://github.com/bastimeyer/distlib-bug/actions/runs/2718462897

    0.3.5-t64 (success): https://github.com/bastimeyer/distlib-bug/runs/7467911880?check_suite_focus=true#step:7:1 0.3.5-w64 (failure): https://github.com/bastimeyer/distlib-bug/runs/7467912062?check_suite_focus=true#step:7:1

    0.3.3-t64 (success): https://github.com/bastimeyer/distlib-bug/runs/7467911132?check_suite_focus=true#step:7:1 0.3.3-w64 (success): https://github.com/bastimeyer/distlib-bug/runs/7467911332?check_suite_focus=true#step:7:1

    Environment

    • OS, including version: Windows (any)
    • Version of this library: 0.3.5
    opened by bastimeyer 31
  • Add ARM-compatible launchers

    Add ARM-compatible launchers

    Original report by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco).


    In this pull request for setuptools, anthrotype is working to update Setuptools to use distlib for its script launchers, unifying yet another piece of the python packaging infrastructure.

    However, setuptools' executable launchers support one major feature that distlib does not currently, and that's ARM-compatible executables.

    Setuptools builds its launches with this script. Can someone update distlib to include builds of the launcher for ARM as well?

    enhancement major 
    opened by vsajip 23
  • distlib.wheel.SHEBANG_DETAIL_RE is too restrictive

    distlib.wheel.SHEBANG_DETAIL_RE is too restrictive

    Original report by C Anthony Risinger (Bitbucket: anthonyrisinger, GitHub: anthonyrisinger).


    the regex:

    (\s*#!("[^"]+"|[\w/\\:.]+))\s+(.*)$
    

    ...does not reflect all valid shebangs, eg. an @ in the path leads to failure. also, even though .match(...) is used, the regex should be anchored:

    ^(\s*#!("[^"]+"|[\w/\\:.]+))\s+(.*)$
    

    ...anything save \0 and / may be in the path:

    ^(\s*#!("[^"]+"|\S+))\s+(.*)$
    

    ...there are better ways too (such as setuptools method of open() + readline()) but this makes stuff go again.

    bug major 
    opened by vsajip 22
  • Generated wrapper scripts fail when Python executable path contains spaces

    Generated wrapper scripts fail when Python executable path contains spaces

    Original report by Paul Moore (Bitbucket: pmoore, GitHub: pmoore).


    See https://github.com/pypa/pip/issues/5223. This is a priority issue, as it means that the wrapper pip.exe executable generated by pip 10 won't work for a Python installation in a directory containing a space (e.g. C:\Program Files)

    To reproduce, create a virtualenv with a space in the name. I'm here using the copy of distlib vendored with pip 10 (which is distlib 0.2.6), as that's where the bug was first reported.

    The file "a.py" below simply contains

    def f():
        print("Hello from", __file__)
    

    To reproduce the issue:

    PS 13:30 {19:29.901} E:\Work\Scratch\a b                                                               
    >.\Scripts\python.exe                                                                                  
    Python 3.6.2 (v3.6.2:5fd33b5, Jul  8 2017, 04:57:36) [MSC v.1900 64 bit (AMD64)] on win32              
    Type "help", "copyright", "credits" or "license" for more information.                                 
    >>> import sys                                                                                         
    >>> sys.path.append(r'E:\Work\Projects\pip\src\pip\_vendor')                                           
    >>> from distlib import scripts                                                                        
    >>> s = scripts.ScriptMaker(r'E:\Work\Scratch\a b\x', r'E:\Work\Scratch\a b', add_launchers=True)      
    >>> s.make('a=a:f')                                                                                    
    ['E:\\Work\\Scratch\\a b\\a.exe', 'E:\\Work\\Scratch\\a b\\a-3.6.exe']                                 
    

    This creates a.exe, which when run, gives an error as follows:

    >& 'E:\Work\Scratch\a b\a.exe'
    Fatal error in launcher: Unable to create process using '""e:\work\scratch\a b\scripts\python.exe"  "E:\Work\Scratch\a b\a.exe" '
    
    bug critical 
    opened by vsajip 18
  • JSONLocator broken? (corrupt data at red-dove.com?

    JSONLocator broken? (corrupt data at red-dove.com?

    Original report by C Anthony Risinger (Bitbucket: anthonyrisinger, GitHub: anthonyrisinger).


    results from JSONLocator appear to be unusable sometime within the last 4-6 hrs:

    #!python
    >>> from distlib.locators import default_locator
    >>> # get from JSONLocator only
    >>> s = default_locator.locators[0].locate('setuptools')
    >>> s.metadata
    <Metadata 2.0 setuptools (0.9.1)>
    >>> s.metadata.get_requirements()
    [...]
      File "/home/anthony/devel/local/zppy/external-library/distlib/database.py", li
    ne 349, in run_requires
        return self._get_requirements('run_requires', 'run_may_require')
      File "/home/anthony/devel/local/zppy/external-library/distlib/database.py", li
    ne 345, in _get_requirements
        env=self.context))
    TypeError: unhashable type: 'dict'
    

    my app is actually using the DependencyFinder, and i know for a fact that earlier today it was properly returning set(, , [...])

    it could be only some packages are messed up... i changed my tool to cache JSONLocator metadata so i can load it later when it's local (eg. as a way to endow DirectoryLocator with proper metadata)... however, ATM, i don't seem to be able to query anything without an exception, even when using the defaults:

    #!python
    from distlib.locators import default_locator, DependencyFinder
    DependencyFinder(default_locator).find('gitdb')
    

    ...throws the same unhashable type :(

    bug major 
    opened by vsajip 18
  • Allow modification of wheels

    Allow modification of wheels

    Original report by James Tocknell (Bitbucket: aragilar, GitHub: aragilar).


    Currently the distlib.wheel.Wheel only allows for creating a wheel from files or install files from a wheel, not modifying the wheel in place or creating a new wheel wheel from an old one. Use cases for this would be to update the metadata in the wheel from a legacy version to 2.0, or to add dependencies not contained in setup.py (e.g. add numpy dependency to a scipy wheel, which was what I'm trying to do).

    A workaround is to install the wheel into some temporary directory, do the required modifications, and then rebuild, or to modify the zip file directly (what I'm doing currently), but this is more likely to create buggy wheels.

    minor proposal 
    opened by vsajip 17
  • Simplify the Generated Script

    Simplify the Generated Script

    Original report by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft).


    Is there any reason not to simplify the generated script? Something like.

    # -*- coding: utf-8 -*-
    if __name__ == '__main__':
        import sys
        from %(module)s import %(func)s
        sys.exit(%(func)s())
    

    I believe would do the exact same thing as you're doing now, with less code.

    minor proposal 
    opened by vsajip 17
  • sys.stderr is None when running generated exe in bash on Windows

    sys.stderr is None when running generated exe in bash on Windows

    Describe the bug On Windows using distlib version 0.3.4 and using a bash terminal, when I run an exe file generated by distlib, sys.stderr is set to None. Running the same exe from cmd or powershell works as expected. Generating the exe with distlib 0.3.3 also works as expected.

    This was first reported in IPython https://github.com/ipython/ipython/issues/13509. IPython works fine when running python -m IPython or when running ipython.exe from other shells like cmd or powershell. This was later fixed because pip downgraded distlib here but I thought to report it here for visibility.

    I'm including a small example that prints sys.stdout and sys.stderr for comparison.

    To Reproduce Steps to reproduce the behavior:

    1. Download and unzip this file: bugexe.zip
    2. Go to that folder and create a venv: python -m venv .venv
    3. Activate the venv: source .venv/Scripts/activate
    4. Install distlib 0.3.4 python -m pip install distlib==0.3.4
    5. Run python make.py to create an executable called foo.exe
    6. From bash run: PYTHONPATH=. ./foo.exe It prints this:
    sys.stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
    sys.stderr=None
    
    1. For comparison, run from cmd: cmd /V /C "set "PYTHONPATH=." && foo.exe"
    2. Or powershell: powershell -Command { $env:PYTHONPATH="."; .\foo.exe }

    Expected behavior It should print this:

    sys.stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
    sys.stderr=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>
    

    Environment

    • OS: Windows 10
    • Using bash version 4.4.23(2)-release (x86_64-pc-msys) that comes bundled with git for windows
    • Version of this library: 0.3.4

    Additional information This problem doesn't happen with distlib version 0.3.3

    opened by charlieman 16
  • Incorrect shebang generated by Pip when run as part of CPython build and installing to a DESTDIR

    Incorrect shebang generated by Pip when run as part of CPython build and installing to a DESTDIR

    Describe the bug

    Since CPython 3.9.11 / pip 22.0.4 / distlib 0.3.3, the pip3 script generated at CPython install time has an incorrect shebang line.

    To Reproduce Steps to reproduce the behavior:

    In a CPython git clone:

    1. mkdir /tmp/destdir
    2. ./configure --prefix=/usr/local --with-ensurepip=yes
    3. make -j
    4. DESTDIR=/tmp/destdir make install
    5. head /tmp/destdir/usr/local/bin/pip3

    Expected behavior

    The shebang I see with CPython 3.9.11 / pip 22.0.4 / distlib 0.3.3 is: #!/usr/local/bin/python. This file does not exist.

    The shebang I see with CPython 3.9.8 / pip 21.2.4 / distlib 0.3.2 is: #!/usr/local/bin/python3. This file will exist, but it does not at the time the CPython install runs.

    Environment

    • CentOS 7

    Additional information

    This issue is a little hard to track down, as it only reproduces during CPython build, where distlib is embedded inside the file Lib/ensurepip/_bundled/pip-22.0.4-py3-none-any.whl.

    I suspect that the regression was introduced in this commit: https://github.com/pypa/distlib/commit/ad0ea221431689cb82c89b4942228fb07a87784d

    This adds a line to ScriptMaker._get_shebang() which does the following:

            if not os.path.isfile(executable):
                # for Python builds from source on Windows, no Python executables with
                # a version suffix are created, so we use python.exe
                executable = os.path.join(sysconfig.get_config_var('BINDIR'),
                                'python%s' % (sysconfig.get_config_var('EXE')))
    

    In my scenario, executable = /usr/local/bin/python3. This file does not exist, because we ran DESTDIR=/tmp/destdir make install and so the file has been installed to /tmp/destdir/usr/local/bin/python3. However, this is the path that should be used in the script shebang line, and distlib <= 0.3.2 handled the situation correctly. The change introduced in distlib 0.3.3 incorrectly chooses /usr/local/bin/python instead.

    The change is subtle, many OS distributors will create a symlink bin/python3 -> bin/python that hides the issue. In my case we do not have such a symlink, thus we get a pip3 executable that fails to start with an error like:

    Failed to execute process '/usr/local/bin/pip3'. Reason:
    The file '/usr/local/bin/pip3' specified the interpreter '/usr/local/bin/python', which is not an executable command.
    
    opened by sam-thursfield-nutanix 14
  • Distlib fails with latest Wheel release

    Distlib fails with latest Wheel release

    Original report by Anonymous.


    https://github.com/pypa/wheel/issues/294

    They have changed their directory inclusion behavior. Unclear to me if this is a distlib or a wheel bug, but wanted to get a tracking issue here.

    bug major 
    opened by vsajip 14
  • red-dove metadata not being updated?

    red-dove metadata not being updated?

    Original report by C Anthony Risinger (Bitbucket: anthonyrisinger, GitHub: anthonyrisinger).


    For a long time it seemed like pep426 was the way forward, but recently it appears to have fallen out of style (recent build abstraction threads, etc), and even the original author is no longer interested.

    We've been using red-dove via distlib for quite a long time, subconsciously knowing this day would come, but alas, we never moved back to pip vanilla.

    Would it be possible to re-enable the sync for a while longer? We are currently migrating back to a straight pip build and adjusting our CI appropriately.

    major task 
    opened by vsajip 14
  • I'll get to it at some point, but it's not a high priority.

    I'll get to it at some point, but it's not a high priority.

        I'll get to it at some point, but it's not a high priority.
    

    Originally posted by @vsajip in https://github.com/pypa/distlib/issues/159#issuecomment-992982760

    invalid 
    opened by a7mdalqrny 0
  • Rename the default branch to `main`

    Rename the default branch to `main`

    Hi everybody!

    I noticed that, recently, this project moved to the PyPA GitHub organization, and I like to see that!

    However, many of our repositories here have renamed its default branch from master to main. For consistency reasons, can we do the same here?

    opened by DiddiLeija 4
  • In general, ScriptMaker does not validate the Python executable is a shell script or not.

    In general, ScriptMaker does not validate the Python executable is a shell script or not.

    Original report by 林玮 (Jade Lin) (Bitbucket: 林玮 (Jade Lin), ).


    The executable path that ScriptMaker got may be a script. For example, the Python executable installed from nix is a bash script like this.

    #! /nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash -e
    export NIX_PYTHONPREFIX='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env'
    export NIX_PYTHONEXECUTABLE='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9'
    export NIX_PYTHONPATH='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/lib/python3.9/site-packages'
    export PYTHONNOUSERSITE='true'
    exec "/nix/store/7pjbbmnrch7frgyp7gz19ay0z1173c7y-python3-3.9.2/bin/python3.9"  "$@"
    

    ScriptMaker only validates it and adapts it while running on the Java platform. Maybe refactor to apply the adaption for general cases.

    https://bitbucket.org/pypa/distlib/src/4f54a3117db58f6bb00c69ab9a892170b91acd3b/distlib/scripts.py#lines-102:125

    bug trivial 
    opened by vsajip 0
Releases(0.3.6)
Owner
Python Packaging Authority
Python Packaging Authority
A tool used to obfuscate python scripts, bind obfuscated scripts to fixed machine or expire obfuscated scripts.

PyArmor Homepage (中文版网站) Documentation(中文版) PyArmor is a command line tool used to obfuscate python scripts, bind obfuscated scripts to fixed machine

Dashingsoft 1.9k Jan 01, 2023
shiv is a command line utility for building fully self contained Python zipapps as outlined in PEP 441, but with all their dependencies included.

shiv shiv is a command line utility for building fully self-contained Python zipapps as outlined in PEP 441, but with all their dependencies included!

LinkedIn 1.5k Dec 28, 2022
debinstaller - A tool to install .deb files in any distro.

debinstaller A tool to install .deb files in any distro. Installation for debinstaller

Manoj Paramsetti 6 Nov 06, 2022
pipx — Install and Run Python Applications in Isolated Environments

Install and Run Python Applications in Isolated Environments

Python Packaging Authority 5.9k Jan 07, 2023
Ninja is a small build system with a focus on speed.

Ninja Python Distributions Ninja is a small build system with a focus on speed. The latest Ninja python wheels provide ninja 1.10.2.g51db2.kitware.job

33 Dec 19, 2022
An example of repository data as bundles

Bundles This repository is just an example of how we can host Git bundles in a way that supports fetching data from precomputed bundles without the or

Derrick Stolee 1 Jan 02, 2022
Nuitka Organization 8k Jan 07, 2023
A library which implements low-level functions that relate to packaging and distribution of Python

What is it? Distlib is a library which implements low-level functions that relate to packaging and distribution of Python software. It is intended to

Python Packaging Authority 29 Oct 11, 2022
local pypi server (custom packages and auto-mirroring of pypi)

localshop A PyPI server which automatically proxies and mirrors PyPI packages based upon packages requested. It has support for multiple indexes and t

Michael van Tellingen 383 Sep 23, 2022
py2app is a Python setuptools command which will allow you to make standalone Mac OS X application bundles and plugins from Python scripts.

py2app is a Python setuptools command which will allow you to make standalone Mac OS X application bundles and plugins from Python scripts. py2app is

Ronald Oussoren 222 Dec 30, 2022
auto packaging for iOS

iOS Auto Packaging iOS自动打包脚本 准备 脚本第一次执行之前 先检查依赖, packaging目录下终端执行 pip3 install -r requirements.txt 运行 cd packaging packaging.py -h help -s scheme

DeeCo 17 Jul 23, 2022
tool for creating installers from conda packages

(conda) Constructor Description Constructor is a tool which allows constructing an installer for a collection of conda packages. It solves needed pack

Conda 386 Jan 04, 2023
Subpar is a utility for creating self-contained python executables. It is designed to work well with Bazel.

Subpar Subpar is a utility for creating self-contained python executables. It is designed to work well with Bazel. Status Subpar is currently owned by

Google 550 Dec 27, 2022
Auto locust load test config and worker distribution with Docker and GitHub Action

Auto locust load test config and worker distribution with Docker and GitHub Action Install Fork the repo and change the visibility option to private S

Márk Zsibók 1 Nov 24, 2021
FreezeUI is a python package that creates applications using cx_freeze and GUI by converting .py to .exe .

FreezeUI is a python package use to create cx_Freeze setup files and run them to create applications and msi from python scripts (converts .py to .exe or .msi .

4 Aug 25, 2022
Create standalone executables from Python scripts, with the same performance and is cross-platform.

About cx_Freeze cx_Freeze creates standalone executables from Python scripts, with the same performance, is cross-platform and should work on any plat

Marcelo Duarte 1k Jan 04, 2023
executable archive format

XAR XAR lets you package many files into a single self-contained executable file. This makes it easy to distribute and install. A .xar file is a read-

Facebook Incubator 1.5k Dec 29, 2022
A library and tool for generating .pex (Python EXecutable) files

PEX Contents Overview Installation Simple Examples Integrating pex into your workflow Documentation Development Contributing Overview pex is a library

Pants Build 2.2k Jan 01, 2023
Build Windows installers for Python applications

Pynsist is a tool to build Windows installers for your Python applications. The installers bundle Python itself, so you can distribute your applicatio

Thomas Kluyver 818 Jan 05, 2023
Psgcompiler A PySimpleGUI Application - Transform your Python programs in Windows, Mac, and Linux binary executables

psgcompiler A PySimpleGUI Application "Compile" your Python programs into an EXE for Windows, an APP for Mac, and a binary for Linux Installation Old-

PySimpleGUI 77 Jan 07, 2023