A tool to build reproducible wheels for you Python project or for all of your dependencies

Related tags

Miscellaneousasaman
Overview

asaman: Amra Saman (আমরা সমান)

This is a tool to build reproducible wheels for your Python project or for all of your dependencies. What this means is if you use the same Operating System version and similar system level dependencies, you will always get the same wheel generated. This enables us to have a bit more protection from side-channel attacks. Any user of the wheels can verify that they are using the correct build from the exact source via verifying the builds themselves.

Why do we need a reproducible wheel?

A few different positive points:

  • If we build the wheels from a known source (e.g. via pinned hashes in requirements file), we can also verify if we are using the correct wheels built from them.
  • Any user/developer can rebuild the wheels from the pinned source and should get the exact same wheel as output. Thus if anything gets into the build process (say in CI), or the wheel is actually built from a different source, automated tools can identify that difference.

How to install?

python3 -m pip install asaman

How to build reproducible wheels?

asaman --help
Usage: asaman [OPTIONS]

  Tool to build reproducible wheels.

Options:
  -s, --source FILE          A single source tarball or zip file.
  -d, --directory DIRECTORY  A directory containing all source tarballs and
                             zips.
  -o, --output DIRECTORY     The output directory to store all wheel files.
                             Default: ./wheels
  -r, --requirement FILE     Path to the requirement.txt file which contains
                             all packages to build along with hashes.
  --sde TEXT                 Custom SOURCE_DATE_EPOCH value.
  --help                     Show this message and exit.

To build a reproducible wheel for a given source tarball:

asaman -s dist/yourpackage_4.2.0.tar.gz

By default the freshly built wheel will be stored in the ./wheels/ directory. You can specify a different directory using -o/--output.

To build reproducible wheels for all the sources from a directory:

asaman -d path/to/sources/

Or, you can point to a requirements file which contains all the dependencies along with hashes:

asaman -r requirements.txt

How to generate a requirements file with hashes from the reproducible wheels?

asaman-generate requirements.txt

The asaman-generate command will help you to create a fresh verified-requirements.txt, which will contain the hashes from reproducible wheels. You can pass the -o/--output option to pass your custom file name.

asaman-generate --help
Usage: asaman-generate [OPTIONS] REQUIREMENT

  Tool to build verified requirements file from reproducible wheels.

Options:
  -o, --output FILE       The output file. Default: verified-{requirement}.txt
  -w, --wheels DIRECTORY  The directory with reproducible wheels.
  -s, --skip TEXT         The packages we don't want in our final requirement
                          file.
  --help                  Show this message and exit.

How to create a requirements file with hashes from PyPI or your personal index?

Use the pip-tools project.

pip-compile --generate-hashes --allow-unsafe --output-file=requirements.txt requirements.in

Please make sure that include all the build dependencies of any dependency. If you don't then pip will download the build dependencies from PyPI and install them in the build environment.

To help identify build dependencies while you are building from a requirements file, during download and extracting each source tarball via pip, you can notice any dependency which has build time dependency or not. Otherwise, you can manually look at the build-time dependencies.

For example, in the following text you can find a few packages with build time dependencies. Look at the lines with Getting requirements to build wheel.

Collecting build==0.7.0
  Using cached build-0.7.0.tar.gz (15 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting click==8.0.1
  Using cached click-8.0.1.tar.gz (327 kB)
Collecting packaging==21.0
  Using cached packaging-21.0.tar.gz (83 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting pep517==0.11.0
  Using cached pep517-0.11.0.tar.gz (25 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done

Bootstrapping the build environment

For any production use, you should also bootstrap the build environment and create the initial virtual environment to build all dependencies in that environment only. You can store the wheels in any place you want e.g. (S3, or git-lfs), and start from there when creating the environment next time.

In the following commands, we will create a set of wheels for such a bootstrap environment. We will start with listing the build requirements in bootstrap.in with the following contents:

amrasaman >=0.1.0
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install pip-tools # This is being downloaded directly from PyPI.
pip-compile --generate-hashes --allow-unsafe --output-file=bootstrap.txt bootstrap.in
asaman -r bootstrap.txt

This will create all the wheels in the ./wheels directory.

Next time we can install the wheels from the ./wheels directory. But first we will create a new requirements file with only the hashes from our reproducible wheels, the output file name will be verified-bootstrap.txt.

asaman-generate bootstrap.txt

Now we can use this requirements file to create the environment.

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --no-index --find-links ./wheels --require-hashes --only-binary :all: -r verified-bootstrap.txt 

Meaning of the name

In Bengali it means "we are same"

Developer documentation

Read the hacking guide.

Owner
Kushal Das
Kushal Das
This is an online course where you can learn and master the skill of low-level performance analysis and tuning.

Performance Ninja Class This is an online course where you can learn to find and fix low-level performance issues, for example CPU cache misses and br

Denis Bakhvalov 1.2k Dec 30, 2022
HSPyLib is a Python library that will elevate your experience to another level.

HomeSetup Python Library - HSPyLib Your mature python application HSPyLib is a Python library that will elevate your experience to another level. It r

Hugo Saporetti Junior 4 Dec 14, 2022
VCC-Generator is a python script that generate VCC for testing purposes only

VCC-Generator is a python script that generate VCC for testing purposes only

Spider Anongreyhat 10 Oct 23, 2022
Basic Hspice runner with Python

HSpicePy Bilgisayarınıza PATH değişkenlerine eklediğiniz HSPICE programını python ile çalıştırmanızı sağlayan basit bir araç. A simple tool that allow

1 Nov 16, 2021
Automated Birthday Wisher built using Python

Automated Birthday Wisher This Automation of wishing Birthday is achieved using Python. Never forget to wish birthday! Table of contents Overview Scre

yashviradia 1 Nov 29, 2021
Given tool find related trending keywords of input keyword

blog_generator Given tool find related trending keywords of input keyword (blog_related_to_keyword). Then cretes a mini blog. Currently its customised

Shivanshu Srivastava 2 Nov 30, 2021
x-tools is a collection of tools developed in Python

x-tools X-tools is a collection of tools developed in Python Commands\

5 Jan 24, 2022
Control your gtps with gtps-tools!

Note Please give credit to me! Do not try to sell this app, because this app is 100% open source! Do not try to reupload and rename the creator app! S

Jesen N 6 Feb 16, 2022
Script to quickly get the metrics from Github repos to analyze.

commit-prefix-analysis Script to quickly get the metrics from Github repos to analyze. Setup Install the Github CLI. You'll know its working when runn

David Carpenter 1 Dec 17, 2022
Powerful virtual assistant in python

Virtual assistant in python Powerful virtual assistant in python Set up Step 1: download repo and unzip Step 2: pip install requirements.txt (if py au

Arkal 3 Jan 23, 2022
Displays Christmas-themed ASCII art

Christmas Color Scripts Displays Christmas-themed ASCII art. This was mainly inspired by DistroTube's Shell Color Scripts Screenshots ASCII Shadow Tex

1 Aug 09, 2022
An a simple sistem code in python

AMS OS An a simple code in python ⁕¿What is AMS OS? AMS OS is an a simple sistem code writed in python. This code helps you with the cotidian task, yo

1 Nov 10, 2021
CALPHAD tools for designing thermodynamic models, calculating phase diagrams and investigating phase equilibria.

CALPHAD tools for designing thermodynamic models, calculating phase diagrams and investigating phase equilibria.

pycalphad 189 Dec 13, 2022
A package with multiple bias correction methods for climatic variables, including the QM, DQM, QDM, UQM, and SDM methods

A package with multiple bias correction methods for climatic variables, including the QM, DQM, QDM, UQM, and SDM methods

Sebastián A. Aedo Quililongo 9 Nov 18, 2022
A basic animation modding workflow for FFXIV

AnimAssist Provides a quick and easy way to mod animations in FFXIV. You will need: Before anything, the VC++2012 32-bit Redist from here. Havok will

liam 37 Dec 16, 2022
Mini-calculadora escrita como exemplo para uma palestra relâmpago sobre `git bisect`

Calculadora Mini-calculadora criada para uma palestra relâmpado sobre git bisect. Tem até uma colinha! Exemplo de uso Modo interativo $ python -m calc

Eduardo Cuducos 3 Dec 14, 2021
A simple but complete exercise to learning Python

ResourceReservationProject This is a simple but complete exercise to learning Python. Task and flow chart We are going to do a new fork of the existin

2 Nov 14, 2022
Add any Program in any language you like or add a hello world Program ❣️ if you like give us :star:

Welcome to the Hacktoberfest 2018 Hello-world 📋 This Project aims to help you to get started with using Github. You can find a tutorial here What is

Aniket Sharma 1.5k Nov 16, 2022
One Ansible Module for using LINE notify API to send notification. It can be required in the collection list.

Ansible Collection - hazel_shen.line_notify Documentation for the collection. ansible-galaxy collection install hazel_shen.line_notify --ignore-certs

Hazel Shen 4 Jul 19, 2021
TMTC Commander Core

This commander application was first developed by KSat for the SOURCE project to test the on-board software but has evolved into a more generic tool for satellite developers to perform TMTC (Telemetr

robamu 8 Dec 14, 2022