A Python script that wraps the gitleaks tool to enable scanning of multiple repositories in parallel

Overview

mpgitleaks

build Code Grade complexity vulnerabilities python

A Python script that wraps the gitleaks tool to enable scanning of multiple repositories in parallel.

The motivation behind writing this script was:

  • implement workaround for gitleaks intermittent failures when cloning very large repositories
  • implement ability to scan multiple repostiories in parallel
  • implement ability to scan repositories for a user, a specified organization or read from a file

Notes:

  • the script uses https to clone the repos
    • you must set the USERNAME and PASSWORD environment variables - this credential needs to have access to the repos being scanned
    • if using --file then https clone urls must be supplied in the file
  • the maximum number of background processes (workers) that will be started is 35
    • if the number of repos to process is less than the maximum number of workers
      • the script will start one worker per repository
    • if the number of repos to process is greater than the maximum number of workers
      • the repos will be added to a thread-safe queue and processed by all the workers
  • the Docker container must run with a bind mount to the working directory in order to access logs/reports
    • the repos will be cloned to the ./scans/clones folder in the working directory
    • the reports will be written to the ./scans/reports/ folder in the working directory
    • a summary report will be written to mpgitleaks.csv

Usage

usage: mpgitleaks [-h] [--file FILENAME] [--user] [--org ORG] [--exclude EXCLUDE] [--include INCLUDE] [--debug]

A Python script that wraps the gitleaks tool to enable scanning of multiple repositories in parallel

optional arguments:
  -h, --help         show this help message and exit
  --file FILENAME    scan repos contained in the specified file
  --user             scan repos for the authenticated GitHub user where user is owner or collaborator
  --org ORG          scan repos for the specified GitHub organization
  --exclude EXCLUDE  a regex to match name of repos to exclude from scanning
  --include INCLUDE  a regex to match name of repos to include in scanning
  --debug            log debug messages to a log file

Execution

Set the required environment variables:

export USERNAME='--username--'
export PASSWORD='--password-or-token--'

If using --user or --org options and GitHub instance is not api.github.com:

export GH_BASE_URL='--api-address-to-github-instance--'

Execute the Docker container:

docker container run \
--rm \
-it \
-e http_proxy \
-e https_proxy \
-e GH_BASE_URL \
-e USERNAME \
-e PASSWORD \
-v $PWD:/opt/mpgitleaks \
soda480/mpgitleaks:latest \
[MPGITLEAKS OPTIONS]

Note: the http[s]_proxy environment variables are only required if executing behind a proxy server

Examples

Scan all repos contained in the file repos.txt but exclude the repos that match the specified regex, an example of a repos.txt can be found here:

mpgitleaks --file 'repos.txt' --exclude 'soda480/mplogp'

example

Scan all repos for the authenticated user but exclude the repos that match the specified regex:

mpgitleaks --user --exclude 'intel|edgexfoundry|soda480/openhack'

Scan all repos in the specified organization but only include the repos that match the specified regex:

mpgitleaks --org 'myorg' --include '.*-go'

Development

Clone the repository and ensure the latest version of Docker is installed on your development server.

Build the Docker image:

docker image build \
--target build \
--build-arg http_proxy \
--build-arg https_proxy \
-t \
mpgitleaks:latest .

Run the Docker container:

docker container run \
--rm \
-it \
-e http_proxy \
-e https_proxy \
-v $PWD:/code \
mpgitleaks:latest \
/bin/bash

Build application:

pyb -X
You might also like...
Script to post multiple status(posts) on twitter

Script to post multiple status on twitter (i.e. TWITTER STORM) This program can post upto maximum limit of twitter(around 300 tweets) within seconds.

Start multiple bots using one script. VK RAID BOTNET

MultiRaidBotnet Start multiple bots using one script. VK RAID BOTNET Русский launcher.py - главный скрипт, запускающий весь ботнет config.py - в нём х

Discord Mass Report script that uses multiple tokens

Discord-Mass-Report Discord Mass Report script that uses multiple tokens, full credits to https://github.com/hoki0/Discord-mass-report who made it in

Discord bot script for sending multiple media files to a discord channel according to discord limitations.

Discord Bulk Image Sending Bot Send bulk images to Discord channel. This is a bot script that will allow you to send multiple images to Discord channe

DDoS Script (DDoS Panel) with Multiple Bypass ( Cloudflare UAM,CAPTCHA,BFM,NOSEC / DDoS Guard / Google Shield / V Shield / Amazon / etc.. )
DDoS Script (DDoS Panel) with Multiple Bypass ( Cloudflare UAM,CAPTCHA,BFM,NOSEC / DDoS Guard / Google Shield / V Shield / Amazon / etc.. )

KARMA DDoS DDoS Script (DDoS Panel) with Multiple Bypass ( Cloudflare UAM,CAPTCHA,BFM,NOSEC / DDoS Guard / Google Shield / V Shield / Amazon / etc.. )

Python tool to Check running WebClient services on multiple targets based on @leechristensen
Python tool to Check running WebClient services on multiple targets based on @leechristensen

WebClient Service Scanner Python tool to Check running WebClient services on multiple targets based on @tifkin_ idea. This tool uses impacket project.

This is a very easy to use tool developed in python that will search for free courses from multiple sites including youtube and enroll in the ones in which it can.
This is a very easy to use tool developed in python that will search for free courses from multiple sites including youtube and enroll in the ones in which it can.

Free-Course-Hunter-and-Enroller This is a very easy to use tool developed in python that will search for free courses from multiple sites including yo

Free python/telegram bot for easy execution and surveillance of crypto trading plans on multiple exchanges.
Free python/telegram bot for easy execution and surveillance of crypto trading plans on multiple exchanges.

EazeBot Introduction Have you ever traded cryptocurrencies and lost overview of your planned buys/sells? Have you encountered the experience that your

Releases(v0.2.1)
Owner
Emilio Reyes
Emilio Reyes
The program for obtaining a horoscope in Python using API from rapidapi.com site.

Python horoscope The program allows you to get a horoscope for your zodiac sign and immediately translate it into almost any language. Step 1 The firs

Architect 0 Dec 25, 2021
Muzan-Discord-Nuker - A simple discord server nuker in python

Muzan-Discord-Nuker This is Just a simple discord server nuker in python. ✨ Feat

Afnan 3 May 14, 2022
Eva Maria Bot With Python

Eva Maria Bot Features Auto Filter Manual Filter IMDB Admin Commands Broadcast Index IMDB search Inline Search Random pics ids and User info Stats, Us

Aadhi 3 Jan 06, 2022
Mixcloud API wrapper for Python and Async IO

aiomixcloud Mixcloud API wrapper for Python and Async IO aiomixcloud is a wrapper library for the HTTP API of Mixcloud. It supports asynchronous opera

Aristotelis Mikropoulos 26 Dec 31, 2022
Discord bot ( discord.py ), uses pandas library from python for data-management.

Discord_bot A Best and the most easy-to-use Discord bot !! Some simple basic auto moderations, Chat functions. It includes a game similar to Casino, g

Jaitej 4 Aug 30, 2022
Graviti-python-sdk - Graviti Data Platform Python SDK

Graviti Python SDK Graviti Python SDK is a python library to access Graviti Data

Graviti 13 Dec 15, 2022
This is a small package to interact with the OpenLigaDB API.

OpenLigaDB This is a small package to interact with the OpenLigaDB API. Installation Run the following to install: pip install openligadb Usage from o

1 Dec 31, 2021
And now, for the first time, you can send alerts via action from ArcSight ESM Console to the TheHive when Correlation Rules are triggered.

ArcSight Integration with TheHive And now, for the first time, you can send alerts via action from ArcSight ESM Console to the TheHive when Correlatio

Amir Hossein Zargaran 3 Jan 19, 2022
A minimalistic, modern Discord bot for roles and polls using dropdowns

DropBot A minimalistic, modern Discord bot for roles and polls using dropdowns Made by ThatOneCalculator Technologies used Instructions Type /, and na

ModernBots 1 Jun 27, 2022
Video-Player - Telegram Music/ Video Streaming Bot Using Pytgcalls

Video Player 🔥 ᴢᴀɪᴅ ᴠᴄ ᴘʟᴀyᴇʀ ɪꜱ ᴀ ᴛᴇʟᴇɢʀᴀᴍ ᴘʀᴏᴊᴇᴄᴛ ʙᴀꜱᴇᴅ ᴏɴ ᴘʏʀᴏɢʀᴀᴍ ꜰᴏʀ ᴘʟᴀʏ

Zaid 16 Nov 30, 2022
An advanced Filter Bot with nearly unlimitted filters!

Unlimited Filter Bot ㅤㅤㅤㅤㅤㅤㅤ ㅤㅤㅤㅤㅤㅤㅤ An advanced Filter Bot with nearly unlimitted filters! Features Nearly unlimited filters Supports all type of fil

TroJanzHEX 445 Jan 03, 2023
Project to list all resources in an AWS account with tags.

AWS-ListAll Project to list all resources in an AWS account with tags. This script works on any system Get started: Install python3 and pip3 along wit

Connor Shubham Verlekar 3 Jan 30, 2022
This wrapper now has async support, its basically the same except it uses asyncio

This is a python wrapper for my api api_url = "https://api.dhravya.me/" This wrapper now has async support, its basically the same except it uses asyn

Dhravya Shah 5 Mar 10, 2022
A multi exploit instagram exploitation framework

Instagram Exploitation Framework About IEF Is an open source Instagram Exploitation Framework with various Exploits that could be used to mod your pro

Instagram Exploitation Framework - BirdSecurity 1 May 23, 2022
Braintree Python library

Braintree Python library The Braintree Python library provides integration access to the Braintree Gateway. TLS 1.2 required The Payment Card Industry

Braintree 230 Dec 18, 2022
Apps related to Odoo it's calendar features

calendar Apps related to Odoo it's calendar/appointments features: online_appointment_locations: allow setting an online URL per employee online_appoi

Yenthe Van Ginneken 3 Oct 27, 2022
Discord E-Store Bot

A delivery bot for Discord, works like Amazon where real users can pack & deliver orders in different servers!

Amit Pathak 2 Jan 28, 2022
vk.com API python wrapper

Python vk.com API wrapper This is a vk.com (the largest Russian social network) python API wrapper. The goal is to support all API methods (current an

Dmitry Voronin 371 Dec 29, 2022
A Discord bot that may save your day by predicting it.

Sage A Discord bot that may save your day by predicting it.

1 Nov 17, 2022
Gathers data and displays metrics related to climate change and resource depletion on a PowerBI report.

Apocalypse Status Dashboard Purpose Climate change and resource depletion are grave long-term dangers. The code in this repository will pull data from

Summer Is Here 1 Nov 12, 2021