ShadowClone allows you to distribute your long running tasks dynamically across thousands of serverless functions and gives you the results within seconds where it would have taken hours to complete

Overview

ShadowClone

ShadowClone allows you to distribute your long running tasks dynamically across thousands of serverless functions and gives you the results within seconds where it would have taken hours to complete.

You can make full use of the Free Tiers provided by cloud providers and supercharge your mundane cli tools with shadow clone jutsu (Naruto style)!

Installation

Please visit the wiki for installation and intial configuration instructions

Usage

⚡ python shadowclone.py -h
usage: shadowclone.py [-h] -i INPUT [-s SPLITNUM] [-o OUTPUT] -c COMMAND

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
  -s SPLITNUM, --split SPLITNUM
                        Number of lines per chunk of file
  -o OUTPUT, --output OUTPUT
  -c COMMAND, --command COMMAND
                        command to execute

How it works

We create a container image during the initial setup and register it as a runtime for our function in AWS/GCP/Azure whatever. When you execute ShadowClone on your computer, instances of that container are activated automatically and are only active for the duration of its execution. How many instances to activate is dynamically decided at runtime depending on the size of the input file provided and the split factor. The input is then split into chunks and equally distributed between all the instances to execute in parallel. For example, if your input file has 10,000 lines and you set the split factor to 100 lines, then it will be split into 100 chunks of 100 lines each and 100 instances will be run in parallel!

Features

  • Extremely fast
  • No need to maintain a VPS (or a fleet of it :))
  • Costs almost nothing per month
    • Compatible with free tiers of most cloud services
  • Cloud agnostic
    • Same script works with AWS, GCP, Azure etc.
  • Supports upto 1000 parallel invocations
  • Dynamically decide the number of invocations
  • Run any tool in parallel on the cloud
  • Pipe output to other tools

Comparison

This tool was inspired by the awesome Axiom and Fleex projects and goes beyond the concept of VPS for running the tools by using serverless functions and containers.

Features Axiom/Fleex ShadowClone
Instances 10-100s* 1000s
Cost Per instance/per minute Mostly Free**
Startup Time 4-5 minutes 2-3 seconds
Idle Cost $++ Free
On Demand Scalability No

*Most cloud providers do not allow spinning up too many instances by default, so you are limited to around 10-15 instances at max. You have to make a request to the support to increase this number.

** AWS & Azure allow 1 million invocations per month for free. Google allows 2 million invocations per month for free. You will be charged only if you go above these limits

Demo

DNS Bruteforcing using a 43mb file - 34 seconds

asciicast

Running httpx on 94K subdomains in 1 min

asciicast

References

Lithops documentation

Free Tiers

Cloud Provider Free Allowance Link
Google Functions 2 Million invocations, 400,000 GB-seconds per month Google Cloud Free Program
AWS Lambda 1 Million invocations, Up to 3.2 million seconds of compute time per month Free Cloud Computing Services - AWS Free Tier
Azure Functions 1 Million invocations Microsoft Azure Free Services

Obviously, you can make any number of function invocations per month. The table above only shows how many invocations are free.

Similar Tools

Disclaimer

This tool is designed as a proof-of-concept implementation and it's usage is intended for educational purpose only. Usage for attacking targets without prior mutual consent is illegal. It's the end user's responsibility to obey all applicable local, state and federal laws. Developers assume no liability and are not responsible for any misuse or damage caused by this program.

A Wrapper for ScarletAPI

ScarletAPI A Wrapper for ScarletAPI still a work in progress Docs these are the

Amashi 0 Mar 24, 2022
Autodrive is designed to make it as easy as possible to interact with the Google Drive and Sheets APIs via Python

Autodrive Autodrive is designed to make it as easy as possible to interact with the Google Drive and Sheets APIs via Python. It is especially designed

Chris Larabee 1 Oct 02, 2021
A complete Python application to automatize the process of uploading files to Amazon S3

Upload files or folders (even with subfolders) to Amazon S3 in a totally automatized way taking advantage of: Amazon S3 Multipart Upload: The uploaded

Pol Alzina 1 Nov 20, 2021
Python API Client for Close

Close API A convenient Python wrapper for the Close API. API docs: http://developer.close.com Support: Close 56 Nov 30, 2022

A Discord bot to easily and quickly format your JSON data

Invite PrettyJSON to your Discord server Table of contents About the project What is JSON? What is pretty printing? How to use Input options Command I

Sem 4 Jan 24, 2022
A discord token grabber made in Python 3

Discord Token Grabber A Discord token grabber written in Python 3. This version of the grabber only supports Windows. Features Transfers via Discord w

Mega145 4 Aug 04, 2022
Aula-API - a school system widely used in Denmark, as you can see and read about in the python file

Information : Hello, thank you for reading this first of all. This is a Aula-API

Binary.club 2 May 28, 2022
Group Chat Spammer For Discord

Group Chat Spammer For Discord Free and public gc spammer

Dreamy 20 Dec 27, 2022
An advanced telegram country information finder bot.

Country-Info-Bot-V2 An advanced telegram country information finder bot Made with Python3 (C) @FayasNoushad Copyright permission under MIT License Lic

Fayas Noushad 16 Nov 12, 2022
Remedy when Amazon ECR is not running basic scans for container CVEs.

Welcome to your CDK Python project! This is a blank project for Python development with CDK. The cdk.json file tells the CDK Toolkit how to execute yo

4n6ir 4 Nov 05, 2022
Replacement for the default Dark Sky Home Assistant integration using Pirate Weather

Pirate Weather Integrations This integration is designed to replace the default Dark Sky integration in Home Assistant with a slightly modified, but f

Alexander Rey 129 Jan 06, 2023
Bendford analysis of Ethereum transaction

Bendford analysis of Ethereum transaction The python script script.py extract from already downloaded archive file the ethereum transaction. The value

sleepy ramen 2 Dec 18, 2021
Nft-maker - Create your own NFT!

nft-maker How to If you're going to use this program, change the pictures in the "images" folder. All images must be of the same resolution and size.

Georgii Arakelian 4 Mar 13, 2022
Project for the discipline of Visual Data Analysis at EMAp FGV.

Analysis of the dissemination of fake news about COVID-19 on Twitter This project was the final work for the discipline of Visual Data Analysis of the

Giovani Valdrighi 2 Jan 17, 2022
Desktop Backup Client for Borg

Vorta Backup Client Vorta is a backup client for macOS and Linux desktops. It integrates the mighty BorgBackup with your desktop environment to protec

BorgBase.com 1.5k Jan 03, 2023
Telegram bot to host python bots

Host-Bot Setup the api Upload the flask api on your host #its not important to do #i used it just for simple captcha system + save ids on your host!

Plugin 15 Feb 11, 2022
Raphtory-client - The python client for the Raphtory project

Raphtory Client This is the python client for the Raphtory project Install via p

Raphtory 5 Apr 28, 2022
Telegram Music Bot for YouTube/SoundCloud/Mixcloud

Telegram Music Bot Telegram Music Bot for YouTube/SoundCloud/Mixcloud This bot downloads and sends the audio when someone send a YouTube/SoundCloud/Mi

Calls Music 76 Jan 02, 2023
Non official, but friendly QvaPay library for the Python language.

Python SDK for the QvaPay API Non official, but friendly QvaPay library for the Python language. Setup You can install this package by using the pip t

Carlos Lugones 17 Nov 25, 2022
Python wrapper for eBay API

python-ebay - Python Wrapper for eBay API This project intends to create a simple python wrapper around eBay APIs. Development and Download Sites The

Roopesh 99 Nov 16, 2022