Twayback: Downloading deleted Tweets from the Wayback Machine, made easy

Overview

Twayback: Downloading deleted Tweets from the Wayback Machine, made easy

forthebadge made-with-python

windows python

screenshot

Finding and downloading deleted Tweets takes a lot of time. Thankfully, with this tool, it becomes a piece of cake! πŸŽ‚

Twayback is a portmanteau of Twitter and the Wayback Machine. Enter your desired Twitter username, and let Twayback do the rest!

Requirements

  • Python 3
  • waybackpack
    • Install: pip install waybackpack or pip3 install waybackpack
    • Link to repo

Features

  • Can download some or all of a user's archived deleted Tweets.
  • Allows custom time range to narrow search for deleted Tweets archived between two dates.
  • Differentiates between accounts that are active, suspended, or don't/no longer exist.
  • Lets you know if a target handle's archived Tweets have been excluded from the Wayback Machine.

Usage

twayback -u USERNAME [OPTIONS]
Example: twayback -u jack

-u, --username        Specify target user's Twitter handle
-from, --fromdate     Narrow search for deleted Tweets *archived* on and after this date
                      (can be combined with -to)
                      (format YYMMDD)
-to, --todate         Narrow search for deleted Tweets *archived* on and before this date
                      (can be combined with -from)
                      (format YYMMDD)

Installation

For Windows only

  1. Download the latest EXE file.
  2. Launch Command Prompt in the EXE file's directory.
  3. Run the command twayback -u USERNAME (Replace USERNAME with your target handle).

For Windows, Linux, and macOS

  1. Download the latest ZIP file.
  2. Extract ZIP file to a directory of your choice.
  3. Open terminal in that directory.
  4. Run the command pip install -r requirements.txt.
  5. Run the command twayback -u USERNAME (Replace USERNAME with your target handle).

For more information, check out the Usage section above.

Things to keep in mind

  • Quality of the HTML files depends on how the Wayback Machine saved them. Some are better than others.
  • This tool is best for text. You might have some luck with photos. You cannot download videos.
  • By definition, if an account is suspended or no longer exists, all their Tweets would be considered deleted.
  • Custom date range is not about when Tweets were made, but rather when they were archived. For example, a Tweet from 2011 may have been archived today.

Future plans

  • GUI. This is a biggie. I don't know shit about Python, let alone GUI. But I'm hoping I can design one using Tkinter Designer. But I don't know how I can link actions to buttons and shit like that, that stuff is super foreign to me, so any help is appreciated, it would mean so much.

Plenty of thanks to jsvine for his amazing work on waybackpack. Without it, this tool cannot work nearly as well.

I hope you enjoy my little script. Please use it for good. Whatever you are, be a good one.

Comments
  • AttributeError: 'NoneType' object has no attribute 'getText'

    AttributeError: 'NoneType' object has no attribute 'getText'

    I got two errors:

    1. The first error occured after grabbing links from wayback machine UnboundLocalError: local variable 'wayback_id' referenced before assignment

    2. Second, the error occured when I typed 'text' or 'both' AttributeError: 'NoneType' object has no attribute 'getText'

    please help, thanks.

    bug good first issue 
    opened by adrn-mm 9
  • Error in the process

    Error in the process

    Hello,

    Thanks for the app which helps save a lot of time. I have tried with several users and in various ways and it does not work. I pass the screenshots of the errors.

    Thank you very much.

    image image

    opened by barripdmx 8
  • Twayback Partial Re-Write

    Twayback Partial Re-Write

    Hello,

    We recently came across your project, and thought we could contribute by re-writing some parts of the code. An effort was made to keep the logic and structure of the code the same.

    By our metrics, we have achieved a speed-up of around 20-30% for accounts with over 1000 tweets. We don't have any actual tests written for the script however, so we have attached it here for review.

    A substantial change was moving from Selenium to Playwright. Users need to run 'playwright install' to install the playwright browsers before running the script.

    Another smaller, but notable change is that due to the nature of the as_completed function, it is not possible to show any indicator of progress during the gathering of statuses, and the order in which the tweets are returned is scrambled due to the script processing website information based on what web request finishes first.

    If the style & format of the re-write is acceptable, and the changes proposed are not considered to be critical, we will proceed with creating a pull request.

    twayback2.txt

    opened by AccentuSoft 5
  • Parsing fails when encountering text in cyrillic

    Parsing fails when encountering text in cyrillic

    Hi, when I try to search for deleted tweets from an account that uses cyrillic, the process fails with the following exception:

    Parsing text...: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 4/4 [00:07<00:00, 2.00s/it] Traceback (most recent call last): File "twayback.py", line 184, in File "encodings\cp1250.py", line 19, in encode UnicodeEncodeError: 'charmap' codec can't encode character '\u0430' in position 0: character maps to [20476] Failed to execute script 'twayback' due to unhandled exception!

    Anything that could be done on my end? Thanks,

    opened by Traut89now 5
  • KeyError: 'closest' when parsing accounts with larger number of tweets.

    KeyError: 'closest' when parsing accounts with larger number of tweets.

    Hi,

    after the latest update, the tool seems to be working perfectly when scraping accounts with lower number of tweets archived, but when I tried accounts with 1000+ tweets archived, the process failed with: Traceback (most recent call last): File "C:\15tway\twayback B\twayback.py", line 112, in wayback_url = (jsonResponse['archived_snapshots']['closest']['url']) KeyError: 'closest'

    as also seen below on the screenshots.

    image

    opened by Traut89now 4
  • No deleted tweets have been found

    No deleted tweets have been found

    Hi! Does anyone else, after they pull one search of an user, get an "No deleted tweets have been found" when they try to look for another one? I even try with the same one I did first and it pulls the same thing. I even close and open again the terminal but nothing, got the same issue. Was anyone run into this?

    opened by sofiemmc 3
  • Cleaned up code a bit

    Cleaned up code a bit

    No changes to functionality. I went through and added some more comments, renamed variables to be more descriptive and combined some redundancy with list creation

    opened by humandecoded 2
  • Feature proxy

    Feature proxy

    to avoid rate limiting for large groups of tweets added the ability to use proxies with our GET requests. User will need to provide a list of proxy URLs for the script to randomly pull from

    opened by humandecoded 1
  • Twitter Rate Limiting

    Twitter Rate Limiting

    It looks like twitter is throwing up 429 (too many requests) after the first 900 or so hits. According to their API documentation they limit to around 900 hits per 15 minutes. Although we are not using the API, it seems they are putting similar limiting in place per IP.

    I've begun working on a branch that lets users plug in a list of proxies the script will rotate through to avoid 429. This does make the tool less approachable but at this point it's not working on large groups of tweets. Will keep everyone updated

    enhancement 
    opened by humandecoded 1
  • Adding more Detailed information

    Adding more Detailed information

    I am submitting this PR so that newbie cyber people & OSINT Analysts with no python experience can better understand installation and usage.

    • There is no information about using the command git clone in order to clone the repository
    • No information in regards to changing directory | cd
    • No information in regards to the command pip3 install -r requirements.txt
    • No information in regards to the command python3 twayback.py -u username (if we put ourselves in the shoes of anyone without any python knowledge, they will just write in their terminal twayback.py which won't be of any use, it's typically something I would have done in the past as when reading installation & usage instructions I would just copy paste without trying to understand. 🀦

    Hope this helps a little, awesome tool & thanks for putting it out there !

    opened by C3n7ral051nt4g3ncy 1
  • UPDATE requirements.txt

    UPDATE requirements.txt

    Module aiohttp is required

    Traceback (most recent call last):
      File "/home/gdhindii/twayback/twayback.py", line 18, in <module>
        from aiohttp import ClientSession, TCPConnector
    ModuleNotFoundError: No module named 'aiohttp'
    
    opened by sam5epi0l 1
  • Proxy File List

    Proxy File List

    I am not sure this is an issue or just lack of knowledge from my part, I rather suspect it is the latter, however, I am attempting to recover tweets from an account that has a couple of thousand deleted tweets. I eventually encounter a rate limit error as is to be expected. I saw the way to handle requests like these is to use a proxy file. Now I have a .txt file that I added inside the twayback folder with a list of proxies and formatted according to the guidelines:

    url:port url:port url:port

    and so on. I keep getting error for each proxy server so Im not sure if Im doing this right and what the issue could be. It will stay stuck at 0%, switch between proxies until it eventually just gives up with a "" error.

    My request looks like this:

    python3 twayback.py --proxy-file proxyfiles.txt -u USERNAME

    Am I doing this correctly?

    opened by DirkGaston 2
  • it doesn't work anymore

    it doesn't work anymore

    once I launch the app it closes itself, it was working fine before the last update I tried downloading a pervious version and the same thing happened I tried it on 2 devices and it didn't work on any of them

    opened by unaufhaltbar 4
Releases(10-16-22)
Owner
He/Him. Good-for-nothing GitHubber.
Itchio Downloader Tool with python

Itchio Downloader Tool Install pip install git+https://github.com/emersont1/itchio Download All Games in library from account python -m itchio.downloa

Peter Taylor 69 Dec 05, 2022
A script that downloads YouTube videos/audio

YouTube-Downloader A script that downloads YouTube videos/audio from youtube. Usage Download the script by executing the following in your terminal :

Debayan Sarkar 2 Jan 04, 2022
This is a python based web scraping bot for windows to download all ACCEPTED submissions of any user on Codeforces

CODEFORCES DOWNLOADER This is a python based web scraping bot for windows to download all ACCEPTED submissions of any user on Codeforces Requirements

Mohak 6 Dec 29, 2022
Download YOUR files, documents from vk.

vk-documents-downloader ΠšΠ°Ρ€ΠΎΡ‡ эта симпл хСрня ΠΊΠ°Ρ‡Π°Π΅Ρ‚ всС Π’ΠΠ¨Π˜ Π΄ΠΎΠΊΡƒΠΌΠ΅Π½Ρ‚Ρ‹ с Π²ΠΊ. Или я Π΅Π±Π»Π°Π½, Π½ΠΎ Π² Π³Ρ… ΠΈ Ρ‚ΠΌΠ± Π³ΡƒΠ³Π»Π΅ я ΠΏΠΎΠ΄ΠΎΠ±Π½ΠΎΠ³ΠΎ Π½Π΅ нашСл. py main.py Login:

4 Jun 10, 2022
A cross-platform python based utility to download courses from udemy for personal offline use.

udemy-dl A cross-platform python based utility to download courses from udemy for personal offline use. Warning Udemy has started to encrypt many of t

Nasir Khan 4.6k Dec 31, 2022
ImageScraper is a cross-platform tool for downloading a specified count from xkcd, Astronomy Picture of the Day and Existential Comics

ImageScraper The ImageScraper is a cross-platform tool for downloading a specified count from xkcd, Astronomy Picture of the Day and Existential Comic

1amnobody 1 Jan 25, 2022
😷 Dowload dos documentos da CPI da Pandemia

A CPI da Pandemia recebeu milhares de documentos pΓΊblicos, todos disponibilizados no site do Senado Federal.

Eduardo Cuducos 98 Sep 23, 2022
A cli tool to download purchased products from the DLsite.

dlsite-downloader A cli tool to download purchased products from the DLsite. How can I use? This program runs with configurations defined at settings.

AcrylicShrimp 9 Dec 23, 2022
Youtube videos and channels scraper python wrapper!

YouTubeCrawle Wrapper for python Why This wrapper? This is wrapper is not limited to videos only it can scrape both channel and videos seperately ;D

Kei 16 Aug 08, 2022
Application Updater using an download link

Application-Updater This tool will update your app using an storage link

ExtremeDev 1 Dec 20, 2021
πŸ”₯ A Bot To Telegram For Download High Qulity Videos & Songs From Youtube

πŸ”₯ A Bot To Telegram For Download High Qulity Videos & Songs From Youtube πŸŽ— Fast And Free Bot No Need To Pay βœ… By SL-Alpha-X-Team ⚑

Official Alpha-X-Team Account 7 Aug 31, 2022
Python script designed to search and fetch direct download links from nxbrew.com

SwitchGamesDownloader Only for windows nxbrew.com is a website, accessible only using a proxy, where the majority of games for the Nintendo Switch are

Backend 91 Dec 28, 2022
YT-Downloader is a Tool to download youtube video.

YT-Downloader YT-Downloader is a Tool to download youtube video.If you are looking for a simple video downloader tool Than This YT-Downloader may be u

Pradip Thapa 7 May 11, 2022
Pantheon - The fastest YouTube downloader.

A Youtube downloader written in Python3, using HTTP requests and an API.

Billy 38 Nov 21, 2022
Download all games from a public Itch.io Game Jam

Itch Jam Downloader Downloads all games from a public Itch.io Game Jam. What you'll need: Python 3.8+ pip install -r requirements.txt For site mirrori

Dragoon Aethis 19 Dec 07, 2022
A python scripts that downloads doujin from nhentai without having an account

nhentai-downloader a python scripts that downloads doujin from https://nhentai.net without having an account. Usage Needs Python 3^ Linux pip3 install

Earl Sabalo 4 Jun 13, 2022
Fetch papers and metadata.

Fetch PubMed Central for open-access papers as well as Sci-Hub

4 Oct 31, 2022
A Python script that allows you to download all of an anime's episodes at once.

BitAnime A Python script that allows you to download all of an anime's episodes at once. Β· Download executable version Β· About BitAnime BitAnime is a

sh1nobu 17 Aug 10, 2022
Download clips from youtube videos with a few clicks and a GUI!

YouClip v2.0.0 Table Of Contents: What Is YouClip Installation Usage Stuff To Fix Changelog What Is YouClip? ! IMPORTANT: The source files are a total

ador 2 Oct 05, 2021
An Inline Telegram bot that can download YouTube videos with permanent thumbnail support

Tube (YouTube Downloader) An Inline Telegram bot that can download YouTube videos with permanent thumbnail support About Bot need to be in Inline Mode

Renjith Mangal 30 Dec 14, 2022