AI Dungeon Catalog Archive Toolkit

Related tags

DownloaderAIDCAT
Overview

AI Dungeon Content Archive Toolkit (AID CAT)

AID CAT is a command-line utility that will allow you to download JSON backups of:

  1. Your private and published AI Dungeon scenarios, adventures, posts, bookmarks, worlds, friends, followers, and following.
  2. Any user's published scenarios, adventures, and posts, as well as their friends, followers, and following.
  3. Obfuscate all adventures' titles, descriptions, tags, memory, actions, undone actions, and world info with junk data.
    WARNING! Obfuscating adventures wipes all adventures on an account and CANNOT BE UNDONE!

Things AID CAT will NEVER do:

  1. Download another user's private content.
  2. Share your information with anyone else. All data transferred is strictly between your device and the aidungeon.io GraphQL backend.

Go check out KoboldAI!

Saved AI Dungeon games exported using AID CAT can now be imported and played with KoboldAI! KoboldAI is a browser front-end for playing with multiple local & remote AI models. KoboldAI supports interact with the AI models via Tensorflow and includes a easy-to-use automated install process. Currently supported models are GPT Neo 1.3B/2.7B, GPT-2 Med/Large/XL, Megatron (via InferKit API) as well as the ability to run your fine-tuned custom GPT-Neo (e.g., Neo-horni) and custom GPT-2 (e.g., CloverEdition) models.

Latest Release

2021-06-10: v0.7.0

Added the ability to disable Safe Mode
Hardened the obfuscation script against network instability
Optimized the obfuscation script to run faster
**WARNING! Obfuscating adventures wipes all adventures on an account and CANNOT BE UNDONE!**

Instructions below copied with edits from original script author's site (referenced files uploaded here for archival purposes):

The wAIfupocalypse is upon us

Let's save your stories!

Recently, Latitude has announced changes that will end AI cooms as we know them today. Going beyond just the explore page, employees may now ban you for content in your unpublished stories. For this reason, I have programmed a script that can automatically download all of the stories and scenarios in your account. Here are the instructions:

  1. Before you can use the script, you must download and install the Python 3 runtime. Make sure to select "Add Python 3.X to PATH" on the first page of the installer.
  2. The script needs your access token so it can access your private stories. The token will only be used for that purpose, and it will not be stored. See how to access it in Firefox or Chrome.
  3. Download the aidcat.py script here. Move it to where you want your stories saved and run it. (usually by double-clicking, or typing python aidcat.py or python3 aidcat.py in the command prompt while in the directory containing the script file).
  4. Enter your login token and press enter. While you wait, think back to the good times you had with your fictional friends.
  5. When the download is complete, a file called stories.json will be created containing all your stories.

I have also created a script that will turn that JSON file into HTML files so it's easier to read. Download it here and run it in the same folder as the JSON file. You might also want to get this stylesheet (put in same folder as html files) to make things look a little nicer.

For mobile users

You can run these scripts using Termux (Android) or Pythonista (iOS). Since it's difficult to access your login token on a mobile browser, here is a script to help you with that: login.py (1.1 KiB)

Termux Instructions

  1. pkg update && pkg install python && curl -O https://raw.githubusercontent.com/CuriousNekomimi/AIDCAT/main/aidcat.py && curl -O https://raw.githubusercontent.com/CuriousNekomimi/AIDCAT/main/login.py
  2. python login.py (To get your x-access-token)
  3. python aidcat.py (To run AIDCAT)

Changelog

2021-05-30: v0.6.9
Added the ability to obfuscate all adventures' title, description, tags, memory, actions, undone actions, and world info with junk data.
**WARNING! Obfuscating adventures wipes all adventures on an account and CANNOT BE UNDONE!**
Added an info page to sources documenting Latitude.
 
2021-05-11: v0.6.7
HUGE thanks to Eta for making these improvements and for refactoring the program to be object oriented:
- The phrase "any key" was changed to "Enter." (Actually waiting for any keypress is significantly more complicated).
- Option 1 of the auth menu previously said [1] Change your access access token, which was fixed.
- Added a missing error message on "Our Content" page.
- PEP 8 compliance:
  - The header docstring was changed to use """ instead of '''
  - Fixed whitespace around operators in code and method declarations
  - Fixed indentation in a couple places
  - Added whitespace between the # and the start of each comment
  - Added a newline at the end of the file
  - Added two lines of whitespace before and after function definitions
  - (Non-PEP 8) Changed menu choices to all be double quoted strings, for consistency
- Removed default values from actually-mandatory method parameters like save_json's content_type
- Separated the long as heck query strings from the rest of the code and moved them to the bottom of the file, just above if __name__ == '__main__', to improve readability
- Added a User class
  - This class encompasses the operations make_query (new), save_json, get_scenarios/subscenarios/adventures/posts/worlds/social, and get_saves (new, which calls all of the bookmarked operations together)
  - It holds the fields content_cache, and each of the query_s as instance variables (no longer global)
    - This means these can be maintained independently per user, which will help a lot with the batch download feature later on (it allows that to be multithreaded, for example)
- All of the menu code was changed to use User objects representing different users instead of passing around a username parameter
- The menu code was additionally refactored to cut down on duplicated code by a lot

2021-05-10: v0.5.7
Thanks to Eta for these suggestions:
UUID (access token) validation using python's built-in uuid library.
Cleaned up boolean checks.
Better file management. Saving files no longer risks a memory leak if the programmer forgets to call close().

2021-05-09: v0.5.6
Various improvements to the code in terms of efficiency and syntax, thanks Eta!

2021-05-09: v0.5.4
Changed aidcat.py to UTF-8 encoding. If you were seeing weird issues with screens, like corrupted text, this should fix that issue.
Thanks to Eta for the great suggestions for improving efficiency and helping with refactoring overall, including:
Menu headers are now all block quotes.
Reduced the number of escaped characters ( e.g. \' ).
More conventional print() statements.

Future Plans:
Proper GUID validation
HTML format export/converter. (Next priority)
Possibly add the ability to batch download all your friends' and followings' published content.
Ability to get the user's x-access-token from within the application. (Rather than digging around in the browser.)

2021-05-09: v0.5.2 (Completely new program written from scratch)
Command-line user interface with menus.
Download your scenarios, subscenarios, adventures, posts, and worlds (including official purchased worlds).
Download your saved scenarios, adventures, and posts (bookmarks).
Download a list of your friends, following, and followers.
Download other user's published scenarios, subscenarios, and posts.
Download a list of other user's friends, followers, and following.
Saved files tagged with ISO 8601 format times, so previous files aren't accidentally overwritten.
NOTE: Windows (Command prompt/PowerShell) doesn't display UTF-8 encoded characters. If someone has fancy UTF characters in their username, they will show up as empty boxes when pasted in, but the program will still work.

Future plans:
HTML format export/converter. (Next priority)
Possibly add the ability to batch download all your friends' and followings' published content.
Ability to get the user's x-access-token from within the application. (Rather than digging around in the browser.)

2021-05-02:
Added: Ability to save subscenarios (thanks to the original script author)!
Added: Ability to save bookmarks (thanks to the original script author)!
Added: Choice to save scenario bookmarks.
Added: Choice to save adventure bookmarks.

2021-05-02:
Added: Ability to choose whether or not to download scenarios, adventures, or worlds. Useful for users who have thousands of adventures and don't want to download them every time.
Added: More error catching and a reminder to report errors to https://github.com/CuriousNekomimi/AIDungeonRescue/issues.
Note: The original script author has released an update to download scenario options. I'll work on incorporating that code.

2021-05-02:
Added: Ability to save worlds!
Added: Ability to save x-access token for later use (token.txt).
Added: Scenarios, adventures, and worlds, now save to separate json files.
Added: Improved readability of saved scenario, adventure, and world json files with newlines and indents.
Added: Pause for user interaction after running so the user knows the program executed successfully.
Added: Notification that files saved successfully.
Added: Try/Except to main() for several methods.
Refactored: 'stories' to 'adventures'.
Comments
  • Downloading more than 50 actions

    Downloading more than 50 actions

    I ran this almost two weeks ago to export my stories, including one with over 15,000 actions (yes, I know). Everything was exported including that full story, resulting in a 3.88 MB file.

    Since you now have support for importing into KoboldAI, I figured I'd get the latest version of this tool and re-run it to download my AI Dungeon stories. This results in a much smaller file, only 293 KB. I checked and found that it now seems to cap out at the 50 most recent actions of the story.

    I still have the older version of this tool, so I ran it again to see if the change was on your end or AI Dungeon, and it downloaded the full 3.88 MB file again. So that tells me it's a change made to this tool.

    I'm guessing it's to keep things from getting to bloated before importing into KoboldAI or something, only keeping a certain number of relevant actions which would be potentially referenced by it? Is there an option somewhere now to download the full adventures again, or at least can you provide clarification on the reason for this?

    bug 
    opened by DagothRa 3
  • Tweaked and cleaned

    Tweaked and cleaned

    -Used requests sessions instead of urllib. -Put images in their own file. -Separated the UI from the program logic. -Created a module to transform .json to html (I used my tweaked version from Scriptanon's)

    I'm going to start making the automated tests.

    opened by Moist-Cat 2
  • Infinite Loop on Latitude corrupted backwards compatibility

    Infinite Loop on Latitude corrupted backwards compatibility

    Search the term "Latitude " (with the space), which will lead to an error message. It is inside a true while loop with a continue that triggers no advancement nor does it lead to a break, thus causing infinite looping.

    opened by YuliaS11 1
  • Tools to help migrate from AID to KAI

    Tools to help migrate from AID to KAI

    I created some simple tools to help integrate AIDCAT and KAI so people could migrate from AID faster. The scripts I made interpret AIDCAT's output in a "scenario explorer" that lets the person select their scenario, type the placeholders just like in AID and then save the generated story into a file ready to be used in KoboldAI. Same for my own scenarios (it has its own script) and another tool that converts AIDCAT's extracted stories from AID into something readable by KoboldAI.

    I plan to join these three tools into one, but for now they're separated. This might be useful for people using AIDCAT. Here's the link: https://github.com/thaalesalves/aid-scripts/tree/main/other

    opened by thaalesalves 1
  • Saving bookmarked scenarios fails if any sub-scenario fetch returns None.

    Saving bookmarked scenarios fails if any sub-scenario fetch returns None.

    For reasons unknown to me, certain sub-scenarios in my bookmarks return {Scenario: None}. This satisfies the if 'scenario' in result['data'] check but causes an exception when the script tries to progress and halts.

    I was able to do a quick fix locally by changing if 'data' in result and 'scenario' in result['data']: to if 'data' in result and 'scenario' in result['data'] and result['data']['scenario'] != None:

    opened by throwaway47323674 1
  • genhtml_edit.py throws an error when trying to open a file

    genhtml_edit.py throws an error when trying to open a file

    Welcome to the AIDCAT HTML Generator! First we'll ask for your Adventure file, press Enter to continue... Traceback (most recent call last): File "C:\Users\Me\Documents\AIDungeonRescue-main\aidcat\genhtml_edit.py", line 51, in infile = json.load(file) File "C:\Users\Me\AppData\Local\Programs\Python\Python37\lib\json_init_.py", line 293, in load return loads(fp.read(), UnicodeDecodeError: 'cp932' codec can't decode byte 0x9d in position 119195: illegal multibyte sequence

    opened by cjk137 1
  • Improved Token and Error Handling

    Improved Token and Error Handling

    This pull request improves token handling, adding autocorrection for missing hyphens and server validation to check if the token is associated with an account. It will also now display the account name associated with an entered token. Error handling is also improved, and a keyboard interrupt can now properly exit out of the program (without any error text, either).

    opened by Eta0 0
  • Broken with 'Walls Approach' update

    Broken with 'Walls Approach' update

    https://latitude.io/blog/the-walls-approach

    It seems they're now encrypting user stories. When I try to download them with AIDCAT, I get 400 Bad Request. I might have missed my chance, unless this script can be updated to decrypt the stories.

    opened by ImportTaste 16
Releases(v0.7.0)
  • v.0.6.9(May 30, 2021)

    Added the ability to obfuscate all adventures' title, description, tags, memory, actions, undone actions, and world info with junk data. WARNING: Obfuscating adventures wipes all adventures on an account and CANNOT BE UNDONE! Added an info page to sources documenting Latitude.

    Source code(tar.gz)
    Source code(zip)
  • v0.6.7(May 17, 2021)

  • v0.6.6(May 17, 2021)

    Descriptive exceptions and traceback for HTTP connections errors and file saving. Possible fix for Mac devices not saving files to the current directory.

    Source code(tar.gz)
    Source code(zip)
  • v0.6.5(May 16, 2021)

    Re-refactored actionWindow back to actions . It turns out that actionWindow only returns the last 50 actions. I sincerely apologize for any inconvenience this may have caused!

    Source code(tar.gz)
    Source code(zip)
  • v0.6.2_html_utf8(May 15, 2021)

  • v0.6.2_html(May 14, 2021)

  • v0.6.2(May 13, 2021)

    Eta0: This pull request improves token handling, adding autocorrection for missing hyphens and server validation to check if the token is associated with an account. It will also now display the account name associated with an entered token. Error handling is also improved, and a keyboard interrupt can now properly exit out of the program (without any error text, either).

    Source code(tar.gz)
    Source code(zip)
  • v0.6.0(May 11, 2021)

    HUGE thanks to Eta0 for making these improvements and for refactoring the program to be object oriented:

    • The phrase "any key" was changed to "Enter." (Actually waiting for any keypress is significantly more complicated).
    • Option 1 of the auth menu previously said [1] Change your access access token, which was fixed.
    • Added a missing error message on "Our Content" page.
    • PEP 8 compliance:
      • The header docstring was changed to use """ instead of '''
      • Fixed whitespace around operators in code and method declarations
      • Fixed indentation in a couple places
      • Added whitespace between the # and the start of each comment
      • Added a newline at the end of the file
      • Added two lines of whitespace before and after function definitions
      • (Non-PEP 8) Changed menu choices to all be double quoted strings, for consistency
    • Removed default values from actually-mandatory method parameters like save_json's content_type
    • Separated the long as heck query strings from the rest of the code and moved them to the bottom of the file, just above if name == 'main', to improve readability
    • Added a User class
      • This class encompasses the operations make_query (new), save_json, get_scenarios/subscenarios/adventures/posts/worlds/social, and get_saves (new, which calls all of the bookmarked operations together)
      • It holds the fields content_cache, and each of the query_s as instance variables (no longer global)
        • This means these can be maintained independently per user, which will help a lot with the batch download feature later on (it allows that to be multithreaded, for example)
    • All of the menu code was changed to use User objects representing different users instead of passing around a username parameter
    • The menu code was additionally refactored to cut down on duplicated code by a lot
    Source code(tar.gz)
    Source code(zip)
  • v0.5.7(May 10, 2021)

Owner
Mimi
Mimi
VK sticker downloader with python

VK Sticker Downloader This repository is used to automate download file from VK Sticker How to use Execute the file ./downloader.py Writedown full url

Hartawan Bahari M. 1 Dec 29, 2021
Download videos and audio with a graphical interface in python

Youtube-Downloader Download videos and audio with a graphical interface in python Windows To run windows using Command Prompt python main.py linux To

2 Jan 07, 2022
PyQt5 simple files , youtube videos and youtube playlist downloader

PyQt5 simple files , youtube videos and youtube playlist downloader

AmirHossein Mohammadi 7 Jul 25, 2022
Automatically download multiple papers by keywords in CVPR

Automatically download multiple papers by keywords in CVPR

46 Jun 08, 2022
Used Insta Loader to download high quality images from instagram account

Insta Dp Downloader Project Description: In this project, I have used "Insta Loader" to download high quality images from instagram account. You only

Hassan Shahzad 3 Oct 31, 2022
Source code of paper: "HRegNet: A Hierarchical Network for Efficient and Accurate Outdoor LiDAR Point Cloud Registration".

HRegNet: A Hierarchical Network for Efficient and Accurate Outdoor LiDAR Point Cloud Registration Environments The code mainly requires the following

Intelligent Sensing, Perception and Computing Group 3 Oct 06, 2022
Make YouTube videos tasks in Todoist faster and time efficient!

Youtubist Basically fork of yt-dlp python module to my needs. You can paste playlist or channel link on the YouTube. It will automatically format to s

Konrad Konieczny 1 Dec 04, 2022
A cross-platform python based utility to download courses from udemy for personal offline use.

udemy-dl A cross-platform python based utility to download courses from udemy for personal offline use. Warning Udemy has started to encrypt many of t

Nasir Khan 4.6k Dec 31, 2022
Youtube video downloader and info extractor for python.

tube_dl Tube_dl is a Simple Youtube video downloader for Python. A Modular approach to bypass and download Youtube Videos and Playlist from Youtube us

Shekhar Chander 16 Jul 09, 2022
Downloads state flags from wikipedia for states/regions from all countries

world-state-flags Downloads state flags from wikipedia for states/regions from all countries This data is NOT curated Uses https://github.com/dr5hn/co

João Ribeiro Bezerra 2 Dec 15, 2022
Bulk Downloader for Reddit

saveddit is a bulk media downloader for reddit pip3 install saveddit Setting up authorization Register an application with Reddit Write down your clie

Pranav 136 Jan 03, 2023
A simple Python +3.x script to download videos from Facebook.

Facebook Video Downloader A simple Python +3.x script to download videos from Facebook posts

Kerolos Atef Saber 1 Dec 03, 2021
Youtube-music - Youtube music with python

youtube-music fzf on https://github.com/junegunn/fzf python3 ytb.py [no/yes] yes

direskyfer 0 Feb 03, 2022
SubGrab is a utility that allows you to automate subtitles downloading for your media files.

SubGrab - Command-line Subtitles Downloader: A utility which provides an ease for automating media i.e., Movies, TV-Series subtitle scraping from mult

Rafay 106 Dec 17, 2022
Search the gallerys by tag and download pictures to the local

booruDownloader Search the gallerys by tag and download pictures to the local

6 Jun 30, 2022
Code to scrape , download and upload to youtube daily

Youtube_Automated_Channel Code to scrape , download and upload to youtube daily INSTRUCTIONS Download the Github Repository Download and install Pytho

Atsiksdong 2 Dec 19, 2021
Python based YouTube video Downloader GUI Application.

Youtube video Downloader Python based Youtube video Downloader GUI Application. Installation Python Dependencies Import pytube pip install pytube Im

Naem Azam 1 Jan 03, 2022
Fetch papers and metadata.

Fetch PubMed Central for open-access papers as well as Sci-Hub

4 Oct 31, 2022
This is a python based web scraping bot for windows to download all ACCEPTED submissions of any user on Codeforces

CODEFORCES DOWNLOADER This is a python based web scraping bot for windows to download all ACCEPTED submissions of any user on Codeforces Requirements

Mohak 6 Dec 29, 2022
A Python package for downloading / archiving all available episodes from a podcast RSS feed.

allcasts 📻 🗃 A Python package for downloading all available episodes from a podcast RSS feed. Useful for making private archives of your favourite p

Lewis Gentle 5 Nov 20, 2022