A Python package that can be used to download post and comment data from Reddit.

Related tags

Third-party APIs Wrappersreddit-data-collector
Overview

Reddit Data Collector

Reddit Data Collector is a Python package that allows a user to collect post and comment data from Reddit. It is built on top of the Python module PRAW, which stands for "The Python Reddit API Wrapper". It aims to make it very simple for a user to collect data from Reddit for further analysis (e.g. Natural Language Processing), without having to learn the inner workings of PRAW or the Reddit API.

The main functionalities provided by the package currently include:

  1. Ability to collect a sample of post data and comment data from Reddit by simply providing the subreddit names that you wish to collect data from.

  2. Ability to convert that data into a pandas DataFrame in order to inspect it and save it for further use.

  3. Ability to seamlessly update an existing .csv file that contains some sample data collected with the package in the past, with some new sample data that is also collected with the package.

It is currently maintained by Nico Van den Hooff.

Installation

Dependencies

Reddit Data Collector requires Python and:

  • pandas (>=1.3.5)
  • praw (>=7.5.0)
  • tqdm (>=4.62.3)

User installation

The recommended way to install Reddit Data Collector is using pip:

pip install reddit-data-collector

How to Use Reddit Data Collector

Please see the examples directory for step by step instructions on how to use Reddit Data Collector.

Development

Important links

Source code

You can check the latest sources with the command:

git clone https://github.com/nicovandenhooff/reddit-data-collector.git

Contributing

To learn more about making a contribution to Reddit Data Collector, please see the contributing file.

Potential Ideas for Contribution

  • Add ability to collect images from Reddit posts that contain them.
  • Add author information to post and comment data, currently the Reddit API is inconsistent with suspended and deleted author data, so this functionality has not been built in yet.

Testing

After installation, you can launch the test suite, which is contained in the tests/tests.py. Note that you will have to have pytest >= 6.2.5 installed. You can launch the test suite by following these steps from the projects root directory:

  1. Open up tests.py with the following command:
open tests/tests.py

Comment out lines 24 to 30. Change the values in DataCollector() in line 32 to your Reddit credentials.

  1. Run the following command:
pytest tests/test.py

Project History

The project was started in January 2022 by Nico Van den Hooff as a side project while he was completing the UBC Master of Data Science Project. Nico wanted to obtain a sample of posts and comments from Reddit, but noticed that while PRAW existed and provided seamless access to Reddit's API, there was no package available that allowed for a simple method to collect this data.

Inspiration

Certain sections of this README file was inspired by the scikit-learn README.

You might also like...
Auto Join: A GitHub action script to automatically invite everyone to the organization who comment at the issue page.

Auto Invite To Org By Issue Comment A GitHub action script to automatically invite everyone to the organization who comment at the issue page. What is

Max Base 6 Jun 8, 2022
Auto Liker, Auto Reaction, Auto Comment, Auto Follower Tool. RajeLiker Credit Hacker.
Auto Liker, Auto Reaction, Auto Comment, Auto Follower Tool. RajeLiker Credit Hacker.

Auto Liker, Auto Reaction, Auto Comment, Auto Follower Tool. RajeLiker Credit Hacker. Unlimited RajeLiker Credit Hack. Thanks To RajeLiker.

Md. Mehedi Hasan 32 Dec 28, 2022
A simple Discord bot that can fetch definitions and post them in chat.
A simple Discord bot that can fetch definitions and post them in chat.

A simple Discord bot that can fetch definitions and post them in chat. If you are connected to a voice channel, the bot will also read out the definition to you.

Tycho Bellers 4 Sep 29, 2022
A simple fun discord bot using discord.py that can post memes

A simple fun discord bot using discord.py * * Commands $commands - to see all commands $meme - for a random meme from the internet $cry - to make the

Dice Flip 2 Dec 20, 2021
One version package to rule them all, One version package to find them, One version package to bring them all, and in the darkness bind them.

AwesomeVersion One version package to rule them all, One version package to find them, One version package to bring them all, and in the darkness bind

Joakim Sรธrensen 39 Dec 31, 2022
A simple script & container to pull COVID data from covidlive.com.au and post a summary to a slack channel
A simple script & container to pull COVID data from covidlive.com.au and post a summary to a slack channel

CovidLive AU Summary Slackbot This bot is a very simple slackbot that pulls data, summarises and posts up to date AU COVID stats to a provided slack c

James 3 Dec 18, 2021
Track live sentiment for stocks from Reddit and Twitter and identify growing stocks
Track live sentiment for stocks from Reddit and Twitter and identify growing stocks

Market Sentiment About This repository can mainly be used for two things. a. Tracking the live sentiment of stocks from Reddit and Twitter b. Tracking

Market Sentiment 345 Dec 17, 2022
A reddit.com bot that will return reference links from official python documentation site for the standard library.

Python Docs Bot A reddit.com bot that will return documentation links for the library and language reference sections of the python docs website. The

Trevor Miller 2 Sep 14, 2021
A Python bot that uses the Reddit API to send users inspiring messages.

AnonBot By Edric Antoine A Python bot that uses the Reddit API to send users inspiring messages. When a message includes 'What would Anon do?', the bo

null 1 Jan 5, 2022
Releases(v1.1.0)

  • v1.1.0(Mar 14, 2022)

    [1.1.0] - 2022-03-13

    Changed

    • Changed get_data in reddit_data_collector.py to return pandas DataFrame by default
    • Updated tests for the above
    Source code(tar.gz)
    Source code(zip)

  • v1.0.2(Jan 15, 2022)

    [1.0.2] - 2022-01-14

    Fixed

    • Updated _check_subreddit_exists in reddit_data_collector.py to check both names as .lower()
    • Updated tests for the above

    Changed

    • Updated README to include instructions on coverage tests
    Source code(tar.gz)
    Source code(zip)

  • v1.0.1(Jan 12, 2022)

    [1.0.1] - 2022-01-12

    Fixed

    • Spelling error of separate argument in to_pandas function of reddit_data_collector.io.py, previously it was spelt like seperate

    Changed

    • Update example use and move to /examples
    • Update PyPi link in docs to working link
    • Add new potential ideas for contribution
    Source code(tar.gz)
    Source code(zip)

  • v1.0.0(Jan 7, 2022)

Owner
Nico Van den Hooff
UBC Master of Data Science
Nico Van den Hooff
GitHub Repository
A discord bot made by the community (uses python)

discord community bot context: this is a discord bot made by the community by community i mean people adding commands to the bot or changing the bot b

TR ASH 0 Oct 11, 2022
Simple stock price analytics

mune ยท Mune is an open source python web application built to analyze stocks, named after Homma Munehisa. Currently, the forecasting component is powe

Richard Hong 14 Aug 30, 2021
WordPress models and views for Django.

django-wordpress Models and views for reading a WordPress database. Compatible with WordPress version 3.5+. django-wordpress is a project of ISL and t

Jeremy Carbaugh 332 Dec 24, 2022
Changes the Telegram bio, profile picture, first and last name to the song that the user is currently listening to.

TGBIOFY - Telegram & Spotify integration Changes the Telegram bio, profile picture, first and last name to the song that the user is currently listeni

elpideus 26 Dec 07, 2022
A telegram media to pixeldrain stream link bot

Pixeldrain-Bot A telegram media to pixeldrain stream link bot Made with Python3 (C) @FayasNoushad Copyright permission under MIT License License - ht

Fayas Noushad 11 Oct 21, 2022
๐€ ๐ฆ๐จ๐๐ฎ๐ฅ๐š๐ซ ๐“๐ž๐ฅ๐ž๐ ๐ซ๐š๐ฆ ๐†๐ซ๐จ๐ฎ๐ฉ ๐ฆ๐š๐ง๐š๐ ๐ž๐ฆ๐ž๐ง๐ญ ๐›๐จ๐ญ ๐ฐ๐ข๐ญ๐ก ๐ฎ๐ฅ๐ญ๐ข๐ฆ๐š๐ญ๐ž ๐Ÿ๐ž๐š๐ญ๐ฎ๐ซ๐ž๐ฌ

๐‡๐จ๐ฐ ๐“๐จ ๐ƒ๐ž๐ฉ๐ฅ๐จ๐ฒ For easiest way to deploy this Bot click on the below button ๐Œ๐š๐๐ž ๐๐ฒ ๐’๐ฎ๐ฉ๐ฉ๐จ๐ซ๐ญ ๐†๐ซ๐จ๐ฎ๐ฉ ๐’๐จ๐ฎ๐ซ๐œ๐ž๐ฌ ๐†๐ž๐ง๐ž?

Mukesh Solanki 2 Oct 06, 2021
๐Ÿ’ป A fully functional local AWS cloud stack. Develop and test your cloud & Serverless apps offline!

LocalStack - A fully functional local AWS cloud stack LocalStack provides an easy-to-use test/mocking framework for developing Cloud applications. Cur

LocalStack 45.3k Jan 02, 2023
Repo-cloner - Script takes user public liked repos and clone it to a local folder

Liked repos cloner Script takes user public liked repos and clone it to a local

Aleksei 2 Jun 18, 2022
Modern Desktop Jellyfin Client written in Python and Vue for the UI [WIP]

JellyPlayer Modern Jellyfin Client Installation Install Requirements: Install Python 3 Install dependencies Install node deps for frontend, go to Jell

Prayag Prajapati 57 Dec 12, 2022
Subtitle Translater

Subtitle Translater

OshadhaVimukthi 2 Nov 29, 2021
Widevine MPD Content Downloader & Decryptor

Widevine-DL Encrypted MPD Manifest Content Downloader + Decryptor (not a Widevine Key Extractor!) Requirements ffmpeg, yt-dlp, aria2, widevine-l3-decr

Vank0n (SJJeon) 170 Dec 30, 2022
Decode the Ontario proof of vaccination QR code

Decode the contents of the Ontario Proof of Vaccination (the "Smart Health Card QR Code") Output This is from my QR code, hopefully fully redacted alt

Wesley Ellis 4 Oct 22, 2021
A Bot To Find Telegram User ID Easily

Telegram ID Bot ๐Ÿค– A Bot To Find Telegram User ID Easily Made with Python3 (C) @BXBotz Copyright permission under MIT License License - https://githu

MuFaz-TG 6 Nov 21, 2022
An API wrapper for Discord written in Python.

disnake A modern, easy to use, feature-rich, and async ready API wrapper for Discord written in Python. About disnake All the contributors and develop

557 Jan 05, 2023
A calculator telegram bot.

Calculator-Bot A calculator telegram bot. Made with Python3 (C) @FayasNoushad Copyright permission under MIT License License - https://github.com/Fay

Fayas Noushad 33 Nov 30, 2022
This is a translator that i made by myself in python with the 'googletrans' library

Translator-Python This is a translator that i made by myself in python with the 'googletrans' library This application completely made in python allow

Thadeuks 2 Jun 17, 2022
GroupMenter : New Telegram Group Manager Bot๐Ÿ”ธFast ๐Ÿ”ธPython๐Ÿ”ธPyrogram ๐Ÿ”ธ

GroupMenter An PowerFull Group Manager Bot. Written In Pytelethon. Info โ€ข A modular Telegram Python bot running on python3. โ€ข Can be found on telegram

Group Menter 24 Jun 28, 2022
This is a script to export logs from AWS CloudTrail to a local file.

cloudtrail-export-logs This is a script to export logs from AWS CloudTrail to a local file. Getting Started Prerequisites python 3 boto3 pip Installin

Claick Assunรงรฃo de Oliveira 2 Jan 02, 2022
RChecker - Checker for minecraft servers

๐Ÿ”Ž RChecker v1.0 Checker for Minecraft Servers ๐Ÿ’ป Supported operating systems: โœ…

Pedro Vega 1 Aug 30, 2022
A simple test repo created following docker docs.

docker_sampleRepo A simple test repo created following docker docs. Link to docs: https://docs.docker.com/language/python/develop/ Other links: https:

Suraj Verma 2 Sep 16, 2022