Newsscraper - A simple Python 3 module to get crypto or news articles and their content from various RSS feeds.

Overview

NewsScraper

A simple Python 3 module to get crypto or news articles and their content from various RSS feeds.

🔧 Installation

  1. Clone the repo locally.
  2. Use the package manager pip to install the requirements.
pip install -r requirements.txt

Basic Usage

import NewsScraper

all_data = NewsScraper.fetch_all()
news_data = NewsScraper.fetch_news_data()
crypto_data = NewsScraper.fetch_crypto_data()

fetch_all()

Returns a set of NewsScraper.Result containing fetched results from all available RSS feeds

Can include categories: GLOBAL, US, EU, CRYPTO, BLOCKCHAIN, BTC, ETH, LTC.

fetch_news_data()

Returns a set of NewsScraper.Result containing fetched results from CNN, ABC News, Yahoo News, Fox News RSS feeds

Can include categories: GLOBAL, US, EU.

fetch_crypto_data()

Returns a set of NewsScraper.Result containing fetched results from CoinJournal, Crypto Currency News RSS feeds.

Can include categories: CRYPTO, BLOCKCHAIN, BTC, ETH, LTC.

🔨 Advanced Usage

NewsScraper.Result class

A class used to represent a returned article.

Attributes
  • context : str

    A string describing the category of the article.

    ex. "GLOBAL", "US", "BLOCKCHAIN", "BTC".

  • title : str

    A string containing the name of the article.

  • summary : str

    A string containing the summary of the article.

    NOTE: sometimes it can have the value of "", because the RSS feed didn't provide a summary.

  • content : str

    A string containing the content of the article.

Methods
  • Result.json()

    Returns a dictionary with the attributes of the class formatted in JSON.

    ex.

{
  "context": "global",
  "title": "title of the article",
  "summary": "summary of the article",
  "content": "content of the article"
}

News RSS Feeds

All of these functions return a set of NewsScraper.Result containing fetched results of the described RSS feeds.

fetch_abc()
fetch_cnn()
fetch_yahoo()
fetch_fox_news()

Can include categories: GLOBAL, US, EU.

Alternatively, you can use fetch_news_data() to receive results from all of them.


Crypto RSS Feeds

All of these functions return a set of NewsScraper.Result containing fetched results of the described RSS feeds.

fetch_coinjournal()
fetch_cryptocurrencynews()

Can include categories: CRYPTO, BLOCKCHAIN, BTC, ETH, LTC.

Alternatively, you can use fetch_news_data() to receive results from all of them.

🤝 Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

📝 License

This project is licensed under the MIT license.

Owner
Rokas
Rokas
Scrapes all articles and their headlines from theonion.com

The Onion Article Scraper Scrapes all articles and their headlines from the satirical news website https://www.theonion.com Also see Clickhole Article

0 Nov 17, 2021
A social networking service scraper in Python

snscrape snscrape is a scraper for social networking services (SNS). It scrapes things like user profiles, hashtags, or searches and returns the disco

2.4k Jan 01, 2023
Lovely Scrapper

Lovely Scrapper

Tushar Gadhe 2 Jan 01, 2022
爱奇艺会员,腾讯视频,哔哩哔哩,百度,各类签到

My-Actions 个人收集并适配Github Actions的各类签到大杂烩 不要fork了 ⭐️ star就行 使用方式 新建仓库并同步代码 点击Settings - Secrets - 点击绿色按钮 (如无绿色按钮说明已激活。直接到下一步。) 新增 new secret 并设置 Secr

280 Dec 30, 2022
Automated Linkedin bot that will improve your visibility and increase your network.

LinkedinSpider LinkedinSpider is a small project using browser automating to increase your visibility and network of connections on Linkedin. DISCLAIM

Frederik 2 Nov 26, 2021
OSTA web scraper, for checking the status of school buses in Ottawa

OSTA-La-Vista OSTA web scraper, for checking the status of school buses in Ottawa. Getting Started Using a Raspberry Pi, download Python 3, and option

1 Jan 28, 2022
A module for CME that spiders hashes across the domain with a given hash.

hash_spider A module for CME that spiders hashes across the domain with a given hash. Installation Simply copy hash_spider.py to your CME module folde

37 Sep 08, 2022
Creating Scrapy scrapers via the Django admin interface

django-dynamic-scraper Django Dynamic Scraper (DDS) is an app for Django which builds on top of the scraping framework Scrapy and lets you create and

Holger Drewes 1.1k Dec 17, 2022
PyQuery-based scraping micro-framework.

demiurge PyQuery-based scraping micro-framework. Supports Python 2.x and 3.x. Documentation: http://demiurge.readthedocs.org Installing demiurge $ pip

Matias Bordese 109 Jul 20, 2022
Demonstration on how to use async python to control multiple playwright browsers for web-scraping

Playwright Browser Pool This example illustrates how it's possible to use a pool of browsers to retrieve page urls in a single asynchronous process. i

Bernardas Ališauskas 8 Oct 27, 2022
爬取各大SRC当日公告 | 通过微信通知的小工具 | 赏金工具

OnTimeHacker V1.0 OnTimeHacker 是一个爬取各大SRC当日公告,并通过微信通知的小工具 OnTimeHacker目前版本为1.0,已支持24家SRC,列表如下 360、爱奇艺、阿里、百度、哔哩哔哩、贝壳、Boss、58、菜鸟、滴滴、斗鱼、 饿了么、瓜子、合合、享道、京东、

Bywalks 95 Jan 07, 2023
A Web Scraping Program.

Web Scraping AUTHOR: Saurabh G. MTech Information Security, IIT Jammu. If you find this repository useful. I would appreciate if you Star it and Fork

Saurabh G. 2 Dec 14, 2022
A Python module to bypass Cloudflare's anti-bot page.

cloudscraper A simple Python module to bypass Cloudflare's anti-bot page (also known as "I'm Under Attack Mode", or IUAM), implemented with Requests.

VeNoMouS 2.6k Dec 31, 2022
Scrapy-soccer-games - Scraping information about soccer games from a few websites

scrapy-soccer-games Esse projeto tem por finalidade pegar informação de tabela d

Caio Alves 2 Jul 20, 2022
Twitter Scraper

Twitter's API is annoying to work with, and has lots of limitations — luckily their frontend (JavaScript) has it's own API, which I reverse–engineered. No API rate limits. No restrictions. Extremely

Tayyab Kharl 45 Dec 30, 2022
Scrapping Connections' info on Linkedin

Scrapping Connections' info on Linkedin

MohammadReza Ardestani 1 Feb 11, 2022
Web-scraping - Program that scrapes a website for a collection of quotes, picks one at random and displays it

web-scraping Program that scrapes a website for a collection of quotes, picks on

Manvir Mann 1 Jan 07, 2022
✂️🕷️ Spider-Cut is a Network Mapper Framework (NMAP Framework)

Spider-Cut is a Network Mapper Framework (NMAP Framework) Installation | Usage | Creators | Donate Installation # Kali Linux | WSL

XforWorks 3 Mar 07, 2022
Crawl BookCorpus

These are scripts to reproduce BookCorpus by yourself.

Sosuke Kobayashi 590 Jan 03, 2023
Telegram group scraper tool

Telegram Group Scrapper

Wahyusaputra 2 Jan 11, 2022