Newsemble is an API that provides easy access to the current news for programmatic analysis

Last update: Dec 16, 2022

Overview

📰 Newsemble 📰

An API for fetching the current news.

🔖 About 🔖

Newsemble is an API that provides easy access to the current news for programmatic analysis. It has been built using Python, BeautifulSoup and MongoDB.
The data is scraped from these news websites every hour, stored in a database on the cloud and whenever requested, the most recent articles are promptly served.
Developers can make use of this API to fetch current data with each article having the following fields:
Headlines, Content, Source, Link and Time.

💻 Technologies

Newsemble is created with:

Python 3
Flask
PyMongo
BeautifulSoup

📂 File Structure and Description

app.py - Flask code for the API
scraper.py - Collection of scrapers for the various news sites.
db.py - Connecting and Using MongoDB
utils.py - Utility Functions
scheduler.py - Scheduler
Procfile - For Deployment
requirements.txt - Python Requirments

🛠️ Pipeline

🚀 Getting-started

This project can be accessed by using following setup

Links

Links	Description
www.newsemble.ml/news	Link to fetch all the data from all sources
www.newsemble.ml/news/toi	Link to fetch data from Times of India
www.newsemble.ml/news/th	Link to fetch data from The Hindu
www.newsemble.ml/news/tie	Link to fetch data from The Indian Express
www.newsemble.ml/news/ndtv	Link to fetch data from NDTV news
www.newsemble.ml/news/it	Link to fetch data from India Today

Request format

$ import requests
$ url = "http://www.newsemble.ml/news/"
$ requests.get(url).json()

Response format

{   
    ‘link’      :  $source_link$,
    ‘content’   :  $content_text$,    
    ‘source’    :  $news_source$,
    ‘title’     :  $headline$, 
    ‘time       :  $date_time_of_article$  
 }

Sample output

⚙️ Currently Supported Sites

🙏 Thanks!

All contributions are welcome and appreciated. 👍
If you liked this project, or found it useful in any way, please drop a 🌟 !

✍️ Authors ✍️

✒️ Rishabh Gupta
✒️ Vishal Singhania
✒️ Roshan Kumar

You might also like...

music downloader written in python. (Uses jiosaavn API)

35 Jul 20, 2022

Mobile based API for Crunchyroll BETA (and Downloader).

Mobile based API for Crunchyroll BETA (and Downloader). Not restricted on servers and NO CLOUDFLARE

27 Dec 11, 2022

Pypixiv - A fully-typed, asynchronous api wrapper for pixiv

pypixiv this library is a fully-typed, asynchronous api wrapper for pixiv. featu

2 Nov 16, 2022

This project is helps to download contents from Streamtape by utilizing the API

It scrapes Streamtape api and download contents from the site.

5 Dec 28, 2022

A python script that discovers hidden YouTube API clients. Just a research project.

YouTube-Internal-Clients A script that discovers hidden internal clients of the YouTube (Innertube) API using bruteforce methods. The script tries cli

97 Jan 2, 2023

Simple Python script to download images and videos from public subreddits without using Reddit's API 😎

Subreddit Media Downloader Download images and videos from any public subreddit without using Reddit's API Made with ❤ by Nico 💬 About: This script a

106 Jan 7, 2023

This script just scrapes the most recent Nepali news from Kathmandu Post and notifies the user about current events at regular intervals.It sends out the most recent news at random!

Nepali-news-notifier This script just scrapes the most recent Nepali news from Kathmandu Post and notifies the user about current events at regular in

1 Feb 11, 2022

News-app - This is a news web app for reading news from different sources and topics

1 Feb 2, 2022

Nasdaq Cloud Data Service (NCDS) provides a modern and efficient method of delivery for realtime exchange data and other financial information. This repository provides an SDK for developing applications to access the NCDS.

Nasdaq Cloud Data Service (NCDS) Nasdaq Cloud Data Service (NCDS) provides a modern and efficient method of delivery for realtime exchange data and ot

8 Dec 1, 2022

The windML framework provides an easy-to-use access to wind data sources within the Python world, building upon numpy, scipy, sklearn, and matplotlib. Renewable Wind Energy, Forecasting, Prediction

windml Build status : The importance of wind in smart grids with a large number of renewable energy resources is increasing. With the growing infrastr

125 Dec 24, 2022

Repo Home WPDrawBot - (Repo, Home, WP) A powerful programmatic 2D drawing application for MacOS X which generates graphics from Python scripts. (graphics, dev, mac)

DrawBot DrawBot is a powerful, free application for macOS that invites you to write Python scripts to generate two-dimensional graphics. The built-in

342 Dec 27, 2022

A curated list of programmatic weak supervision papers and resources

118 Jan 2, 2023

Learning trajectory representations using self-supervision and programmatic supervision.

Trajectory Embedding for Behavior Analysis (TREBA) Implementation from the paper: Jennifer J. Sun, Ann Kennedy, Eric Zhan, David J. Anderson, Yisong Y

58 Jan 6, 2023

Programmatic interface to Synapse services for Python

A Python client for Sage Bionetworks' Synapse, a collaborative, open-source research platform that allows teams to share data, track analyses, and collaborate

54 Dec 23, 2022

Django API that scrapes and provides the last news of the city of Carlos Casares by semantic way (RDF format).

"Casares News" API Api that scrapes and provides the last news of the city of Carlos Casares by semantic way (RDF format). Usage Consume the articles

6 May 12, 2022

This python module can analyse cryptocurrency news for any number of coins given and return a sentiment. Can be easily integrated with a Trading bot to keep an eye on the news.

Python script that analyses news headline or body sentiment and returns the overall media sentiment of any given coin. It can take multiple coins an

185 Dec 22, 2022

NLP project that works with news (NER, context generation, news trend analytics)

СоАвтор СоАвтор – платформа и открытый набор инструментов для редакций и журналистов-фрилансеров, который призван сделать процесс создания контента ма

38 Jan 4, 2023

Your own movie streaming service. Easy to install, easy to use. Download, manage and watch your favorite movies conveniently from your browser or phone. Install it on your server, access it anywhere and enjoy.

Vigilio Your own movie streaming service. Easy to install, easy to use. Download, manage and watch your favorite movies conveniently from your browser

141 Jan 6, 2023

💰 An Alfred Workflow that provides current price of cryptocurrency

Coin Ticker for Alfred Workflow An Alfred Workflow that provides current price and status about cryptocurrency from cryptocompare.com. Supports Alfred

14 Nov 17, 2022

Comments

Bump lxml from 4.5.0 to 4.6.5
Bumps lxml from 4.5.0 to 4.6.5.

Changelog

Sourced from lxml's changelog.

4.6.5 (2021-12-12)

Bugs fixed

A vulnerability (GHSL-2021-1038) in the HTML cleaner allowed sneaking script content through SVG images (CVE-2021-43818).

A vulnerability (GHSL-2021-1037) in the HTML cleaner allowed sneaking script content through CSS imports and other crafted constructs (CVE-2021-43818).

4.6.4 (2021-11-01)

Features added

GH#317: A new property system_url was added to DTD entities. Patch by Thirdegree.

GH#314: The STATIC_* variables in setup.py can now be passed via env vars. Patch by Isaac Jurado.

4.6.3 (2021-03-21)

Bugs fixed

A vulnerability (CVE-2021-28957) was discovered in the HTML Cleaner by Kevin Chung, which allowed JavaScript to pass through. The cleaner now removes the HTML5 formaction attribute.

4.6.2 (2020-11-26)

Bugs fixed

A vulnerability (CVE-2020-27783) was discovered in the HTML Cleaner by Yaniv Nizry, which allowed JavaScript to pass through. The cleaner now removes more sneaky "style" content.

4.6.1 (2020-10-18)

... (truncated)

Commits

a9611ba Fix a test in Py2.

a3eacbc Prepare release of 4.6.5.

b7ea687 Update changelog.

69a7473 Cleaner: cover some more cases where scripts could sneak through in specially...

54d2985 Fix condition in test decorator.

4b220b5 Use the non-depcrecated TextTestResult instead of _TextTestResult (GH-333)

d85c6de Exclude a test when using the macOS system libraries because it fails with li...

cd4bec9 Add macOS-M1 as wheel build platform.

fd0d471 Install automake and libtool in macOS build to be able to install the latest ...

f233023 Cleaner: Remove SVG image data URLs since they can embed script content.

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0

Releases(v1.0)

v1.0(Jul 17, 2021)

Source code(tar.gz)
Source code(zip)
v0.1(Jul 16, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Rishabh

CSE Undergrad at IIIT-D.

GitHub Repository http://www.newsemble.ml/news

YouTube Downloader is extremely simple program for downloading songs or playlists (in audio or video) from YouTube. Created using Python, PyTube and PySimpleGUI.

YouTube Downloader YouTube Downloader is extremely simple program for downloading songs or playlists (in audio or video) from YouTube. Disclaimer It's

3 Dec 14, 2022

squid-dl is a massively parallel yt-dlp-based YouTube downloader.

squid-dl squid-dl is a massively parallel yt-dlp-based YouTube downloader. Installation Run the setup.py, which will install squid-dl and its two depe

51 Jan 05, 2023

Neon: an add-on for making it easier to handle component interactions

Neon Neon is an add-on for Lightbulb making it easier to handle component interactions. Installation pip install git+https://github.com/neonjonn/light

9 Apr 29, 2022

Mobile based API for Crunchyroll BETA (and Downloader).

Mobile based API for Crunchyroll BETA (and Downloader). Not restricted on servers and NO CLOUDFLARE

27 Dec 11, 2022

A tool written in Python to download all Snapmaps content from a specific location.

snapmap-archiver A tool written in Python to download all Snapmaps content from a specific location.

46 Dec 09, 2022

A tool to make easy to search for directories in the URL.

Welcome to Brutos Directory Scanner 🚀 The Brutos is a python script used to provide agility in obtaining verifications to informations about related

4 Apr 14, 2022

Youtube video downloader and info extractor for python.

tube_dl Tube_dl is a Simple Youtube video downloader for Python. A Modular approach to bypass and download Youtube Videos and Playlist from Youtube us

16 Jul 09, 2022

YTPY Youtube Downloader Made by: Ferreira, Amarau and Rodric

YTPY Youtube Downloader Made by: Ferreira, Amarau and Rodric How to Install on Linux: sudo apt install python3 python3-pip git pip install pytube git

7 Nov 24, 2022

📼Command line tool based on youtube-dl to easily download selected channels from your subscriptions.

youtube-cdl Command line tool based on youtube-dl to easily download selected channels from your subscriptions. This tool is very handy if you want to

64 Dec 25, 2022

pubmex.py - a script to get a fancy paper title based on given DOI or PMID

pubmex.py is a script to get a fancy paper title based on given DOI or PMID (can be also combined with macOS Finder)

13 Nov 20, 2022

Quickly, simply, and asynchronously download NFT's from an Opensea collection

iRightClick Quickly, simply, and asynchronously download NFT's from an Opensea collection. NOTICE This tool is not developed to encourage or facilitat

34 Dec 30, 2022

FireDM is a python open source (Internet Download Manager) with multi-connections, high speed engine, it downloads general files and videos from youtube and tons of other streaming websites .

python open source (Internet Download Manager) with multi-connections, high speed engine, based on python, LibCurl, and youtube_dl https://github.com/firedm/FireDM

1.6k Apr 12, 2022

Newsemble is an API that provides easy access to the current news for programmatic analysis

Related tags

Overview

📰 Newsemble 📰

🔖 About 🔖

🗒️ Table of contents

💻 Technologies

📂 File Structure and Description

🛠️ Pipeline

🚀 Getting-started

⚙️ Currently Supported Sites

🙏 Thanks!

✍️ Authors ✍️

You might also like...

music downloader written in python. (Uses jiosaavn API)

Mobile based API for Crunchyroll BETA (and Downloader).

Pypixiv - A fully-typed, asynchronous api wrapper for pixiv

This project is helps to download contents from Streamtape by utilizing the API

A python script that discovers hidden YouTube API clients. Just a research project.

Simple Python script to download images and videos from public subreddits without using Reddit's API 😎

This script just scrapes the most recent Nepali news from Kathmandu Post and notifies the user about current events at regular intervals.It sends out the most recent news at random!

News-app - This is a news web app for reading news from different sources and topics

Nasdaq Cloud Data Service (NCDS) provides a modern and efficient method of delivery for realtime exchange data and other financial information. This repository provides an SDK for developing applications to access the NCDS.

The windML framework provides an easy-to-use access to wind data sources within the Python world, building upon numpy, scipy, sklearn, and matplotlib. Renewable Wind Energy, Forecasting, Prediction

Repo Home WPDrawBot - (Repo, Home, WP) A powerful programmatic 2D drawing application for MacOS X which generates graphics from Python scripts. (graphics, dev, mac)

A curated list of programmatic weak supervision papers and resources

Learning trajectory representations using self-supervision and programmatic supervision.

Programmatic interface to Synapse services for Python

Django API that scrapes and provides the last news of the city of Carlos Casares by semantic way (RDF format).

This python module can analyse cryptocurrency news for any number of coins given and return a sentiment. Can be easily integrated with a Trading bot to keep an eye on the news.

NLP project that works with news (NER, context generation, news trend analytics)

Your own movie streaming service. Easy to install, easy to use. Download, manage and watch your favorite movies conveniently from your browser or phone. Install it on your server, access it anywhere and enjoy.

💰 An Alfred Workflow that provides current price of cryptocurrency

Comments

Bump lxml from 4.5.0 to 4.6.5

4.6.5 (2021-12-12)

Bugs fixed

4.6.4 (2021-11-01)

Features added

4.6.3 (2021-03-21)

Bugs fixed

4.6.2 (2020-11-26)

Bugs fixed

4.6.1 (2020-10-18)

Releases(v1.0)

v1.0(Jul 17, 2021)

v0.1(Jul 16, 2021)

Owner

Rishabh

YouTube Downloader is extremely simple program for downloading songs or playlists (in audio or video) from YouTube. Created using Python, PyTube and PySimpleGUI.

squid-dl is a massively parallel yt-dlp-based YouTube downloader.

Neon: an add-on for making it easier to handle component interactions

Mobile based API for Crunchyroll BETA (and Downloader).

A tool written in Python to download all Snapmaps content from a specific location.

A tool to make easy to search for directories in the URL.

Youtube video downloader and info extractor for python.

YTPY Youtube Downloader Made by: Ferreira, Amarau and Rodric

📼Command line tool based on youtube-dl to easily download selected channels from your subscriptions.

pubmex.py - a script to get a fancy paper title based on given DOI or PMID

Quickly, simply, and asynchronously download NFT's from an Opensea collection

FireDM is a python open source (Internet Download Manager) with multi-connections, high speed engine, it downloads general files and videos from youtube and tons of other streaming websites .

Download all games from a public Itch.io Game Jam

A script that downloads YouTube videos/audio

Gogoanime-dl - Gogoanime downloader for downloading anime.

Python based YouTube video Downloader GUI Application.

Ebook downloader built using python

Music, Album and Playlist downloader for JioSaavn

Terminal based YouTube player and downloader

A CLI that searches and download Youtube videos in mp3 format.