SCRAPEGOAT
Scrapegoat is a python library that can be used to scrape the websites from internet based on the relevance of the given topic irrespective of language using Natural Language Processing. It can be mainly used for non-English language to get accurate and relevant scraped text.
Concept
Initially the data is scraped from a website and processed ( to remove English words if the data required is in other language). The BERT model is feed with processed data and topic to compute the cosine similarity of the given topic with each word of the scraped data then mean of cosine similarity scores of is computed. If the mean is greater than threshold then scraped data is generated as output. Also there is a section where we are using Adaptive threshold.
BERT Model
BERT, which stands for Bidirectional Encoder Representations from Transformers, is based on Transformers, a deep learning model in which every output element is connected to every input element, and the weightings between them are dynamically calculated based upon their connection. The BERT framework was pre-trained using text from Wikipedia. The transformer is the part of the model that gives BERT its increased capacity for understanding context and ambiguity in language. The transformer does this by processing any given word in relation to all other words in a sentence, rather than processing them one at a time. By looking at all surrounding words, the Transformer allows the BERT model to understand the full context of the word, and therefore better understand searcher intent.
Cosine Similarity
Cosine similarity is one of the metrics to measure the text-similarity between two documents irrespective of their size in Natural language Processing. A word can be represented in the vector form, therefore the text documents are represented in n-dimensional vector space. If the Cosine similarity score is 1, it means two vectors have the same orientation. The value closer to 0 indicates that the two documents have less similarity. The Cosine similarity of two documents will range from 0 to 1.
Multi Processing
The multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. The basic ideology of Multi-Processing is that if you have an algorithm that can be divided into different workers (small processors/cores), then you can speed up the program. Machines nowadays come with 4,6,8 and 16 cores, therefore parts of the code can be deployed in parallel.
Using Scrapegoat
The examples/test.py file contains these
url = "https://hindi.newslaundry.com/2021/01/22/loan-developing-countries-and-epidemics#:~:text=120%20%E0%A4%A8%E0%A4%BF%E0%A4%AE%E0%A5%8D%E0%A4%A8%20%E0%A4%94%E0%A4%B0%20%E0%A4%AE%E0%A4%A7%E0%A5%8D%E0%A4%AF%E0%A4%AE%20%E0%A4%86%E0%A4%AF,8.1%20%E0%A4%85%E0%A4%B0%E0%A4%AC%20%E0%A4%A1%E0%A5%89%E0%A4%B2%E0%A4%B0%20%E0%A4%B9%E0%A5%8B%20%E0%A4%97%E0%A4%AF%E0%A4%BE."
topic = "Debt of developing countries"
language = 'hi'
if __name__=="__main__":
from scrapegoat.utils import automate
from scrapegoat.main import getLinkData
text,score = getLinkData(url, topic, language=language, tag='p')
print(text, score)
3 Feb 13, 2022
7 Dec 3, 2022
7 Sep 30, 2022
1 Jan 12, 2022
2 Jul 20, 2022
5 Jun 23, 2022
3 Feb 18, 2022
15 Jul 16, 2022
105 Dec 29, 2022
1 Jan 06, 2022
793 Jan 05, 2023
33 Sep 03, 2022
9.3k Jan 02, 2023
9 Apr 06, 2022
1 Nov 05, 2021
4.3k Jan 07, 2023
763 Nov 17, 2022
5 Oct 27, 2022
3k Jan 04, 2023
1 Nov 14, 2021
3 Jan 03, 2022
213 Nov 12, 2022
5 Nov 29, 2022
512 Jan 03, 2023
1 Mar 18, 2022
73 Dec 03, 2022
2 Jan 24, 2022
1 Jan 04, 2022
87 Jan 06, 2023