SCRAPEGOAT
Scrapegoat is a python library that can be used to scrape the websites from internet based on the relevance of the given topic irrespective of language using Natural Language Processing. It can be mainly used for non-English language to get accurate and relevant scraped text.
Concept
Initially the data is scraped from a website and processed ( to remove English words if the data required is in other language). The BERT model is feed with processed data and topic to compute the cosine similarity of the given topic with each word of the scraped data then mean of cosine similarity scores of is computed. If the mean is greater than threshold then scraped data is generated as output. Also there is a section where we are using Adaptive threshold.
BERT Model
BERT, which stands for Bidirectional Encoder Representations from Transformers, is based on Transformers, a deep learning model in which every output element is connected to every input element, and the weightings between them are dynamically calculated based upon their connection. The BERT framework was pre-trained using text from Wikipedia. The transformer is the part of the model that gives BERT its increased capacity for understanding context and ambiguity in language. The transformer does this by processing any given word in relation to all other words in a sentence, rather than processing them one at a time. By looking at all surrounding words, the Transformer allows the BERT model to understand the full context of the word, and therefore better understand searcher intent.
Cosine Similarity
Cosine similarity is one of the metrics to measure the text-similarity between two documents irrespective of their size in Natural language Processing. A word can be represented in the vector form, therefore the text documents are represented in n-dimensional vector space. If the Cosine similarity score is 1, it means two vectors have the same orientation. The value closer to 0 indicates that the two documents have less similarity. The Cosine similarity of two documents will range from 0 to 1.
Multi Processing
The multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. The basic ideology of Multi-Processing is that if you have an algorithm that can be divided into different workers (small processors/cores), then you can speed up the program. Machines nowadays come with 4,6,8 and 16 cores, therefore parts of the code can be deployed in parallel.
Using Scrapegoat
The examples/test.py file contains these
url = "https://hindi.newslaundry.com/2021/01/22/loan-developing-countries-and-epidemics#:~:text=120%20%E0%A4%A8%E0%A4%BF%E0%A4%AE%E0%A5%8D%E0%A4%A8%20%E0%A4%94%E0%A4%B0%20%E0%A4%AE%E0%A4%A7%E0%A5%8D%E0%A4%AF%E0%A4%AE%20%E0%A4%86%E0%A4%AF,8.1%20%E0%A4%85%E0%A4%B0%E0%A4%AC%20%E0%A4%A1%E0%A5%89%E0%A4%B2%E0%A4%B0%20%E0%A4%B9%E0%A5%8B%20%E0%A4%97%E0%A4%AF%E0%A4%BE."
topic = "Debt of developing countries"
language = 'hi'
if __name__=="__main__":
from scrapegoat.utils import automate
from scrapegoat.main import getLinkData
text,score = getLinkData(url, topic, language=language, tag='p')
print(text, score)
3 Feb 13, 2022
7 Dec 3, 2022
7 Sep 30, 2022
1 Jan 12, 2022
2 Jul 20, 2022
5 Jun 23, 2022
3 Feb 18, 2022
15 Jul 16, 2022
105 Dec 29, 2022
1 Jan 06, 2022
798 Dec 29, 2022
0 Jan 07, 2022
0 Dec 06, 2021
129 Dec 14, 2022
8 Aug 07, 2022
12 Oct 28, 2022
28 Dec 20, 2022
3 Feb 10, 2022
61 Jun 21, 2021
13 Oct 15, 2022
15 May 17, 2022
4 Aug 20, 2022
53 Dec 16, 2022
1 Nov 23, 2021
1 Feb 17, 2022
543 Jan 03, 2023
457 Jan 05, 2023
1 Dec 30, 2021
32 Dec 31, 2022