crypto currency scraping

Related tags

Web Crawlingscrypto
Overview

SCRYPTO

What ?

Crypto currencies scraping

(At the moment, only bitcoin and ethereum crypto currencies are supported)

How ?

A python script is running in a container, and scrape informations (using CoinGecko API) about crypto currencies. Then this informations are send to a SQL database.
(You can also run the script alone with a .env file and a systemd service)

SETUP

  • You can launch this scraper by simply using the "docker-compose.yml" file that i let you on this repo.

Docker-compose file

Here is an example of docker-compose file.

version: "3"
services:
  scrypto:
    image: baldurr/scrypto:latest
    container_name: scrypto
    environment:
      - CRYPTO_LIST=bitcoin,ethereum
      - DEVISE_1=eur
      - DEVISE_2=usd
      - SCRAPE_TIME=300
      - SQL_USER=root2
      - SQL_PASSWORD=mypwd
      - SQL_HOST=192.168.1.20
      - SQL_DB=db_scrypto
      - SQL_PORT=3308
    restart: unless-stopped

Environment variable

List of all currencies available:
"btc", "eth", "ltc", "bch", "bnb", "eos", "xrp", "xlm", "link", "dot", "yfi", "usd", "aed", "ars", "aud", "bdt", "bhd", "bmd", "brl", "cad", "chf", "clp", "cny", "czk", "dkk", "eur", "gbp", "hkd", "huf", "idr", "ils", "inr", "jpy", "krw", "kwd", "lkr", "mmk", "mxn", "myr", "ngn", "nok", "nzd", "php", "pkr", "pln", "rub", "sar", "sek", "sgd", "thb", "try", "twd", "uah", "vef", "vnd", "zar", "xdr", "xag", "xau", "bits", "sats"

Var Usage Info
CRYPTO_LIST List of currencies separated by ',' Max: 2 currencies
DEVISE_1 Name of the 1st currencie defined ex: eur
DEVISE_2 Name of the 1nd currencie defined ex: usd
SCRAPE_TIME Scrape interval in second ex: 300 = 5min
SQL_USER SQL user used
SQL_PASSWORD SQL user password
SQL_HOST SQL host which host the database ex: 192.168.1.20, don't set localhost while this will refer to the scrypto container if you use the docker method
SQL_DB SQL database name This var must be set to 'db_scrypto'
SQL_PORT SQL database port

Insatallation

SQL configuration

To use correctly this image, you must create a database named 'db_scrypto'.

If you use the docker method, connect to the scrypto container:

docker exec -it scrypto /bin/bash

Then connect to the SQL database:

mysql -u myuser -p

Enter your password and display databases like this:

SHOW databases;

If 'db_scrypto' doesn't exist, create it:

CREATE DATABASE db_scrypto;

Then you have to create the tables to store the data.
NOTICE:

  • For the table name, please name it like the crypto currencie name: bitcoin, ethereum
  • For the value columns, set the name of the column like this: value_mycurrencie (ex: value_usd)
CREATE TABLE ethereum (data_id INT NOT NULL AUTO_INCREMENT, time DATETIME, metric VARCHAR(20), value_eur numeric(10,2), value_usd numeric(10,2), PRIMARY KEY(data_id));
CREATE TABLE bitcoin (data_id INT NOT NULL AUTO_INCREMENT, time DATETIME, metric VARCHAR(20), value_eur  numeric(10,2), value_usd numeric(10,2), PRIMARY KEY(data_id));

Now you are ready to collect data

Docker configuration

DockerHub image: https://hub.docker.com/repository/docker/baldurr/scrypto

mkdir scrypto
cd scrypto
wget https://raw.githubusercontent.com/Baldurrr/scrypto/main/docker-compose.yml
docker-compose up -d

Wait a bit and:
docker logs scrypto (will display the api response if the configuration worked)

THE RESULT

In this repo, you will also find a json fill that contain the grafana dashboard configuration

Grafana dashboard example: image

Automatically download and crop key information from the arxiv daily paper.

Arxiv daily 速览 功能:按关键词筛选arxiv每日最新paper,自动获取摘要,自动截取文中表格和图片。 1 测试环境 Ubuntu 16+ Python3.7 torch 1.9 Colab GPU 2 使用演示 首先下载权重baiduyun 提取码:il87,放置于code/Pars

HeoLis 20 Jul 30, 2022
A simple reddit scraper to get memes (only images) from r/ProgrammerHumor.

memey A simple reddit scraper to get memes (only images) from r/ProgrammerHumor. Note Only works if you have firefox installed (yet). Instructions foo

2 Nov 16, 2021
Example of scraping a paginated API endpoint and dumping the data into a DB

Provider API Scraper Example Example of scraping a paginated API endpoint and dumping the data into a DB. Pre-requisits Python = 3.9 Pipenv Setup # i

Alex Skobelev 1 Oct 20, 2021
An Automated udemy coupons scraper which scrapes coupons and autopost the result in blogspot post

Autoscraper-n-blogger An Automated udemy coupons scraper which scrapes coupons and autopost the result in blogspot post and notifies via Telegram bot

GOKUL A.P 13 Dec 21, 2022
Luis M. Capdevielle 1 Jan 14, 2022
Rottentomatoes, Goodreads and IMDB sites crawler. Semantic Web final project.

Crawler Rottentomatoes, Goodreads and IMDB sites crawler. Crawler written by beautifulsoup, selenium and lxml to gather books and films information an

Faeze Ghorbanpour 1 Dec 30, 2021
Examine.com supplement research scraper!

ExamineScraper Examine.com supplement research scraper! Why I want to be able to search pages for a specific term. For example, I want to be able to s

Tyler 15 Dec 06, 2022
Displays market info for the LUNI token on the Terra Blockchain

LuniBot for Discord Displays market info for the LUNI/LUNA token on the Terra Blockchain (Webscrape method currently scraping CoinMarketCap). Will evo

0 Jan 22, 2022
HappyScrapper - Google news web scrapper with python

HappyScrapper ~ Google news web scrapper INSTALLATION ♦ Clone the repository ♦ O

Jhon Aguiar 0 Nov 07, 2022
Binance Smart Chain Contract Scraper + Contract Evaluator

Pulls Binance Smart Chain feed of newly-verified contracts every 30 seconds, then checks their contract code for links to socials.Returns only those with socials information included, and then submit

14 Dec 09, 2022
A spider for Universal Online Judge(UOJ) system, converting problem pages to PDFs.

Universal Online Judge Spider Introduction This is a spider for Universal Online Judge (UOJ) system (https://uoj.ac/). It also works for all other Onl

TriNitroTofu 1 Dec 07, 2021
Web scraping library and command-line tool for text discovery and extraction (main content, metadata, comments)

trafilatura: Web scraping tool for text discovery and retrieval Description Trafilatura is a Python package and command-line tool which seamlessly dow

Adrien Barbaresi 704 Jan 06, 2023
Dex-scrapper - Hobby project for scrapping dex data on VeChain

Folders /zumo_abis # abi extracted from zumo repo /zumo_pools # runtime e

3 Jan 20, 2022
Web scraper for Zillow

Zillow-Scraper Instructions All terminal commands are highlighted. Make sure you first have python 3 installed. You can check this by running "python

Ali Rastegar 1 Nov 23, 2021
此脚本为 python 脚本,实现原理为利用 selenium 定位相关元素,再配合点击事件完成浏览器的自动化.

此脚本为 python 脚本,实现原理为利用 selenium 定位相关元素,再配合点击事件完成浏览器的自动化.

N0el4kLs 5 Nov 19, 2021
API which uses discord to scrape NameMC searches/droptime/dropping status of minecraft names

NameMC Scrape API This is an api to scrape NameMC using message previews generated by discord. NameMC makes it a pain to scrape their website, but som

Twilak 2 Dec 22, 2021
京东云无线宝积分推送,支持查看多设备积分使用情况

JDRouterPush 项目简介 本项目调用京东云无线宝API,可每天定时推送积分收益情况,帮助你更好的观察主要信息 更新日志 2021-03-02: 查询绑定的京东账户 通知排版优化 脚本检测更新 支持Server酱Turbo版 2021-02-25: 实现多设备查询 查询今

雷疯 199 Dec 12, 2022
👁️ Tool for Data Extraction and Web Requests.

httpmapper 👁️ Project • Technologies • Installation • How it works • License Project 🚧 For educational purposes. This is a project that I developed,

15 Dec 05, 2021
UsernameScraperTool - Username Scraper Tool With Python

UsernameScraperTool Username Scraper for 40+ Social sites. How To use git clone

E4crypt3d 1 Dec 20, 2022
Raspi-scraper is a configurable python webscraper that checks raspberry pi stocks from verified sellers

Raspi-scraper is a configurable python webscraper that checks raspberry pi stocks from verified sellers.

Louie Cai 13 Oct 15, 2022