爬虫案例合集。包括但不限于《淘宝、京东、天猫、豆瓣、抖音、快手、微博、微信、阿里、头条、pdd、优酷、爱奇艺、携程、12306、58、搜狐、百度指数、维普万方、Zlibraty、Oalib、小说、招标网、采购网、小红书》

Last update: Jan 05, 2023

Overview

lxSpider

爬虫案例合集。包括但不限于《淘宝、京东、天猫、豆瓣、抖音、快手、微博、微信、阿里、头条、pdd、优酷、爱奇艺、携程、12306、58、搜狐、百度指数、维普万方、Zlibraty、Oalib、小说网站、招标采购网》

简介：

时光荏苒，记不清写了多少案例了。作者文章发布在csdn，代码随后往github上更新。csdn部分文章为收费案例，合理订阅。

声明：

本库以教学为基准、本库提供的可操作性不得用于任何商业用途和违法违规场景。
作者对任何原因在使用本库中提供的代码和策略时可能对用户自己或他人造成的任何形式的损失和伤害不承担责任。
因本库引起的或与之有关的任何争议，各方应友好协商解决，协商不成的任何后果与作者无关。

专栏

网络爬虫基础：适合有python语法基础准备学爬虫的同学

web逆向基础：有爬虫经验即可（包含猿人学爬虫题目解析）

安卓逆向基础：工具介绍、逆向记录、案例分享

爬虫案例合集：付费专栏、经典案例、持续更新

博客

交流

Releases(快手弹幕采集工具)

快手弹幕采集工具(Jan 30, 2021)
使用说明：

1、启动dist目录下的run.exe程序。

2、填入主播uid，你的cookie，房间id

3、点击启动后，等待即可，不可重复点击。

4、需要确认主播当前是否还在直播。

参数获取：

主播uid：浏览器上的网址最后一个参数。

比如网址为： https://live.kuaishou.com/u/yingjia2019

主播的uid为： yingjia2019

你的cookie：

1、打开控制台，鼠标右键点击审查元素或者按F12.

2、点击控制台的Network。

3、刷新页面，可已按F5刷新

4、找到和主播uid一样html文件，然后点击右侧的headers

5、鼠标划到最下面找到cookie一行。复制里面的did=web_xxxxxxxxxxxxxx;

6、需要在软件上填入的cookie是 web_xxxxxxxxxxxxxx

房间id：

1、点击控制台的 Elements，按ctrl+F，打开搜索框。输入： live-stream-id

2、复制 live-stream-id="Zo9Upaz8w90"

3、要输入的房间id是 Zo9Upaz8w90

运行时最好保持页面打开，关闭页面后过一段时间会导致cookie失效。

此工具以学习为主，禁止滥用
Source code(tar.gz)
Source code(zip)
default.rar(21.47 MB)
小说下载器(Feb 2, 2021)
简介

1、小说下载(优势：速度快，直接从网络上搜集完整txt文件速度快) 2、在线小说爬取(优势：资源全，已上架的小说几乎都能找到)

特别声明:

本脚本仅用于测试和学习研究，禁止用于商业用途，不能保证其合法性，准确性，完整性和有效性，请根据情况自行判断。

本项目内所有资源文件，禁止任何公众号、自媒体进行任何形式的转载、发布。

本项目内任何脚本问题概不负责，包括但不限于由任何脚本错误导致的任何损失或损害.

请勿将项目的任何内容用于商业或非法目的，否则后果自负。

本项目遵循GPL-3.0 License协议，如果本特别声明与GPL-3.0 License协议有冲突之处，以本特别声明为准。

Source code(tar.gz)
Source code(zip)
default.zip(44.16 MB)

Owner

lx

Every noble work is at first impossible.

GitHub Repository

Scrapy, a fast high-level web crawling & scraping framework for Python.

Scrapy Overview Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pag

45.5k Jan 07, 2023

A simple proxy scraper that utilizes the requests module in python.

Proxy Scraper A simple proxy scraper that utilizes the requests module in python. Usage Depending on your python installation your commands may vary.

3 Sep 08, 2021

a way to scrape a database of all of the isef projects

ISEF Database This is a simple web scraper which gets all of the projects and abstract information from here. My goal for this is for someone to get i

1 Mar 18, 2022

Jobinja.ir jobs scraper.

Jobinja.ir Dataset Introduction This project is a simple web scraper that scraps pages of jobinja.ir concurrently and writes and update (if file gets

3 Apr 15, 2022

Web Scraping OLX with Python and Bsoup.

webScrap WebScraping first step. Authors: Paulo, Claudio M. First steps in Web Scraping. Project carried out for training in Web Scrapping. The export

5 Sep 25, 2022

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

1.1k Jan 06, 2023

Create crawler get some new products with maximum discount in banimode website

crawler-banimode create crawler and get some new products with maximum discount in banimode website. این پروژه کوچک جهت یادگیری و کار با ابزار سلنیوم

2 Feb 17, 2022

Examine.com supplement research scraper!

ExamineScraper Examine.com supplement research scraper! Why I want to be able to search pages for a specific term. For example, I want to be able to s

15 Dec 06, 2022

This was supposed to be a web scraping project, but somehow I've turned it into a spamming project

Introduction This was supposed to be a web scraping project, but somehow I've turned it into a spamming project.

1 Jan 23, 2022

EBay-email-tracker - Scapes an entire search page of a particular item on eBay and sends regular updates to an email address

Introduction This is a project I built with the sole intent to learn more about

1 Jan 14, 2022

爬虫案例合集。包括但不限于《淘宝、京东、天猫、豆瓣、抖音、快手、微博、微信、阿里、头条、pdd、优酷、爱奇艺、携程、12306、58、搜狐、百度指数、维普万方、Zlibraty、Oalib、小说、招标网、采购网、小红书》

Related tags

Overview

lxSpider

专栏

目录

博客

推荐

交流

You might also like...

Releases(快手弹幕采集工具)

快手弹幕采集工具(Jan 30, 2021)

使用说明：

参数获取：

你的cookie：

房间id：

小说下载器(Feb 2, 2021)

简介

特别声明:

Owner

lx

Scrapy, a fast high-level web crawling & scraping framework for Python.

A simple proxy scraper that utilizes the requests module in python.

a way to scrape a database of all of the isef projects

Jobinja.ir jobs scraper.

Web Scraping OLX with Python and Bsoup.

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster

Create crawler get some new products with maximum discount in banimode website

Examine.com supplement research scraper!

This was supposed to be a web scraping project, but somehow I've turned it into a spamming project

EBay-email-tracker - Scapes an entire search page of a particular item on eBay and sends regular updates to an email address

A python module to parse the Open Graph Protocol

Crawl the information of a given keyword on Google search engine

Scrapes mcc-mnc.com and outputs 3 files with the data (JSON, CSV & XLSX)

An application that on a given url, crowls a web page and gets all words, sorts and counts them.

Free-Game-Scraper is a useful script that allows you to track down free games and DLCs on many platforms.

Web scraped S&P 500 Data from Wikipedia using Pandas and performed Exploratory Data Analysis on the data.

This is a module that I had created along with my friend. It's a basic web scraping module

Get paper names from dblp.org

This is a python api to scrape search results from a url.

script to scrape direct download links (ddls) from google drive index.