Travis build is running and all tests are passing for both OS (centos 7 + ubuntu trusty) inside docker.

I am getting one error log due to which build is failing in my fork

You have to provide either repo_token in .coveralls.yml, or launch via Travis or CircleCI

It looks like the https://coveralls.io/ requires all coverall calls to go from istresearch/scrapy-cluster fork.

Let me know if anything is required to do the merge.

Pull request to provide the first steps to satisfying #25.

A simple angularjs + Flask UI provides a UI to the user to check status of the Scrapy Cluster and submit crawl requests easily.

The status checks and crawl requests are provided via the rest service.

PR to merge my UI branch into IST UI branch. For discussions and collaboration.

This code has been lingering around on my computer for a while now I think its about time I share it and try and get it to a place where it can be merged in. Further work still required on testing but core pieces of the ui_service are in place.

Having a bit of trouble getting started. Below I've included commands and their outputs (note: some outputs are truncated):

python kafka_monitor.py run
2015-12-06 19:59:00,030 [kafka-monitor] INFO: Kafka Monitor Stats Dump:
{
    "fail_21600": 0,
    "fail_3600": 0,
    "fail_43200": 0,
    "fail_604800": 0,
....
    "plugin_StatsHandler_lifetime": 0,
    "total_21600": 13,
    "total_3600": 13,
    "total_43200": 13,
    "total_604800": 13,
    "total_86400": 13,
    "total_900": 1,
    "total_lifetime": 13
}

python redis_monitor.py
....
    "total_604800": 6,
    "total_86400": 6,
    "total_900": 0,
    "total_lifetime": 6
}
2015-12-06 20:02:39,862 [redis-monitor] INFO: Crawler Stats Dump:
{
    "total_spider_count": 0
}


scrapy runspider crawling/spiders/link_spider.py
2015-12-06 19:56:46,817 [scrapy-cluster] INFO: Changed Public IP: None -> 52.91.192.73

(scrapy_dev)[email protected]:~/scrapy-cluster/kafka-monitor$ python kafka_monitor.py feed '{"url": "http://dmoz.org", "appid":"testapp", "crawlid":"abc1234", "maxdepth":1}'
No override settings found
2015-12-06 19:58:44,573 [kafka-monitor] INFO: Feeding JSON into demo.incoming
{
    "url": "http://dmoz.org",
    "maxdepth": 1,
    "crawlid": "abc1234",
    "appid": "testapp"
}
2015-12-06 19:58:44,580 [kafka-monitor] INFO: Successly fed item to Kafka

python kafkadump.py dump -t demo.crawled_firehose


(scrapy_dev)[email protected]:~/scrapy-cluster/kafka-monitor$ python kafkadump.py dump -t demo.outbound_firehose
No override settings found
2015-12-06 19:35:31,640 [kafkadump] INFO: Connected to localhost:9092
{u'server_time': 1449430706, u'crawlid': u'abc1234', u'total_pending': 0, u'total_domains': 0, u'spiderid': u'link', u'appid': u'testapp', u'domains': {}, u'uuid': u'someuuid'}

I haven't changed any of the default settings and I'm currently using the dev branch. However, I don't think my setup is working. I was expecting some updates in dump -t demo.crawled_firehose. So while I think I've successfully feed a url to be crawled scrapy isn't doing the crawl ? Any ideas?

I'm attempting to get started with 1.2.1 in docker. I've downloaded the project and followed the docker instructions in getting started. When doing the first scrape I can dump and get output from the crawl but not demo.incoming or demo.outbound_firehose.

I don't think this is related but I ran into compatibility issues with the latest Kafka image so I set the version to 1.0.0 in the docker-compose.yml which seemed to be the latest when 1.2.1 was released. This got me past that issue. It's the only change I've made to the project.

Also all the tests pass in the docker images. However in the redis monitor on the first run I get:

OK
test_process_item (__main__.TestRedisMonitor) ... No handlers could be found for logger "redis_lock"
ok

My steps are:

1. docker-compose up -d
2. [terminal 1] docker exec -i scrapycluster121_kafka_monitor_1 python kafkadump.py dump -t demo.crawled_firehose
3. [terminal 2] docker exec -i scrapycluster121_kafka_monitor_1 python kafkadump.py dump -t demo.incoming
4. [terminal 3] docker exec -i scrapycluster121_kafka_monitor_1 python kafkadump.py dump -t demo.outbound_firehose
5. [terminal 4] docker exec -i scrapycluster121_kafka_monitor_1 python kafka_monitor.py feed '{"url": "http://dmoztools.net", "appid":"testapp", "crawlid":"abc1234", "maxdepth":1}'
6. [terminal 4] docker exec -i scrapycluster121_kafka_monitor_1 python kafka_monitor.py feed '{"action":"info", "appid":"testapp", "uuid":"someuuid", "crawlid":"abc1234", "spiderid":"link"}'

After step 5 I start getting Scrapy output on terminal 1. I never get output on terminal 2 or 3.

This PR provides a starting point for registering callbacks using the LogFactory. This PR addresses Issue #91

Usage

Given a logging object logger, you can register a callback via

logger.register_callback(log_level, callback_function, optional_criteria_dict)

Some examples:

logger.register_callback('ERROR', report)

Explanation: The callback function report will fire when the .error() logging method is called

logger.register_callback('<=INFO', add_1, {'key': 'val1'})

Explanation: The callback function add_1 will fire when .debug() or .info() are called AND the {'key': 'val1'} is a subdict of theextras` passed to the logging functions

logger.register_callback('>INFO', negate, {'key': 'val2'})

Explanation: The callback function negate will fire when .warning(), .error(), or .critical() are called AND {'key': 'val2'} is a subdict of extras passed to the logging functions.

logger.register_callback('*', always_fire)

Explanation: The callback function always_fire will fire for all log levels with no concern of the extras dict passed to the logging functions.

Testing

$ python -m unittest tests.test_log_factory

Notes

The callbacks respect the log level. If the log level for a logger is CRITICAL then a .debug() invocation will not trigger the callbacks registered for CRITICAL.

Now Docker containers based Official Python container imaged which are also based on Alpine Linux. Optimizations were done for OS packages that used only during building on some Python packages by removing them after finishing packages installations. Also each subproject contains own requirements.txt to decrease container size.

As result new image sizes became (with shared python layer of 71.95 MB size):

crawler: 144.6 MB (own layers: 72.65 MB)
kafka-monitor: 91.95 MB (own layers: 20 MB)
redis-monitor: 88.67 MB (own layers: 16.72 MB)

In contrast to previous (with shared python layer of 675.1 MB size):

crawler-dev: 780 MB (own layers: 104.9 MB)
redis-monitor-dev: 746.8 MB (own layers: 71.7 MB)
kafka-monitor-dev: 746.8 MB (own layers: 71.7 MB)

Use decode_responses option on Redis client and value_deserializer,value_serializer option on Kafka client to handle unicode problem. Also fix several syntax error and update several test cases for python 3. And as scrapy-cluster 1.2 use ujson instead of pickle, I think no migration is needed.

I'm scraping etsy.com and queue size become more than 1M. When I query for info/statistics it stuck on _get_bin function in scrapy-cluster/redis-monitor/plugins/info_monitor.py file. Also i takes 500MB memory for redis-monitor in that moment.

1. What is the best way to keep queue size small?
2. Perhaps _get_bin should be rewritten in more efficient way to calc statistics in the database.

With Scrapy soon supporting Python 3, we should consider supporting it as well. At a first glance, most of the functionality changes do not affect the code within, but I am sure there needs to be more work done.

test_feed (main.TestKafkaMonitor) ... ERROR test_run (main.TestKafkaMonitor) ... ERROR

====================================================================== ERROR: test_feed (main.TestKafkaMonitor)

Traceback (most recent call last): File "tests/online.py", line 56, in setUp self.kafka_monitor._load_plugins() File "/root/scrapy-cluster/kafka-monitor/kafka_monitor.py", line 75, in _load_plugins the_class = self._import_class(key) File "/root/scrapy-cluster/kafka-monitor/kafka_monitor.py", line 59, in _import_class m = import(cl[0:d], globals(), locals(), [classname]) ImportError: No module named online

====================================================================== ERROR: test_run (main.TestKafkaMonitor)

Traceback (most recent call last): File "tests/online.py", line 56, in setUp self.kafka_monitor._load_plugins() File "/root/scrapy-cluster/kafka-monitor/kafka_monitor.py", line 75, in _load_plugins the_class = self._import_class(key) File "/root/scrapy-cluster/kafka-monitor/kafka_monitor.py", line 59, in _import_class m = import(cl[0:d], globals(), locals(), [classname]) ImportError: No module named online

Ran 2 tests in 0.600s

use ui mode exception stack info,but browser normal File "ui_service.py", line 121, in _kafka_stats r = requests.post(self.settings['REST_HOST'] + "/feed", json=data) File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 110, in post return request('post', url, data=data, json=json, **kwargs) File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 56, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 603, in send adapter = self.get_adapter(url=request.url) File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 685, in get_adapter raise InvalidSchema("No connection adapters were found for '%s'" % url) InvalidSchema: No connection adapters were found fo 5343

Great project, thanks for sharing - and supporting for so long!

I ran into a few problems running the ELK stack - the Elasticsearch container kept restarting with java.lang.IllegalStateException docker-elk-logs.txt

I couldn't find the root cause for this, but in the end switched to using a later version of the ELK stack - v7.10 - which gave good results, and used Filebeat rather than Logstash as there seemed to be more documentation around this use-case. Not sure if this is a change you wanted to make to the project, but have my files on a branch here - happy to submit a pull request if you think that it might be useful: https://github.com/4OH4/scrapy-cluster/tree/elk-update

Haven't managed to properly import the Kibana dashboard configuration from export.json though - I guess a few things have changed between the different versions of Kibana.

Cheers

Hi.

I don't know how much it happens or how much it already happenned but one of my crawl fell on the error below. I ran a thousand requests and it happened to only one of them. But still my crawl was put down. Here is the stack trace:

2021-02-08 15:03:30 [scrapy.core.scraper] ERROR: Error downloading <GET https://pccomponentes-prod.mirakl.net/login> Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks result = g.send(result) File "/usr/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 66, in process_exception spider=spider) File "/usr/src/app/crawler/crawling/log_retry_middleware.py", line 89, in process_exception self._log_retry(request, exception, spider) File "/usr/src/app/crawler/crawling/log_retry_middleware.py", line 102, in _log_retry self.logger.error('Scraper Retry', extra=extras) File "/usr/src/app/crawler/scutils/log_factory.py", line 244, in error extras = self.add_extras(extra, "ERROR") File "/usr/src/app/crawler/scutils/log_factory.py", line 319, in add_extras my_copy = copy.deepcopy(dict) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct state = deepcopy(state, memo) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 264, in _deepcopy_method return type(x)(x.im_func, deepcopy(x.im_self, memo), x.im_class) File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct state = deepcopy(state, memo) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct state = deepcopy(state, memo) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct state = deepcopy(state, memo) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct state = deepcopy(state, memo) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 230, in _deepcopy_list y.append(deepcopy(a, memo)) File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct state = deepcopy(state, memo) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct state = deepcopy(state, memo) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/local/lib/python2.7/copy.py", line 182, in deepcopy rv = reductor(2) TypeError: can't pickle thread.lock objects

Some help would be highly appreciated

Hi. I've just come across this project and it is exactly what we need. However, I've noticed there haven't been any updates for a while now. Could you guys please share your vision for this project? Is it still being maintained? Thank you very much.