Makes grequests fully compatible with Requests v1.1.0

• Adds callback parameter to get, post, etc, works as shortcut for hooks={'response': callback}.
• Restores ability to pass Session object.
• Adds basic tests.

I was looking forward to using this awesome package with my app and gevent until I noticed that it accomplishes gevent-compatibility via gevent.monkey.patch_all():

try:
    import gevent
    from gevent import monkey as curious_george
    from gevent.pool import Pool
except ImportError:
    raise RuntimeError('Gevent is required for grequests.')

# Monkey-patch.
curious_george.patch_all(thread=False, select=False)

Unfortunately, patch_all() impacts all other packages/modules used by the app, breaking some of them that were not designed for reentrancy. For example, gevent.monkey.patch_all() broke the combination of pymysql/pooled_db (with sporadic, hard-to-reproduce failures); my app has several greenlets that on occasion make pymysql calls; I isolated the problem to gevent's monkey-patching of the socket module in that case.

A local solution involving monkey-patching of just the requests package to use gevent's variants of blocking API's would greatly enhance compatibility of grequests with apps and eliminate unexpected/undesirable side-effects on other packages. For example, I recently monkey-patched HBase/Thrift via the following code snippet with the desired effect on Thrift and no harm to the rest of the app (pymysql, etc.):

# Monkey-patch Thrift's TSocket to use gevent.socket to make our HBase interface
# gevent-friendly
import gevent.socket
from thrift.transport import TSocket
TSocket.socket = gevent.socket

gevent-zeromq is another example of local monkey-patching, albeit a more complex one: https://github.com/traviscline/gevent-zeromq/blob/master/gevent_zeromq/init.py

Thank you, Vitaly

Resubmitting this due to a mistake I made in the branching. This builds off of this pull request here, so you will not want to merge both:

https://github.com/kennethreitz/grequests/pull/15

The aggressive python stdlib monkey patching originating from https://github.com/kennethreitz/grequests/blob/master/grequests.py#L21 causes a lot of breakage throughout my app. The degree of monkey patching causes the redis client to error out which in turn then breaks all async celery processing.

I really like the grequests lib, but cannot use it if it's going to make things that used to work fine not work anymore.

  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/billiard/pool.py", line 1426, in safe_apply_callback
    fun(*args)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/celery/worker/job.py", line 347, in on_success
    self.acknowledge()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/celery/worker/job.py", line 453, in acknowledge
    self.on_ack(logger, self.connection_errors)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/base.py", line 100, in ack_log_error
    self.ack()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/base.py", line 95, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/virtual/__init__.py", line 511, in basic_ack
    self.qos.ack(delivery_tag)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/redis.py", line 127, in ack
    self._remove_from_indices(delivery_tag).execute()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 1919, in execute
    return execute(conn, stack, raise_on_error)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 1811, in _execute_transaction
    self.parse_response(connection, '_')
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 1882, in parse_response
    self, connection, command_name, **options)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 387, in parse_response
    response = connection.read_response()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/connection.py", line 307, in read_response
    response = self._parser.read_response()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/connection.py", line 105, in read_response
    response = self.read()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/connection.py", line 90, in read
    return self._fp.readline()[:-2]
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 447, in readline
    data = self._sock.recv(self._rbufsize)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/socket.py", line 392, in recv
    self._wait(self._read_event)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/socket.py", line 298, in _wait
    self.hub.wait(watcher)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/hub.py", line 341, in wait
    result = waiter.get()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/hub.py", line 568, in get
    return self.hub.switch()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/hub.py", line 331, in switch
    return greenlet.switch(self)
LoopExit: This operation would block forever

Additional references pertaining to gevent monkey patching issues:

• http://stackoverflow.com/questions/13040458/why-is-gevent-spawn-different-than-a-monkeypatched-threading-thread
• https://github.com/jrief/django-websocket-redis/issues/3
• http://bighow.net/5332403-Why_is__gevent_spawn__different_than_a_monkeypatched__threading_Thread____.html
• http://stackoverflow.com/questions/9192539/using-gevent-monkey-patching-with-threading-makes-thread-work-serially
• http://pastebin.com/TPYMvuVr

Hi Kenneth,

May I suggest that you consider using this instead of gevent:

http://docs.python.org/dev/library/concurrent.futures.html

It has a backport to previous versions

http://pypi.python.org/pypi/futures

It has a good interface IMO, it's in the standard library so will be supported indefinitely, it doesn't have external native dependencies like gevent does, and it can help avoid the problems stemming from gevent monkey patching as in issue 1. To be fair, gevent has awesome performance but it's just not as easy a dependency to work with as I'd like (I had to update to 1.0b2 to get pool.imap to work, but that version is not in pypi, etc.)

test fix for requests hook kwargs [1] is handy for packagers / porters (QA) , along with the test_suite and tests_require changes [2]

[1] https://github.com/kennethreitz/grequests/commit/f50782ad63607b85e5e009f4a0a4ce0a8a6aef0d [2] https://github.com/kennethreitz/grequests/pull/47

>>> import requests
>>> import grequests
>>> requests.get('https://google.com')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 844, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 314, in connect
    cert_reqs=resolve_cert_reqs(self.cert_reqs),
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/util/ssl_.py", line 264, in create_urllib3_context
    context.options |= options
  File "/usr/lib/python3.6/ssl.py", line 459, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/lib/python3.6/ssl.py", line 459, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/lib/python3.6/ssl.py", line 459, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  [Previous line repeated 323 more times]
RecursionError: maximum recursion depth exceeded

If the order of imports is reversed, the problem no longer occurs.

Can't find anything that solves my problem, I don't know what else to do.

Installing collected packages: gevent, grequests
  Running setup.py install for gevent
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-09c1Gh/gevent/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-0hQM3Z-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.10-intel-2.7
    creating build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/__init__.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/_ssl2.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/_sslgte279.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/_threading.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/backdoor.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/baseserver.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/coros.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/event.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/fileobject.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/greenlet.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/hub.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/local.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/lock.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/monkey.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/os.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/pool.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/pywsgi.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/queue.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/resolver_ares.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/resolver_thread.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/select.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/server.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/socket.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/ssl.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/subprocess.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/thread.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/threading.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/threadpool.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/timeout.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/util.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/win32util.py -> build/lib.macosx-10.10-intel-2.7/gevent
    copying gevent/wsgi.py -> build/lib.macosx-10.10-intel-2.7/gevent
    running build_ext
    Running '/bin/sh /private/tmp/pip-build-09c1Gh/gevent/libev/configure > configure-output.txt' in /private/tmp/pip-build-09c1Gh/gevent/build/temp.macosx-10.10-intel-2.7/libev
    building 'gevent.core' extension
    creating build/temp.macosx-10.10-intel-2.7/gevent
    cc -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -U__llvm__ -arch x86_64 -arch i386 -pipe -DLIBEV_EMBED=1 -DEV_COMMON= -DEV_CLEANUP_ENABLE=0 -DEV_EMBED_ENABLE=0 -DEV_PERIODIC_ENABLE=0 -Ibuild/temp.macosx-10.10-intel-2.7/libev -Ilibev -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c gevent/gevent.core.c -o build/temp.macosx-10.10-intel-2.7/gevent/gevent.core.o
    In file included from gevent/gevent.core.c:249:
    In file included from gevent/libev.h:2:
    libev/ev.c:483:48: warning: '/*' within block comment [-Wcomment]
    /*#define MIN_INTERVAL  0.00000095367431640625 /* 1/2**20, good till 2200 */
                                                   ^
    libev/ev.c:1029:42: error: '_Noreturn' keyword must precede function declarator
      ecb_inline void ecb_unreachable (void) ecb_noreturn;
                                             ^~~~~~~~~~~~
      _Noreturn
    libev/ev.c:832:26: note: expanded from macro 'ecb_noreturn'
      #define ecb_noreturn   _Noreturn
                             ^
    libev/ev.c:1625:31: warning: 'extern' variable has an initializer [-Wextern-initializer]
      EV_API_DECL struct ev_loop *ev_default_loop_ptr = 0; /* needs to be initialised to make it a definition despite extern */
                                  ^
    libev/ev.c:1796:7: warning: unused variable 'ocur_' [-Wunused-variable]
          array_needsize (ANPENDING, pendings [pri], pendingmax [pri], w_->pending, EMPTY2);
          ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    libev/ev.c:1807:3: warning: unused variable 'ocur_' [-Wunused-variable]
      array_needsize (W, rfeeds, rfeedmax, rfeedcnt + 1, EMPTY2);
      ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    libev/ev.c:1934:7: warning: unused variable 'ocur_' [-Wunused-variable]
          array_needsize (int, fdchanges, fdchangemax, fdchangecnt, EMPTY2);
          ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    In file included from gevent/gevent.core.c:249:
    In file included from gevent/libev.h:2:
    In file included from libev/ev.c:2484:
    libev/ev_kqueue.c:50:3: warning: unused variable 'ocur_' [-Wunused-variable]
      array_needsize (struct kevent, kqueue_changes, kqueue_changemax, kqueue_changecnt, EMPTY2);
      ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    In file included from gevent/gevent.core.c:249:
    In file included from gevent/libev.h:2:
    In file included from libev/ev.c:2490:
    libev/ev_poll.c:66:7: warning: unused variable 'ocur_' [-Wunused-variable]
          array_needsize (struct pollfd, polls, pollmax, pollcnt, EMPTY2);
          ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    libev/ev.c:3648:34: warning: '&' within '|' [-Wbitwise-op-parentheses]
      fd_change (EV_A_ fd, w->events & EV__IOFDSET | EV_ANFD_REIFY);
                           ~~~~~~~~~~^~~~~~~~~~~~~ ~
    libev/ev.c:3648:34: note: place parentheses around the '&' expression to silence this warning
      fd_change (EV_A_ fd, w->events & EV__IOFDSET | EV_ANFD_REIFY);
                                     ^
                           (                      )
    libev/ev.c:3687:3: warning: unused variable 'ocur_' [-Wunused-variable]
      array_needsize (ANHE, timers, timermax, ev_active (w) + 1, EMPTY2);
      ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    libev/ev.c:4367:5: warning: unused variable 'ocur_' [-Wunused-variable]
        array_needsize (ev_idle *, idles [ABSPRI (w)], idlemax [ABSPRI (w)], active, EMPTY2);
        ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    libev/ev.c:4407:3: warning: unused variable 'ocur_' [-Wunused-variable]
      array_needsize (ev_prepare *, prepares, preparemax, preparecnt, EMPTY2);
      ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    libev/ev.c:4445:3: warning: unused variable 'ocur_' [-Wunused-variable]
      array_needsize (ev_check *, checks, checkmax, checkcnt, EMPTY2);
      ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    libev/ev.c:4592:3: warning: unused variable 'ocur_' [-Wunused-variable]
      array_needsize (ev_fork *, forks, forkmax, forkcnt, EMPTY2);
      ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    libev/ev.c:4675:3: warning: unused variable 'ocur_' [-Wunused-variable]
      array_needsize (ev_async *, asyncs, asyncmax, asynccnt, EMPTY2);
      ^
    libev/ev.c:1758:22: note: expanded from macro 'array_needsize'
          int ecb_unused ocur_ = (cur);                                     \
                         ^
    14 warnings and 1 error generated.
    error: command 'cc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-09c1Gh/gevent/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-0hQM3Z-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-09c1Gh/gevent```

See kennethreitz/requests#500 for context, but basically the timeout is no longer respected after importing the async module that monkeypatches socket.

>>> import requests
>>> import time
>>> start = time.time(); r = requests.get('http://httpbin.org/delay/4', timeout=2.0); print time.time() - start, r.status_code
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/api.py", line 52, in get
    return request('get', url, **kwargs)
  File "requests/api.py", line 40, in request
    return s.request(method=method, url=url, **kwargs)
  File "requests/sessions.py", line 229, in request
    r.send(prefetch=prefetch)
  File "requests/models.py", line 604, in send
    raise Timeout('Request timed out.')
requests.exceptions.Timeout: Request timed out.
>>> start = time.time(); r = requests.get('http://httpbin.org/delay/4', timeout=2.0); print time.time() - start, r.status_code
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/api.py", line 52, in get
    return request('get', url, **kwargs)
  File "requests/api.py", line 40, in request
    return s.request(method=method, url=url, **kwargs)
  File "requests/sessions.py", line 229, in request
    r.send(prefetch=prefetch)
  File "requests/models.py", line 604, in send
    raise Timeout('Request timed out.')
requests.exceptions.Timeout: Request timed out.
>>> import requests.async
>>> start = time.time(); r = requests.get('http://httpbin.org/delay/4', timeout=2.0); print time.time() - start, r.status_code
4.38320612907 200
>>> start = time.time(); r = requests.get('http://httpbin.org/delay/4', timeout=2.0); print time.time() - start, r.status_code
4.67993688583 200

Is it possible to initialize a pool to return the result but continue the code execution like so?

import grequests

def print_res(res, *args, **kwargs):
    from pprint import pprint
    pprint (vars(res))

print 'starting grequest'

req = grequests.post('http://www.cricket.com.au/', hooks=dict(response=print_res))
job = grequests.send(req, grequests.Pool(1))

print 'request made'

for i in range(10):
    print i

So that would show the request was made but still print the range while the request was in process and then finally print out the results of the send?

I'd really love to use grequests but the examples/documentation are fairly sparse so I'm struggling.

Thanks for any help.

I am using grequests to test the performance of my service. Both the service and the machine from which I start the requests are inside Amazon EC2. It seems that I cannot start more requests than ulimit -n, even though I explicitly specify the size parameter for map() / imap()

My setup is roughly like this:

headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
params = [json.dumps({'url': urls[i]}) for i in urls]
reqs = [grequests.post(SERVICE_URL, data=params[i % len(params)], headers=headers)
            for i in xrange(no_requests)]
res = grequests.imap(reqs, size=200)
for r in res:
    ...
    r.close()

Doesn't the size parameter here tell grequests that it should issue no_requests, 200 at a time?

Adds the imap_enumerated function.

Like imap, but yields tuple of original request index (as it was provided in the requests parameter sequence) and response object.

Unlike imap, failed results and responses from exception handlers that return None are not ignored. Instead, a tuple of (index, None) is yielded. Additionally, the requests parameter must be a sequence of Request objects (generators will be exhausted, but this may cause issues if the generator is infinite or otherwise very large).

The index is merely the original index of the original request in the requests list and does NOT provide any indication of the order in which requests or responses are sent or received. Responses are still yielded in arbitrary order.

Resolves #159

Hi!, thx for this lib!, I was looking long time ago how to do this in a fast way.

The actual imap function does not keep the order of map, is not necessary bad, but I think would be great if it can return the index. At least in my case, the data usually need more things than the request it self, and the index help to identify and relate information.

Thx!

Python 3.8.3 (default, May 19 2020, 06:50:17) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32

pip list
Package                Version
---------------------- -------------------
alembic                1.4.3
apache-airflow         1.10.12
apispec                1.3.3
argcomplete            1.12.1
attrs                  19.3.0
Babel                  2.8.0
cached-property        1.5.2
cattrs                 1.0.0
certifi                2020.4.5.1
cffi                   1.14.0
chardet                3.0.4
click                  7.1.2
colorama               0.4.4
colorlog               4.0.2
conda                  4.8.3
conda-package-handling 1.7.0
configparser           3.5.3
croniter               0.3.35
cryptography           2.9.2
defusedxml             0.6.0
dill                   0.3.2
dnspython              2.0.0
docutils               0.16
email-validator        1.1.1
Flask                  1.1.2
Flask-Admin            1.5.4
Flask-AppBuilder       2.3.4
Flask-Babel            1.0.0
Flask-Caching          1.3.3
Flask-JWT-Extended     3.24.1
Flask-Login            0.4.1
Flask-OpenID           1.2.5
Flask-SQLAlchemy       2.4.4
flask-swagger          0.2.14
Flask-WTF              0.14.3
funcsigs               1.0.2
future                 0.18.2
gevent                 20.9.0
graphviz               0.14.2
greenlet               0.4.17
grequests              0.6.0
gunicorn               20.0.4
idna                   2.9
iso8601                0.1.13
itsdangerous           1.1.0
Jinja2                 2.11.2
json-merge-patch       0.2
jsonschema             3.2.0
lazy-object-proxy      1.5.1
lockfile               0.12.2
Mako                   1.1.3
Markdown               2.6.11
MarkupSafe             1.1.1
marshmallow            2.21.0
marshmallow-enum       1.5.1
marshmallow-sqlalchemy 0.24.0
menuinst               1.4.16
natsort                7.0.1
Nuitka                 0.6.9.4
numpy                  1.19.2
pandas                 1.1.3
pendulum               1.4.4
pip                    20.0.2
prison                 0.1.3
psutil                 5.7.3
pycosat                0.6.3
pycparser              2.20
Pygments               2.7.2
PyJWT                  1.7.1
pyOpenSSL              19.1.0
pyotp                  2.4.1
pyrsistent             0.17.3
PySocks                1.7.1
python-daemon          2.2.4
python-dateutil        2.8.1
python-editor          1.0.4
python-nvd3            0.15.0
python-slugify         4.0.1
python3-openid         3.2.0
pytz                   2020.1
pytzdata               2020.1
pywin32                227
PyYAML                 5.3.1
requests               2.23.0
requests-toolbelt      0.9.1
ruamel-yaml            0.15.87
setproctitle           1.1.10
setuptools             46.4.0.post20200518
six                    1.14.0
SQLAlchemy             1.3.20
SQLAlchemy-JSONField   0.9.0
SQLAlchemy-Utils       0.36.8
tabulate               0.8.7
tenacity               4.12.0
text-unidecode         1.3
thrift                 0.13.0
tornado                6.0.4
tqdm                   4.46.0
tzlocal                1.5.1
unicodecsv             0.14.1
urllib3                1.25.8
Werkzeug               0.16.1
wheel                  0.34.2
win-inet-pton          1.1.0
wincertstore           0.2
WTForms                2.3.3
zope.deprecation       4.4.0
zope.event             4.5.0
zope.interface         5.1.2

i'm trying to use grequests to check live stream ,i just expect a status_code 200,so i use stream = True and close it my code like :

class ExInfoAdapter(HTTPAdapter): 
    def send(self, request, **kwargs):
        request.raw_info = request.headers.pop("data", None)
        return super(ExInfoAdapter, self).send(request, **kwargs)

cdnsession = requests.session()
cdnsession.headers.update(self.header)
retries = Retry(total=2, backoff_factor=0, status_forcelist=[500, 502, 503, 504, 404], raise_on_redirect=False)
cdnsession.mount('http://', ExInfoAdapter(max_retries=retries, pool_connections=200, pool_maxsize=400))
cdnsession.mount('https://', ExInfoAdapter(max_retries=retries, pool_connections=200, pool_maxsize=400))

for ...
	greqs.append(grequests.get(
                    "http://{rip}/live/{zuid}.flv".format(rip=ip, zuid=zuid),
                    headers={"Host": host,
                             "data": json.dumps( # this will pop out and never send to server
                                 {"rip": ip, "zuid": zuid, "domain": domain, "type": ctype})},
                    timeout=15,
                    session=cdnsession
                ))
def fqecp(request, exception):
    return [request, exception]

resps = grequests.imap(greqs, stream=True, size=200, exception_handler=fqecp)

for resp in resps:
    if isinstance(resp, list):  # handle err
        rd = json.loads(resp[0].kwargs["headers"].get("data", None))
        gerrs[rd['type']].append(rd)
    else:
        rd = json.loads(resp.request.raw_info)
        if resp.status_code != 200:
            gerrs[rd['type']].append(rd)
            print("non-200 : %s" % resp.status_code)
        else:
            pass
            # print("%s check ok" % resp.url)
        resp.close()

it workd fine and prety faster than just requests,but by checking net usage,it seems like stream=True not effect last 5 low: it semms like keep download until close() . i tried requests in shell with stream=True,code like:

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

class ExInfoAdapter(HTTPAdapter):
    def send(self, request, **kwargs):
        request.raw_info = request.headers.pop("data", None)
        return super(ExInfoAdapter, self).send(request, **kwargs)

s=requests.Session()
retries = Retry(total=2, backoff_factor=0, status_forcelist=[500, 502, 503, 504, 404], raise_on_redirect=False)
s.mount('http://', ExInfoAdapter(max_retries=retries, pool_connections=200, pool_maxsize=400))
s.mount('https://', ExInfoAdapter(max_retries=retries, pool_connections=200, pool_maxsize=400))

r=s.get("http://116.207.172.73/live/2358590143_1603765853.flv",headers={"Host": "v2.zb.marketing.i.mi.com"},timeout=15,stream=True)
assert r.status_code == 200
r.close()
# this ip and url may unavilable when you try,please find another live stream or call me to get a new one

here 's the result: i tryedboth grequests and requests 2 times , grequests download avg 3MB(2.8and3.2) and requests download avg 322KB(321and323) 323KB is mach accessable but still a lot,i just want to chek http code ,it's still a lot where's my fault and how to resolve this issue?

first of all. thanks for your work. when I use the grequests the RAM will increase slowly. code such as :

def exception_handler(request, exception):
    print("Request failed request:{} \n exception:{} ".format(request,exception))

if __name__ == '__main__':
    task = []
    f_file= "./data_scp/3031_xiuxiu_coverImage_v1.dat"

    session = requests.session()
    with open(f_file,"r") as r_f:
        for i in r_f:
            tmp = i.strip("\n").split(",")
            url = tmp[-1]
            feed_id = tmp[0]
            rs = grequests.request("GET", url,session=session)
            task.append(rs)

    resp = grequests.imap(task, size=30,exception_handler=exception_handler)

    for i in resp:
        if i.status_code ==200:
            print(i.status_code)

the 3031_xiuxiu_coverImage_v1.dat such as 6650058925696645684,http://***8.jpg 6650058925696645684,http://***8.jpg 6650058925696645684,http://***8.jpg 6650058925696645684,http://***8.jpg

my grequest version is 0.4.0 . thanks in advance