Create a mass of asynchronous requests using Grequests

Requests good, but grequests better. I don’t know better, more efficiently a library that can quickly and elegantly perform HTTP requests rather than requests, this library is an undoubted leader in this respect.

But since it is lame with asynchrony, it is possible to perform asynchronous requests using threading or gevent.

Wrote grequests, the same author wrote requests. Only using gevent + requests. I will not spend too much time on the topic, I will give you detailed information about this library.

Grequests is an asynchronous wrapper over regular requests.
')
Let's make an ordinary POST request for multiple URLs:

import grequests with open("C:\\path\\urls.txt") as werewolves: array = [row.strip() for row in werewolves] params = {'a':'b', 'c':'d'} rs = [grequests.post(u, data=params) for u in array] for r in grequests.imap(rs, size=16) print(r[0].status_code, r[0].url) 

Everything is quite simple, the library is imported, the file is opened for reading, a list is created, and the params variable is assigned the values ​​a: b, c: d.

Next, we create a variable rs that will be responsible for the POST request itself, for the variable r we create grequests.map ([rs], size = is an asynchronous value, the higher the value, the faster the http requests will be executed, though more than 16 it does not make sense to set) .

Now, since we passed all the arguments to the variable r, that is, in grequests.imap () we can interact with this variable as in normal requests.

And the last step we need to display all the status code, url address, also rs is a list, we do this so that there are no indexing errors by type:

TypeError: 'Response' object does not support indexing

If you have a smooth traceback with this error, I suggest the option:

 def exception_handlerr(request, exception): print("Request failed", request.url) import grequests with open("C:\\path\\urls.txt") as werewolves: array = [row.strip() for row in werewolves] params = {'a':'b', 'c':'d'} rs = [grequests.post(u, data=params) for u in array] for r in grequests.map([rs], size=16, exception_handler=exception_handlerr) print(r[0].status_code, r[0].url) 

Now we will also refer to the variable r as a list in order to avoid indexing errors.
The main steps we have done. You can "DUDA" server. Although, this library does not possess transcendental asynchrony. It is possible to have a look at Aiohttp .

I would also like to talk about exceptions in grequests. Since grequests does not use the error classes from requests, it does as follows:

 def send(self, **kwargs): """ Prepares request based on parameter passed to constructor and optional ``kwargs```. Then sends request and saves response to :attr:`response` :returns: ``Response`` """ merged_kwargs = {} merged_kwargs.update(self.kwargs) merged_kwargs.update(kwargs) try: self.response = self.session.request(self.method, self.url, **merged_kwargs) except Exception as e: self.exception = e self.traceback = traceback.format_exc() return self 

We catch by exception_handler:

 def exception_handlerr(request, exception): print("Request failed", request.url) # print(str(exception)) 

Full source code:

 def exception_handlerr(request, exception): print("Request failed", request.url) import grequests with open("C:\\path\\urls.txt") as werewolves: array = [row.strip() for row in werewolves] params = {'a':'b', 'c':'d'} rs = [grequests.post(u, data=params) for u in array] for r in grequests.map([rs], size=16, exception_handler=exception_handlerr) print(r.status_code, r.url) 

This way we will be able to catch errors with complete confidence.

With GET requests, everything is as easy as with POST requests:

 def exception_handlerr(request, exception): print("Request failed", request.url) import grequests with open("C:\\path\\urls.txt") as werewolves: array = [row.strip() for row in werewolves] rs = [grequests.get(u) for u in array] for r in grequests.map([rs], size=16, exception_handler=exception_handlerr) print(r.status_code, r.url) 

→ Source code grequests
→ Requests documentation

Have a nice day!