dlang-requests - type python-requests, only for D (part 2)
Good day!
In the second part of the article I will describe the use of the dlang-requests library for less standard cases.
')
At a lower level of the library is the Request structure, which provides all the functionality of the library.
Request, among other methods, has get and post methods used by getContent () and postContent (), described in the first part. The parameters of their call are the same as the parameters for getContent () and postContent (). Why is it needed, this structure?
To begin, I will describe how through its methods you can control the execution of queries.
The python-requests feature is streaming - the user receives a response from the server not at the end, but in the process of receiving the document. For receiving and processing large documents, this method can help save memory. python-requests allows you to receive a document as an iterator. For D, it was natural to use InputRange. In this case, we can use the answer not only for data acquisition, but also for direct use with algorithms working with InputRange.
Here you can see that the element type for stream is an array of bytes. Therefore, in the following character-number counting code, the use of a joiner is required:
In addition to processing large documents on the fly, streaming provides the easiest way to save documents to disk.
Streaming is valid not only for GET requests, but also for any requests that return a document along with code 200.
All the methods listed so far ultimately use the Request.exec (method) template method, which, in addition to the template parameter controlling the HTTP method, accepts all those combinations of parameters that were mentioned earlier.
In the same way, you can call any HTTP methods.
This concludes the second part of the article.
Just in case, once again, the link to the project page on Github
Good luck to everyone and enjoy programming!
In the second part of the article I will describe the use of the dlang-requests library for less standard cases.
')
Request and Response
At a lower level of the library is the Request structure, which provides all the functionality of the library.
Request, among other methods, has get and post methods used by getContent () and postContent (), described in the first part. The parameters of their call are the same as the parameters for getContent () and postContent (). Why is it needed, this structure?
To begin, I will describe how through its methods you can control the execution of queries.
- verbosity - setting it to 0,1,2 allows you to increase the detail of the output in stdout.An example of an increase in verbosity
import std.stdio; import requests; void main() { auto rq = Request(); rq.verbosity = 2; auto rs = rq.get("http://httpbin.org/get", ["a":"b"]); writeln(rs.responseBody); }> GET /get?a=b HTTP/1.1 > Connection: Keep-Alive > User-Agent: dlang-requests > Accept-Encoding: gzip, deflate > Host: httpbin.org > < HTTP/1.1 200 OK < server: nginx < date: Sat, 25 Jun 2016 13:37:20 GMT < content-type: application/json < content-length: 229 < connection: keep-alive < access-control-allow-origin: * < access-control-allow-credentials: true < 229 bytes of body received >> Connect time: 143 ms and 666 μs >> Request send time: 304 μs >> Response recv time: 144 ms and 121 μs { "args": { "a": "b" }, "headers": { "Accept-Encoding": "gzip, deflate", "Host": "httpbin.org", "User-Agent": "dlang-requests" }, "origin": "xxx.xxx.xxx.xxx", "url": "http://httpbin.org/get?a=b" } - timeout - sets the timeout for I / O operations and connection setup.An example of setting a timeout.
auto rq = Request();
rq.timeout = 30.seconds; - maxRedirects. Controls the allowable amount of redirects when executing queries.An example of limiting the number of redirects
import std.stdio; import requests; void main() { auto rq = Request(); rq.verbosity = 2; rq.maxRedirects = 2; auto rs = rq.get("https://httpbin.org/absolute-redirect/3"); writeln(rs.code); }
Conclusion:> GET /absolute-redirect/3 HTTP/1.1 > Connection: Keep-Alive > User-Agent: dlang-requests > Accept-Encoding: gzip, deflate > Host: httpbin.org > < HTTP/1.1 302 FOUND < server: nginx < date: Sat, 25 Jun 2016 13:54:16 GMT < content-type: text/html; charset=utf-8 < content-length: 283 < connection: keep-alive < location: http://httpbin.org/absolute-redirect/2 < access-control-allow-origin: * < access-control-allow-credentials: true < 283 bytes of body received >> Connect time: 505 ms and 705 μs >> Request send time: 247 μs >> Response recv time: 145 ms and 626 μs > GET /absolute-redirect/2 HTTP/1.1 > Connection: Keep-Alive > User-Agent: dlang-requests > Accept-Encoding: gzip, deflate > Host: httpbin.org > < HTTP/1.1 302 FOUND < server: nginx < date: Sat, 25 Jun 2016 13:54:16 GMT < content-type: text/html; charset=utf-8 < content-length: 283 < connection: keep-alive < location: http://httpbin.org/absolute-redirect/1 < access-control-allow-origin: * < access-control-allow-credentials: true < 283 bytes of body received >> Connect time: 135 ms and 621 μs >> Request send time: 128 μs >> Response recv time: 136 ms and 689 μs > GET /absolute-redirect/1 HTTP/1.1 > Connection: Keep-Alive > User-Agent: dlang-requests > Accept-Encoding: gzip, deflate > Host: httpbin.org > < HTTP/1.1 302 FOUND < server: nginx < date: Sat, 25 Jun 2016 13:54:16 GMT < content-type: text/html; charset=utf-8 < content-length: 251 < connection: keep-alive < location: http://httpbin.org/get < access-control-allow-origin: * < access-control-allow-credentials: true < 251 bytes of body received >> Connect time: 2 μs >> Request send time: 140 μs >> Response recv time: 136 ms and 279 μs 302 - authenticator - allows you to manage authorization in requests.Example
import std.stdio; import requests; void main() { auto rq = Request(); rq.verbosity = 2; rq.authenticator = new BasicAuthentication("user", "passwd"); rs = rq.get("http://httpbin.org/basic-auth/user/passwd"); assert(rs.code==200); } - bufferSize - sets the size of the read buffer (in bytes).
- proxy - allows you to set a proxy for requests in the form
http: // host: port /
Content streaming
The python-requests feature is streaming - the user receives a response from the server not at the end, but in the process of receiving the document. For receiving and processing large documents, this method can help save memory. python-requests allows you to receive a document as an iterator. For D, it was natural to use InputRange. In this case, we can use the answer not only for data acquisition, but also for direct use with algorithms working with InputRange.
import std.stdio; import std.format; import requests; void main() { auto rq = Request(); rq.useStreaming = true; rq.verbosity = 2; auto rs = rq.get("https://api.github.com/search/repositories?order=desc&sort=updated&q=language:D"); if ( rs.code == 200 ) { auto stream = rs.receiveAsRange(); while( !stream.empty ) { writefln("portion of %d bytes received".format(stream.front.length)); stream.popFront; } } } conclusion
> GET /search/repositories?order=desc&sort=updated&q=language:D HTTP/1.1 > Connection: Keep-Alive > User-Agent: dlang-requests > Accept-Encoding: gzip, deflate > Host: api.github.com > < HTTP/1.1 200 OK < server: GitHub.com < date: Sat, 25 Jun 2016 15:45:28 GMT < content-type: application/json; charset=utf-8 < transfer-encoding: chunked < content-encoding: gzip < x-github-request-id: B077660C:560B:7F2F21:576EA717 < 277 bytes of body received < 1370 bytes of body received portion of 751 bytes received portion of 2988 bytes received portion of 4632 bytes received portion of 6002 bytes received portion of 7474 bytes received portion of 9106 bytes received portion of 10246 bytes received portion of 11356 bytes received portion of 12290 bytes received portion of 12870 bytes received portion of 63904 bytes received Here you can see that the element type for stream is an array of bytes. Therefore, in the following character-number counting code, the use of a joiner is required:
import std.stdio; import std.ascii; import std.algorithm; import requests; void main() { auto rq = Request(); rq.useStreaming = true; auto stream = rq.get("https://api.github.com/search/repositories?order=desc&sort=updated&q=language:D").receiveAsRange; writeln(stream.joiner.filter!isDigit.count); } In addition to processing large documents on the fly, streaming provides the easiest way to save documents to disk.
Streaming is valid not only for GET requests, but also for any requests that return a document along with code 200.
Methods PUT / DELETE / HEAD ...
All the methods listed so far ultimately use the Request.exec (method) template method, which, in addition to the template parameter controlling the HTTP method, accepts all those combinations of parameters that were mentioned earlier.
import std.stdio; import std.ascii; import std.range; import std.algorithm; import requests; void main() { auto rq = Request(); rq.useStreaming = true; auto rs = rq.exec!"HEAD"("https://api.github.com/search/repositories?order=desc&sort=updated&q=language:D"); rs.code.writeln; rs.responseHeaders. byKeyValue. take(5). each!(p=>writeln(p.key, ": ", p.value)); } Conclusion
200
x-frame-options: deny
cache-control: no-cache
x-xss-protection: 1; mode = block
vary: Accept-Encoding
content-type: application / json; charset = utf-8
x-frame-options: deny
cache-control: no-cache
x-xss-protection: 1; mode = block
vary: Accept-Encoding
content-type: application / json; charset = utf-8
In the same way, you can call any HTTP methods.
This concludes the second part of the article.
Just in case, once again, the link to the project page on Github
Good luck to everyone and enjoy programming!
Source: https://habr.com/ru/post/304106/
All Articles