xNet - C # library for working with the Web
Gradually, with the study of C # and the .NET Framework , I began writing various helperes that hid the routine code behind calling just one method. After that, it turned into the development of a full-fledged library that I want to present to you. This library is written completely from scratch.
xNet is a class library for the .NET Framework , which includes:
Download: releases
Sources: github.com/X-rus/xNet
The article was significantly rewritten in connection with the release of the new version of the library. Last updated 09/12/2015
')
To send requests using HttpRequest . You can set various settings in it: headers, waiting time, whether to follow the redirection, whether to keep a permanent connection and others. When you send a request, it first accepts the response headers using HttpResponse , and then returns a link to this object so that you can download the message body. You can download the message body using one of the special methods.
Request example:
The using statement is used to close the connection to the server. The same can be done by calling the HttpRequest.Close method.
You can also get a link to HttpResponse using the HttpRequest.Response property. The returned HttpResponse is not unique. For each HttpRequest, there is only one HttpResponse (it is used repeatedly with each request).
Sample GET request:
RequestParams is used to set URL parameters. In this case, this is equivalent to the following entry:
You can set the URL parameters using the HttpRequest.AddUrlParam method. These parameters will be erased after the first request.
Example of a GET request with temporary URL parameters:
POST request example:
You can set the request parameters using HttpRequest.AddParam . These parameters will be erased after the first request.
An example of a POST request with temporary parameters:
You can also send a string, an array of bytes, a file, or any object that inherits from HttpContent .
To load the message body, use one of five HttpResponse methods: ToString , ToBytes , ToFile , ToStream, or ToMemoryStream . If the message body is not needed, then the method None should be called.
If you want the request parameters to be encoded, say, in 1251 encoding, then you need to set the HttpRequest.CharacterSet property = Encoding.GetEncoding (1251)
About Multipart / form data . To work with them, the MultipartContent class is used.
Example Multipart / form request:
In order not to create the object of the MultipartContent class, you can use two special HttpRequest methods: AddField and AddFile . These parameters will be erased after the first request.
Example Multipart / form request with temporary parameters:
HttpRequest supports persistent connections. They can be disabled using the HttpRequest.KeepAlive property.
Example:
All three requests will be made using the same connection, but this is only if the server does not close the connection earlier. Please note - in the last query I do not call the method None . The fact is that it downloads the message body so that the next request can be correctly executed when using a persistent connection, and since this is the last request, you can not download the message body, because the connection will still be closed.
A little more about the method None . If the persistent connection is not used, then this method simply closes the connection. If you do not call this method, it will be called automatically upon the next request. I recommend calling this method yourself.
Keep in mind that when working with persistent connections, disconnections from the server may occur. This can occur due to the fact that the server has reached the limit of requests for one connection, or it has been time to wait for the next request. HttpRequest is able to handle such situations by trying to reconnect (this behavior is not related to Reconnect ). If you fail to connect, it will throw an exception. For more fine-tuning of persistent connections, you can use the KeepAliveTimeout and MaximumKeepAliveRequests properties .
Additionally, you can manage reconnects using the Reconnect , ReconnectLimit, and ReconnectDelay properties. By default, this behavior is disabled. HttpRequest will try to reconnect if an error occurs while connecting to the server, or if an error occurs while loading / sending data. But reconnection will not work if an error occurs while loading the message body (in the ToString () method, for example). This behavior can be useful when working with a proxy or if you have an unstable internet connection.
Installation of additional headers is performed using a special indexer. Keep in mind that some headers can only be set using special properties. Their list can be found in the documentation. Headers specified via the indexer are sent in all requests. If you need to set a temporary header for a single request, then use the HttpRequest.AddHeader method. Such headers overlap the headers specified through the indexer.
Cookies are set using the HttpRequest.Cookies property. Cookies may vary depending on server response. To prevent this, you must set the CookieDictionary.IsLocked property to true .
Example:
HttpRequest and HttpResponse have additional methods for working with cookies and headers: ContainsHeader , EnumerateHeaders , ClearAllHeaders , ContainsCookie , ContainsRawCookie and others.
To send requests you need to set the User-Agent. It can be generated by one of the methods Http : IEUserAgent , OperaUserAgent , ChromeUserAgent , FirefoxUserAgent or OperaMiniUserAgent .
And ask this:
Previously, there was the RandomUserAgent method, but I deleted it, since its use can lead to some problems. Suppose a site for Google Chrome provides one content, and for IE it’s a little different, or completely different. Therefore, carefully use User Agents from different browsers.
HttpRequest has many more different properties with which you can manipulate the headers sent. See the documentation.
xNet supports the following types of proxy servers:
All of these classes inherit from the ProxyClient class. There is an additional class ChainProxyClient , which allows you to create a connection through a chain of proxy servers.
Example:
HttpRequest supports work through a proxy server. To do this, set the HttpRequest.Proxy property.
If an error occurs while working with the HTTP protocol, a HttpException will be raised, and if an error occurs while working with a proxy server, a ProxyException will be raised. Both of these exceptions inherit from the class NetException .
Example:
The WinInet class allows you to interact with the network settings of the Windows operating system. With the help of it you can find out whether the computer is connected to the Internet or get the value of the Internet Explorer proxy server.
The Html class is intended to aid in working with HTML and other textual data. It contains a method that replaces HTML text in the text with the characters that represent them. Example: & _quot; test & _quot; in "test". A method that replaces Unicode entities with the characters representing them. Example: \ u2320 or \ U044F. And methods for extracting strings: Substring, LastSubstring, Substrings.
So what is xNet?
xNet is a class library for the .NET Framework , which includes:
- Classes for working with proxy servers: HTTP, Socks4 (a), Socks5, Chain .
- Classes for working with HTTP 1.0 / 1.1 protocol: keep-alive, gzip, deflate, chunked, SSL, proxy, and more .
Download: releases
Sources: github.com/X-rus/xNet
The article was significantly rewritten in connection with the release of the new version of the library. Last updated 09/12/2015
')
The basics
To send requests using HttpRequest . You can set various settings in it: headers, waiting time, whether to follow the redirection, whether to keep a permanent connection and others. When you send a request, it first accepts the response headers using HttpResponse , and then returns a link to this object so that you can download the message body. You can download the message body using one of the special methods.
Request example:
using (var request = new HttpRequest()) { request.UserAgent = Http.ChromeUserAgent(); // . HttpResponse response = request.Get("habrahabr.ru"); // . string content = response.ToString(); } The using statement is used to close the connection to the server. The same can be done by calling the HttpRequest.Close method.
You can also get a link to HttpResponse using the HttpRequest.Response property. The returned HttpResponse is not unique. For each HttpRequest, there is only one HttpResponse (it is used repeatedly with each request).
This is what this query looks like.
GET / HTTP/1.1 Host: habrahabr.ru Connection: keep-alive Accept-Encoding: gzip,deflate User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/24.0.1309.0 Safari/537.17 Sending simple requests and loading the message body
Sample GET request:
using (var request = new HttpRequest()) { var urlParams = new RequestParams(); urlParams["param1"] = "val1"; urlParams["param2"] = "val2"; string content = request.Get("habrahabr.ru", urlParams).ToString(); } RequestParams is used to set URL parameters. In this case, this is equivalent to the following entry:
Get("habrahabr.ru/?param1=val1¶m2=val2") You can set the URL parameters using the HttpRequest.AddUrlParam method. These parameters will be erased after the first request.
Example of a GET request with temporary URL parameters:
using (var request = new HttpRequest()) { request.AddUrlParam("param1", "val1").AddUrlParam("param2", "val2"); string content = request.Get("habrahabr.ru").ToString(); } POST request example:
using (var request = new HttpRequest()) { var reqParams = new RequestParams(); reqParams["login"] = "neo"; reqParams["password"] = "knockknock"; string content = request.Post( "www.whitehouse.gov", reqParams).ToString(); } You can set the request parameters using HttpRequest.AddParam . These parameters will be erased after the first request.
An example of a POST request with temporary parameters:
using (var request = new HttpRequest()) { request.AddParam("login", "neo").AddParam("password", "knockknock"); string content = request.Post("www.whitehouse.gov").ToString(); } You can also send a string, an array of bytes, a file, or any object that inherits from HttpContent .
To load the message body, use one of five HttpResponse methods: ToString , ToBytes , ToFile , ToStream, or ToMemoryStream . If the message body is not needed, then the method None should be called.
If you want the request parameters to be encoded, say, in 1251 encoding, then you need to set the HttpRequest.CharacterSet property = Encoding.GetEncoding (1251)
Sending Multipart / form data
About Multipart / form data . To work with them, the MultipartContent class is used.
Example Multipart / form request:
using (var request = new HttpRequest()) { var multipartContent = new MultipartContent() { {new StringContent("Bill Gates"), "login"}, {new StringContent("qwerthahaha"), "password"}, {new FileContent(@"C:\windows_9_alpha.rar"), "file1", "1.rar"} }; request.Post("www.microsoft.com", multipartContent).None(); } In order not to create the object of the MultipartContent class, you can use two special HttpRequest methods: AddField and AddFile . These parameters will be erased after the first request.
Example Multipart / form request with temporary parameters:
using (var request = new HttpRequest()) { request .AddField("login", "Bill Gates") .AddField("password", "qwerthahaha") .AddFile("file1", @"C:\windows_9_alpha.rar"); request.Post("www.microsoft.com").None(); } Persistent connection and reconnection
HttpRequest supports persistent connections. They can be disabled using the HttpRequest.KeepAlive property.
Example:
using (var request = new HttpRequest("habrahabr.ru")) { request.Get("/").None(); request.Get("/feed").None(); request.Get("/feed/posts"); } All three requests will be made using the same connection, but this is only if the server does not close the connection earlier. Please note - in the last query I do not call the method None . The fact is that it downloads the message body so that the next request can be correctly executed when using a persistent connection, and since this is the last request, you can not download the message body, because the connection will still be closed.
A little more about the method None . If the persistent connection is not used, then this method simply closes the connection. If you do not call this method, it will be called automatically upon the next request. I recommend calling this method yourself.
Keep in mind that when working with persistent connections, disconnections from the server may occur. This can occur due to the fact that the server has reached the limit of requests for one connection, or it has been time to wait for the next request. HttpRequest is able to handle such situations by trying to reconnect (this behavior is not related to Reconnect ). If you fail to connect, it will throw an exception. For more fine-tuning of persistent connections, you can use the KeepAliveTimeout and MaximumKeepAliveRequests properties .
Additionally, you can manage reconnects using the Reconnect , ReconnectLimit, and ReconnectDelay properties. By default, this behavior is disabled. HttpRequest will try to reconnect if an error occurs while connecting to the server, or if an error occurs while loading / sending data. But reconnection will not work if an error occurs while loading the message body (in the ToString () method, for example). This behavior can be useful when working with a proxy or if you have an unstable internet connection.
Setting cookies, headers and other
Installation of additional headers is performed using a special indexer. Keep in mind that some headers can only be set using special properties. Their list can be found in the documentation. Headers specified via the indexer are sent in all requests. If you need to set a temporary header for a single request, then use the HttpRequest.AddHeader method. Such headers overlap the headers specified through the indexer.
Cookies are set using the HttpRequest.Cookies property. Cookies may vary depending on server response. To prevent this, you must set the CookieDictionary.IsLocked property to true .
Example:
using (var request = new HttpRequest("habrahabr.ru")) { request.Cookies = new CookieDictionary() { {"hash", "yrttsumi"}, {"super-hash", "df56ghd"} }; request[HttpHeader.DNT] = "1"; request["X-Secret-Param"] = "UFO"; request.AddHeader("X-Tmp-Secret-Param", "42"); request.AddHeader(HttpHeader.Referer, "http://site.com"); request.Get("/"); } HttpRequest and HttpResponse have additional methods for working with cookies and headers: ContainsHeader , EnumerateHeaders , ClearAllHeaders , ContainsCookie , ContainsRawCookie and others.
To send requests you need to set the User-Agent. It can be generated by one of the methods Http : IEUserAgent , OperaUserAgent , ChromeUserAgent , FirefoxUserAgent or OperaMiniUserAgent .
And ask this:
request.UserAgent = Http.ChromeUserAgent(); Previously, there was the RandomUserAgent method, but I deleted it, since its use can lead to some problems. Suppose a site for Google Chrome provides one content, and for IE it’s a little different, or completely different. Therefore, carefully use User Agents from different browsers.
HttpRequest has many more different properties with which you can manipulate the headers sent. See the documentation.
Connection through a proxy server
xNet supports the following types of proxy servers:
- HTTP - class HttpProxyClient
- Socks4 (a) - Socks4ProxyClient (a) class
- Socks5 - class Socks5ProxyClient
All of these classes inherit from the ProxyClient class. There is an additional class ChainProxyClient , which allows you to create a connection through a chain of proxy servers.
Example:
var proxyClient = HttpProxyClient.Parse("127.0.0.1:8080"); var tcpClient = proxyClient.CreateConnection("habrahabr.ru", 80); // . HttpRequest supports work through a proxy server. To do this, set the HttpRequest.Proxy property.
Error processing
If an error occurs while working with the HTTP protocol, a HttpException will be raised, and if an error occurs while working with a proxy server, a ProxyException will be raised. Both of these exceptions inherit from the class NetException .
Example:
try { using (var request = new HttpRequest()) { request.Proxy = Socks5ProxyClient.Parse("127.0.0.1:1080"); request.Get("habrahabr.ru"); } } catch (HttpException ex) { Console.WriteLine(" HTTP-: {0}", ex.Message); switch (ex.Status) { case HttpExceptionStatus.Other: Console.WriteLine(" "); break; case HttpExceptionStatus.ProtocolError: Console.WriteLine(" : {0}", (int)ex.HttpStatusCode); break; case HttpExceptionStatus.ConnectFailure: Console.WriteLine(" HTTP-."); break; case HttpExceptionStatus.SendFailure: Console.WriteLine(" HTTP-."); break; case HttpExceptionStatus.ReceiveFailure: Console.WriteLine(" HTTP-."); break; } } Additional classes
The WinInet class allows you to interact with the network settings of the Windows operating system. With the help of it you can find out whether the computer is connected to the Internet or get the value of the Internet Explorer proxy server.
The Html class is intended to aid in working with HTML and other textual data. It contains a method that replaces HTML text in the text with the characters that represent them. Example: & _quot; test & _quot; in "test". A method that replaces Unicode entities with the characters representing them. Example: \ u2320 or \ U044F. And methods for extracting strings: Substring, LastSubstring, Substrings.
Related Links
Source: https://habr.com/ru/post/146475/
All Articles