Simple web scraping on f #

Quite a legitimate question why such a hackneyed topic like web scraping and why f #. 1. on f # web scraping is much more interesting than on c # 2. I wanted to try as far as f # is applicable for developing not demo examples but something that really makes programs 3. F # has an interactive console, which when tinkering in the depths of HTML becomes just a salvation. Today, with the help of f # we will buy VW Touareg.

Touareg

In my opinion the most optimal car for the harsh winter and no less severe roads. Suppose we have a million four hundred thousand, a great desire and nothing more. There is also an auto.ru site, but my recent involvement in the purchase of a used car revealed several shortcomings: a. you need to constantly go to the correct section b. you need to constantly fill out a search form, which is especially annoying when you need to do it on the road from mobile devices, on the way to inspect the next candidate, we used the iPad and it was “not ice”, and from some kind of smartphone I would definitely shoot myself to perform all these operations. Total requirements: the program bypasses the offer page with the corresponding request, searches for new offers and if it finds it, sends an email with the parameters of the new offer (s) and at the same time a list of all the offers that satisfy the request so that you can compare visually.

General methods

Auto.ru is quite loyal to collecting its content, so there will be no perversions like emulation of pressing a button and slipping cookies, and all that interests us can be obtained via a direct url via GET. We will also send letters via gmail, which will require the client’s SMTP settings specified in the comments

module WebUtil = let LoadPage (x:WebRequest) = use resp = x.GetResponse() let rs = resp.GetResponseStream() let rr = new StreamReader(rs,System.Text.Encoding.GetEncoding(1251)) rr.ReadToEnd() let LoadPageByUrl (x:string) = let request = WebRequest.Create(x) LoadPage request let SentByMail (recepinet: string) (subj:string) (content: string) = let client = new SmtpClient() client.DeliveryMethod <- SmtpDeliveryMethod.Network use message = new MailMessage() message.To.Add(recepinet) message.Body <- content message.Subject <- subj message.IsBodyHtml <- true client.Send(message) (* <system.net> <smtp from="YourMail@gmail.com"> <network host="smtp.gmail.com" port="587" enableSsl="true" password="$ecretPa$$w0rd" defaultCredentials="false" userName="YourMail@gmail.com" /> </system.net> *)

')
What is here from f #? Functionally, nothing, the standard methods of the platform, but strikingly briefly, if it were not for the footcloth of the MailMessage properties setters. The readability of the code is very personal, but in my opinion there is little that can be compared with f # in readability.

Data structures

Because you need to distinguish which sentences were added since the last check, the result of the previous query will be stored in a file. It would be true to keep only the date of the last check, but then it would not be at all interesting and the topic of the serialization of complex objects would be missed. So records (records):

module CarParser = [<DataContract>] type Car = { [<field: DataMember(Name= "Year" ) >] Year: int; [<field: DataMember(Name= "Price" ) >] Price: int; [<field: DataMember(Name= "Model" ) >] Model: string; [<field: DataMember(Name= "Engine" ) >] Engine: string; [<field: DataMember(Name= "Url" ) >] Url: string; } [<DataContract>] type CarRequest = { [<field: DataMember(Name= "Email" ) >] Email:string; [<field: DataMember(Name= "RequestUrl" )>] RequestUrl: string; [<field: DataMember(Name= "Cars" ) >] Cars: Car list; }

Why additional attributes? The fact is that standard serialization in XML via the XmlSerializer does not work, because f # records has no constructor without parameters, which is mandatory. In this case, it will save the DataContractSerializer, the methods for serializing and deserializing to the file look like this:

open System; open System.IO; open System.Xml; open System.Runtime.Serialization; open System.Text.RegularExpressions; module SerializationUtils = let SerializeToFile (req: 'T) (fileName: string) = let xmlSerializer = DataContractSerializer(typeof<' T>); use fs = File.OpenWrite(fileName) xmlSerializer.WriteObject(fs, req) //T' will be calculated automatically let Deserialize< 'T> (fileName:string) = let xmlSerializer = DataContractSerializer(typeof<' T>); use fs = File.OpenRead(fileName) xmlSerializer.ReadObject(fs) :?> 'T

Content parsing

If we talk about the parameters of a particular car, then the priority is as follows: price, engine - how much tax I will pay and gasoline or diesel, a year - very indirectly indicating the state. If everything suits me, then you can look at the photo by clicking on the link, this moment, too, by the way, it would be interesting to remake and indicate the link to your site, which will show me the photos and the description of the car without any advertisements there. But back to the more urgent task.

Using the HTMLAgilityPack (in my opinion this is really cool - any .net libraries are available from f #) we get a table with sentences and further analysis is just a matter of technology. Again on f # parsing looks very short and clear, I know for sure that at least part of the next real project on collecting and analyzing content I will do on f # because it is much easier to read.

let private ParseCar (cnt: HtmlNode) = let columns = cnt.SelectNodes( "td" ) |> Seq.toList let model = columns.[0].InnerText let txt = columns.[1].InnerText let price = txt |> ( fun x -> Regex.Replace(x, "\\W" ,System.String.Empty)) |> Int32.Parse let url = columns.[0].ChildNodes |> Seq.find ( fun x -> x.Name.ToLower() = "a" ) |> ( fun x-> x.Attributes) |> Seq.find ( fun x -> x.Name = "href" ) |> ( fun x -> x.Value) let year = columns.[2].InnerText |> Int32.Parse let engine = columns.[3].InnerText let c: Car = { Year = year; Price = price; Model = model; Url = url; Engine = engine; } c let private ParsePage (node: HtmlNode) (parseCar: HtmlNode -> Car) = node.SelectNodes( "//div[@id='cars_sale']/table[@class='list']/descendant::tr[not(@class='header first')]" ) |> Seq.map parseCar

And several methods for summarizing the data obtained into interesting requests are probably only currying and initializing the record by copying from the old record. f # overrides the CompareTo, Equals and GetHashCode functions, so the comparison of records in this case works correctly and you can write x = y.

let private ParseCarNode x = ParsePage x ParseCar let private GetCars (cntnt:string) (pars: HtmlNode -> seq) = let doc = new HtmlDocument() doc.LoadHtml(cntnt) pars doc.DocumentNode let CreateCarRequest mail url = let cars = GetCars (LoadPageByUrl url) ParseCarNode { Email = mail; RequestUrl = url; Cars = cars |> List.ofSeq } let UpdateCarList (oldRequest: CarRequest) = let newCars = GetCars (LoadPageByUrl oldRequest.RequestUrl) ParseCarNode let isContains y = Seq.tryFind ( fun x -> x = y) let diff = newCars |> Seq.filter ( fun x -> (oldRequest.Cars |> isContains x) = None) let res = { oldRequest with Cars = newCars |> List.ofSeq } // , (res,diff)

Results

Overboard were the functions of formatting e-mail messages and the function that brings it all together and runs on a timer, but their implementation is obvious. Testing can be done on c #, in particular, testing the correctness of finding new machines was implemented with the help of Moles and it was there that the rakes described in this post surfaced.
Main advantages: 200 lines of code. Together. All 5 files. There are 30 percent less than the average c # file in programs that carry at least some kind of functional load, and not just call the framework methods with a different order of arguments. Readability code. The speed of development in programs collecting kontent very big advantage is the ability to execute code without compiling. In my opinion, f # is a more understandable and natural way of developing programs, familiar and routine tasks become interesting again. The main drawbacks: the main drawback is of course hand curves, because the program has no logging or clear error handling, which of course is unacceptable (but the truth is, we just buy a car and not sell software).
In any case, it is possible and necessary to write real world programs on f # and this task is simply not complex enough algorithmically and logically to show all the capabilities and advantages of the language, but even such tasks are not bad, and most importantly, interesting and fast enough, except for the time to learn the language .
PS: It remains to write the web interface and ask donation for subscribing to pay for sms gateway and start sending messages if Avto.ru does not ban me earlier :)

Source: https://habr.com/ru/post/112553/

All Articles