Buying the best apartment with R

Many people are faced with the issue of buying or selling real estate, and an important criterion here is how not to buy more expensive or not to sell cheaper relative to other, comparable options. The simplest method is comparative, focusing on the average price of a meter in a particular place and expertly adding or reducing the percentage of the cost for the advantages and disadvantages of a particular apartment. But this approach is laborious, inaccurate and will not allow to take into account the diversity of the differences of apartments from each other. Therefore, I decided to automate the process of choosing a property using data analysis by predicting the “fair” price. This publication describes the main stages of this analysis, selected the best predictive model from eighteen tested models based on three quality criteria, and the best (undervalued) apartments are immediately marked on the map, all using one web application created with R.

Data collection

With the formulation of the problem it is clear, the question arises, where do you get the data, there are several main real estate search sites in the Russian Federation, and there is a WinNER database, which in a fairly user-friendly interface contains the maximum number of ads and allows you to upload in csv format. But if earlier this base allowed time-based access (minutes, hours, days), then now the minimum is only 3 months access, which is somewhat redundant for an ordinary buyer or seller (although if seriously approached, then these costs can be incurred). With this option it would be quite simple, so let's go the other way, by parsing from some real estate site. Of the several options, I chose the most convenient well-known cian.ru. At that time, the display of the data was presented in a simple tabular form, but when I tried to parse all the data on R, then I had a failure. In some cells, the data could be underfilled, but with the standard means of the R functions I could not catch hold of anchor words or symbols, and I had to either use cycles (which is wrong for this task in R) or use regular expressions with which I am not at all familiar so I had to use an alternative. This alternative was a great service import.io , which allows you to extract information from pages (or sites), has its own REST API and gives the result to JSON. And R can and request to API to carry out and JSON to sort.
Quickly accustomed to this service, I created an extractor on it, which extracts all the required information (all parameters of each apartment) from one page. And R already goes through all the pages, calling this API for each and gluing the received JSON data into a single table. Although import.io allows you to make a completely autonomous API, which would pass through all pages, I thought it was logical that I could not do (parsing one page) correctly put on a third-party API, and continue to do everything else in R. So, The data source is selected, now about the limitations of the future model:

1. Moscow city
2. One type of apartment per model (that is, one-room or two-room or three-room)
3. Within one metro station (as geocoding is used, and the distance to the nearest metro)
4. Secondary (since the new buildings are all fundamentally different, and qualitatively differ from each other, and in the ads it either does not say, or something is written in the comments, but neither one nor the other, does not allow creating an adequate model)

')

Data overview

As is often the case, data may contain outliers, omissions, or simply deception, when new buildings are issued as a secondary housing, or they sell land in general.
Therefore, initially it is necessary to bring all the data to a "neat" form.

Cheating

Here, the main focus on the description of the announcement, if there are words clearly not related to the sample I need, these observations are excluded, this is verified by the function R grep . And since in R the computations are vector, this function immediately returns the vector of valid observation values, applying it, we filter the sample.

Check for missing values

The missing values ​​are quite rare (and mainly in the "left" ads, with new buildings and land, which by this time have already been excluded), but with those that remain, it is necessary to do something. There are two options, in general, to exclude such observations or to replace these missing values. Since I did not want to sacrifice observations, but on the assumption that the data is not filled on the basis of the principle “that it is filled in, and so it is clear that this is like everything around,” I decided to replace the qualitative variables with the mode of their values, and median values. Of course, this is not entirely correct, and it would be entirely correct to conduct a correlation relationship between observations, and fill in the gaps according to the results, but for this task I considered this to be redundant, moreover, there were few such observations.

Emission check

Emissions are even less common, and can only be quantitative, namely by price and by area. Here, it was based on the assumption that the buyer (I) specifically knows at what price and about what footage his apartment should be, therefore, by setting the initial values ​​of the upper and lower prices, by limiting the footage, we automatically get rid of emissions. But even if this is not the case (not to make restrictions), then upon receiving the result or by looking at the scatterplot and seeing that there is an outlier, you can make a request with updated data, thereby removing these observations, which will improve the model.

Basic assumptions of the Gauss-Markov theorem

Set of regressors

In addition to the variables (obtained directly from the site), I decided to add additional regressors, namely, a five-story house or not (since this is usually a very qualitative difference), and the distance to the nearest metro station (similarly). The Google geocoding service API is used to determine the distance (I chose it as the most accurate, loyal to restrictions, and there is a ready function in R), first the addresses of apartments and subways are geocoded, the geocode function from the ggmap package. And the distance is determined by the formula of haversinus, the finished function distHaversine from the package geosphere .
The total number of regressors was 14 pieces:

1. Distance to metro
2. total area
3. Living space
4. Kitchen area
5. House type
6. Availability and types of elevators
7. Availability and types of balconies
8. Number and types of bathrooms
9. Where the windows go
10. Phone availability
11. Type of sale
12. Ground floor
13. Top floor
14. Five-story building

Tested Predictive Analysis Models

Model performance criteria

Model test results

Map Display

The winner is determined (random forest), this model predicts “theoretical” prices for all observations with minimal errors. Now you can calculate the absolute, relative underestimation and output the result in the form of a sorted table, but in addition to the table view, you immediately want information, so we will display some of the best results on the map immediately. And for this purpose in R there is a googleVis package intended for integration with Google mapping system (however, there is a package for Leaflet too). I continued to use Google as well, since the obtained coordinates from their geocoding are forbidden to be displayed on other maps. The display on the map is carried out by a single function gvisMap from the googleVis package.

Web GUI

Transferring all the required parameters via the console is slow and inconvenient, so I wanted to do everything automatically. And traditionally, for this, you can again use R with the shiny , and shinydashboard frameworks , which have sufficient I / O controls.


The result of all this becomes a convenient GUI application, with actually two (the remaining items for control) the main items on the side menu - the first and the last. In the first ( Source data ) item of the side menu (Fig. 1), all required parameters are set (similar to cian) for searching and evaluating apartments.


Fig.1 Window of the selected menu Source data

The remaining items on the side menu display:

• Summary Report on Regressors
• data tables ( Raw ( Raw data ) - original after parsing, neat ( Tidy data ) - after adjusting the parameters to a neat look and adjusting the parameters and adding geolocation, and the final ( Predict data ) table with the predicted prices)
• three diagrams ( Plots ) (Fig. 2) —the accuracy of the model and the importance of the regressors in the random forest algorithm (almost always all the regressors are important) and the scatter diagram of the original and predicted prices.

Fig.2 Window of the selected menu Plots

Well, in the last paragraph ( Result map ) (Fig. 3), this is the reason for which everything was started, a map with the best results selected and a table with the calculated predicted price and the main characteristics of the apartments.


Fig.3 Window of the selected menu Result map

Also in this table immediately there is a link (*) to go to this ad. This integration (the inclusion of JS elements in the table) allows you to make a package DT .

Conclusion

Summarizing all the above, how does it all work:

1. On the first page using the controls set the original request
2. Based on the selection of these input elements, a query string is formed (also indicated as a hyperlink for verification)
3. This line with the page is passed to the import.io API (in the process of creating all of this, cian began to change the output interface, thanks to import.io, I retrained the extractor within literally 5 minutes)
4. Processing received JSON from API
5. All pages are scrolled (in the process of work, the status bar by processes is displayed)
6. Tables are glued together, checked (incorrect values ​​are eliminated, missing values ​​are replaced), are given in a uniform form suitable for analysis
7. Geocoding addresses and determining distances
8. Model is built using random forest algorithm.
9. Predicted prices, absolute, relative deviations are determined.
10. The best results are displayed on the map and in the table below (the number of apartments to be displayed is indicated on the first page)

The entire operation of the application (from the beginning of the request to display on the map) is performed in less than a minute (most of the time is spent on geolocation, the restriction of Google for residential use).

This publication wanted to show how for simple domestic needs, in the framework of one application, we managed to solve many small, but fundamentally different interesting subtasks:

• crawling
• parsing
• integration with third-party API
• JSON processing
• geocoding
• working with different regression models
• evaluation of their effectiveness in different ways
• geolocation
• map display

Everything else is implemented in a convenient graphical application, which can be both local and hosted on the network, and all this is done on one R (not counting import.io ), with a minimum of lines of code with a simple and elegant syntax. Of course, something is not taken into account, for example, the house near the highway or the condition of the apartment (since there is no such thing in the ads), but the final, ranked list of options immediately displayed on the map and referring to the original ad itself, greatly facilitate the choice of apartments, Well, plus to everything I learned a lot of new things in R.