The steady beauty of indecent models

- Could you build us a statistical model?
- With pleasure. Can I look at your historical data?
- We have no data yet. But the model is still needed.

Familiar dialogue, is not it? Further, there are two possible scenarios:

A. “Then come when data is available.” The option will not be considered as trivial.
B. “Tell us which factors do you think are most important.” The rest of the article is about this.

Under the cut there is a story about what an improper model is, why their beauty is stable and what it costs. All on the example of the long-suffering data on the survival of the passengers of the Titanic.

Where does this strange name come from

From the article The Robust Beauty of Improving Linear Models in Decision Making of comrade Robyn M. Dawes, who in turn refers to the work of predecessors.
')

Proper linear models are those in which weights are assigned in such a way that the resulting linear combination predicts the quantity of interest in an optimal way. An example would be the usual linear regression, fitted by the method of least squares. [...]

Indecent linear models (improper linear models) - those in which the weights are determined optimally. For example, they are appointed on the basis of intuition, previous experience, or equal to one.

A typical linear model looks like this:

ŷ = β ₀ + β ₁ x ₁ + β ₂ x ₂ +… + β _m x x _m

Here is the estimate of the dependent variable, β ₀ is the free term, β ₁ ... β _m are the regression coefficients, x ₁ ... x _m are independent variables.

Due to the lack of data to determine the values ​​of β ₁ ... β _m, we will act voluntarily: assign them the values ​​of +1 (positive effect), -1 (negative effect) and 0 (no effect).

Far-fetched example

Typical examples of statistical models that were discussed at the beginning of the article are scoring cards for consumer lending (they predict the likelihood of a client’s deep delay or non-return of debt), response models (they predict the likelihood that a client will respond to an offer to buy or subscribe to product), outflow models (predict the likelihood that a customer will go to a competitor), etc.

This article will use a simpler data set, most likely already familiar to all interested in the topic of machine learning for readers: the list of passengers of the Titanic. Earlier on Habré already looked at him:

• Predicting Titanic Passenger Survival with Azure Machine Learning
• Titanic on Kaggle: you do not finish reading this post until the end

Competitors

• ImproperModel - regression with unit weights (topic of this article)
• ProperModel - logistic regression with the same independent variables.
• LogisticRegression - logistic regression with all independent variables
• RandomForest - one of the most common modern algorithms

How will we compare

The passenger list will be divided into two parts, on one we will fit the model, on the other to check the quality of its work.

There are no miracles and absolutely no knowledgeable model can be built without any knowledge. In this case, such knowledge would be gallantry - women and children were let forward when boating. Based on this prior knowledge for the regression with unit weights, we assign the variable female weight +1, and the age variable weight -1. In addition, passengers with second and third class tickets, who have cabins in the hold, continue to run to the boats, so the variable pclass is assigned a weight of -1.

In order not to get up two times, we will immediately normalize it - divide by the standard deviation in the training set. This is necessary to bring the independent variables to a comparable scale. It was possible to divide by the range and get a similar result.

Please note that in the single weights model we did not use information about who survived and who drowned , only the standard deviation of independent variables for scaling.

Repeat the experiment a thousand times and look at the results. The quality of the model will be measured using the Gini coefficient ( G1 ).

results

Model	Gini	Percentage of best
ImproperModel	0.639	90.4%
ProperModel	0.667	94.3%
LogisticRegression	0.679	96.0%
RandomForest	0.707	100%

Far-reaching conclusions

A pessimist will say that the model with unit weights ranked last in this comparison, and this is true. But it is also true that she gained 90% of the best result without using historical data , and lagged behind the usual logistic regression with the same independent variables by only 4% .

Why it happens

In the article with the unusual name Estimating Coefficients in Linear Models: its author (Howard Weiner) gives the following theorem of equal weights:

If k linearly independent variables x _i (i = 1, ..., k) with zero mean value and unit variance are used to predict the variable y , which is also scaled to zero mean and unit variance, and the standardized values ​​of the least squares regression coefficients β _i (i = 1, ..., k) are evenly distributed over the interval [0.25, 0.75] , then when going to equal weights (0.5), the expectation of reducing the proportion of dispersion of the dependent variable explained by the model will be less than k / 96 . Losses are even less if x _{i are} correlated with each other.

In the example above, the regression is logistic, but the effect is still visible.

Further in the same article, the author notes that the models with unit weights are stable in particular because they are not affected by emissions in the training set and cannot be overtrained.