There was a recent need to solve the seemingly classic task mate. statistics.
Testing of a certain push impact on a group of people. You must evaluate the effect. Of course, you can do this with a probabilistic approach.

But it’s completely useless and counterproductive to talk with businesses about null hypotheses and p-value.

How can, as of February 2019, make it as simple and fast as possible with a “medium-sized” laptop at hand? Reference note, there are no formulas.

It is a continuation of previous publications .

Formulation of the problem

There are two statistically identical groups of users (A and B) according to the measured indicator. Group B is affected. Does this effect change the average value of the measured indicator?

The most popular option is to calculate the statistical criteria and draw a conclusion. I like the example "Classical Statistical Methods: Chi-Square Test" . It does not matter how it is done, with the help of specials. programs, Excel, R or something else.

However, one can doubt the reliability of the obtained conclusions for the following reasons:

1. In fact, mat. Few people understand statistics from the beginning to the end. You should always keep in mind the conditions under which you can apply certain methods.
2. As a rule, the use of tools and the interpretation of the results obtained follow the principle of a one-time calculation and a “traffic light” decision. The fewer questions, the better for all participants in the process.

Criticism p-value

Materials mass, links to the most spectacular of the found:

• Nature. Scientific method: Statistical errors. The values ​​of the gold standard, of the validity of the scientists,. Regina Nuzzo. Nature 506, 150–152
• Nature Methods. Lewis G Halsey, Douglas Curran-Everett, Sarah L Vowler & Gordon B Drummond. Nature Methods volume 12, pages 179–185 (2015)
• ELSEVIER. A Dirty Dozen: Twelve P-Value Misconceptions, Steven Goodman. Seminars in Hematology Volume 45, Issue 3, July 2008, Pages 135-140

What can be done?

Now everyone has a computer at hand, so the Monte Carlo method saves the situation. From calculations of p-value, we turn to the calculation of confidence intervals (confidence interval) for the difference of the average.

There are a lot of books and materials, but in a nutshell (resamapling & fitting) is very compactly described in the report Jake Vanderplas - “Statistics for Hackers” - PyCon 2016 . The presentation itself.

One of the initial works on this topic, including proposals for graphical visualization, was written by a well-known Soviet author of mathematics popularization by Martin Gardner: Confidence intervals rather than: Valuation rather than hypothesis testing. MJ Gardner and DG Altman, Br Med J (Clin Res Ed). 1986 Mar 15; 292 (6522): 746-750 .

How to use R for this task?

In order not to do everything at the lower level, let's look at the current state of the ecosystem. Not so long ago, a very convenient dabestr : Data Analysis using Bootstrap-Coupled Estimation package was transferred to R.

The principles of computing and analyzing the results used in dabestr in the format of cheat sheets are described here: ESTIMATION STATISTICS BETA ANALYZE YOUR DATA WITH EFFECT SIZES .

 --- title: "A/B   bootstrap" output: html_notebook: self_contained: TRUE editor_options: chunk_output_type: inline --- 

 library(tidyverse) library(magrittr) library(tictoc) library(glue) library(dabestr) 

 my_rlnorm <- function(n, mean, sd){ #  . : https://en.wikipedia.org/wiki/Log-normal_distribution#Arithmetic_moments location <- log(mean^2 / sqrt(sd^2 + mean^2)) shape <- sqrt(log(1 + (sd^2 / mean^2))) print(paste("location:", location)) print(paste("shape:", shape)) rlnorm(n, location, shape) } # N   (A = Control) A_control <- my_rlnorm(n = 10^3, mean = 500, sd = 150) %T>% {print(glue("mean = {mean(.)}; sd = {sd(.)}"))} # N   (B = Test) B_test <- my_rlnorm(n = 10^3, mean = 525, sd = 150) %T>% {print(glue("mean = {mean(.)}; sd = {sd(.)}"))} 

 df <- tibble(Control = A_control, Test = B_test) %>% gather(key = "group", value = "value") tic("bootstrapping") two_group_unpaired <- df %>% dabest(group, value, # The idx below passes "Control" as the control group, # and "Test" as the test group. The mean difference # will be computed as mean(Test) - mean(Control). idx = c("Control", "Test"), paired = FALSE, reps = 5000 ) toc() 

 two_group_unpaired plot(two_group_unpaired) 

Result in CI

 DABEST (Data Analysis with Bootstrap Estimation) v0.2.0 ======================================================= Unpaired mean difference of Test (n=1000) minus Control (n=1000) 223 [95CI 209; 236] 5000 bootstrap resamples. All confidence intervals are bias-corrected and accelerated. 

and pictures

quite understandable and convenient to talk with the business. All calculations were on "a cup of coffee."

Previous publication - "Data Science" special forces "on its own . "