Antifraud (part 4): an analytical system for recognizing fraudulent payments

In the final fourth part of the article we will discuss in detail the most difficult from a technical point of view part of an antifraud-service - an analytical system for recognizing fraudulent bank card payments .

Detection of various kinds of frauds is a typical case for supervised learning tasks; therefore, the analytical part of the anti-fraud service, in accordance with the best industry practices, will be built using machine learning algorithms.
')
For the task ahead, we will use Azure Machine Learning - a cloud service for performing predictive analytics tasks . Understanding the article will require basic knowledge of machine learning and familiarity with the Azure Machine Learning service .

purpose

In this part, I will describe the project, in the first step of which we will train four models using logistic regression, perceptron, support vector machine and decision tree . From the trained models, choose the one that gives greater accuracy on the test sample and publish it in the form of a REST / JSON service . Further for the received service we will write the software client and we will carry out load testing on REST service.

Creating a model

Let's create a new experiment in Azure ML Studio. In its final form, it will look like the one shown in the illustration below. Let us relate each element of the experiment with the stage of the sequence that the average data scientist does in the process of teaching the model.



Consider each of the stages of creating a model of recognition of fraudulent payments, taking into account the technical details described in the last part 3 of the article.

Hypothesis

The basic concepts and assumptions useful for creating the model were discussed in the first 2 parts of the article. I will not repeat, just note that the creation of a good hypothesis is an iterative process of trial and error, the foundation for which is knowledge both in the studied subject area and in the field of Data Science.

Data acquisition

The data set for the recognition model of fraudulent payments will be a transaction log, which consists of 2 tables in the NoSQL storage (Azure Table): a fact table about transactions TransactionsInfo and a table with previously calculated statistical metrics of TransactionsStatistics.

At the data acquisition stage, we will load these 2 tables through the Reader control.

Data preparation and research

Let's make Inner Join of the loaded tables across the field TransactionId. With the help of the Metadata Editor control, we specify the data types (string, integer, timestamp), mark the column with answers (label) and the columns with predictors (features), as well as the type of scale in these data: nominal, absolute.

Do not underestimate the importance of preparation for the creation of an adequate model: I will give a simple example with the currency of payment, which is stored as ISO codes (integer value). ISO codes - has a nominal (classification) scale. But it is hardly worth hoping that the system will automatically determine that the “Currency” column does not contain an integer value with an absolute scale (that is, operations such as + or> are possible). Because it is too unobvious rule, knowledge of which the system does not possess.

The dataset may contain missing values. In our case, it is not always possible to determine the country or IP address of the payer, such fields may contain null values. After checking the existing data set, we replace the empty values ​​of the countries with “undefined” using the Clean Missing Data control. Using the same control, we will delete the lines where in the card holder field, the payment amount or currency does not contain values, like lines containing obviously incorrect data, that is, making noise in the model.

At the next stage, we’ll get rid of unused fields in the model: the address (we are only interested in whether the payer's country coincided with the country where the request came from), the hash of the cardholder’s name (since it has no effect on the result of the payment) data that came to us from the Azure Table).

Finally, using the Normalize Data control, we will perform a ZScore-normalization of data containing large numerical values, such as the amount of payment (TransactionAmount column).

Data division

Let's divide the resulting data set into a training and test sample. Choose the optimal ratio of data in the training sample and test. For our purposes, using the Split control, we “send” 70% of all available data to the training set, additionally including arbitrary data mixing (the Randomized split flag) when divided into subsets of data. Mixing data when dividing will allow you to avoid "distortions" in the training sample associated with large leaks of plastic card numbers (and, as a result, anomalous activity of fraud robots during this period).

Construction and evaluation of the model

We initialize several classification algorithms and compare which one gives the best result (accuracy) on the test sample. It is important to note that it is not at all the fact that the real data will achieve the same performance as the test data. Therefore, it is very important to understand that the model did not take into account why one of the algorithms gives a significantly worse or better result, correct errors and start the learning algorithm again. This process ends when the researcher receives an acceptable model.

Azure ML allows us to connect an unlimited number of machine learning algorithms in one experiment. This makes it possible at the research stage to compare the performance of several algorithms in order to identify which one is best suited for our task. In our experiment, we use several algorithms of two-class classification: Two-Class Logistic Regression (logistic regression), Two-Class Boosted Decision Tree (decision tree, built using gradient growth), Two-Class Support Vector Machine (method of support vectors), Two- Class Neural Network (neural network).

Another opportunity to get the best model performance is to customize the machine learning algorithm using a large number of parameters available to customize the algorithm. So for the Two-Class Boosted Decision Tree algorithm, the number of trees to be built was specified, as well as the minimum / maximum number of leaves on each tree; For the Two-Class Neural Network algorithm, the number of hidden nodes, learning iterations, and initial weights.

At the final stage, we will review the output of the Evaluate Model control (Visualize command from the context menu of the element) for each of the algorithms.



The Evaluate Model control contains a confusion matrix, calculated Accuracy, Precision, Recall, F1 Score , AUC, ROC and Precision / Recall graphics accuracy indicators. To put it simply, we will choose an algorithm whose Accuracy, Precision, AUC values ​​are closer to 1, the ROC graph is more concave toward the Y axis for both the training and the test sample.

In addition, it is impassable to look at the change in AUC depending on the Threshold value to be set. In the case of fraud, this is important, since the cost of unrecognized fraudulent payments ( False Positive ) is much higher than the cost of payments mistakenly taken as fraud ( False Negative ).

In such cases, it is necessary to select a Threshold value other than the default value of 0.5.

When choosing the most suitable algorithm for obtaining the optimal model of fraud recognition, besides the Threshold level, we also take into account the fact that the decision-making logic for some algorithms (for example, a decision tree) can be reproduced, and for some there is no (perceptron). The availability of such a possibility can be critical if it is important to know why, according to a certain precedent, the system made a specific decision.

The best accuracy was shown by the Two-Class Neural Network algorithm of the two-class neural network (the accuracy figures are shown in the illustration above), followed by the algorithm based on decision trees - the Two-Class Boosted Decision Tree.

Publishing the model as a web service

Once a model has been obtained that works with the required accuracy, we will publish our experiment as a web service. The publishing operation takes place by clicking the Publish Web Service button in Azure ML Studio. The process of creating a web service from an experiment is trivial and I will skip its description.

As a result, Azure ML will deploy a scalable fault-tolerant (SLA 99.95%) web service. After the service is published, the service API documentation page will be available - the API help, which in addition to the general service description, description of the formats of expected input and output messages, also contains examples of calling the service in C #, Python and R.

The principle of calling a service by a software client can be represented as follows.

Connect to the Azure ML web service

Take an example in C # from the API help and, having slightly changed it, call the web service Azure ML.

Listing 1. Call the Azure ML web service

private async Task<RequestStatistics> InvokePredictorService(TransactionInfo transactionInfo, TransactionStatistics transactionStatistics) { Contract.Requires<ArgumentNullException>(transactionInfo != null); Contract.Requires<ArgumentNullException>(transactionStatistics != null); var statistics = new RequestStatistics(); var watch = new Stopwatch(); using (var client = new HttpClient()) { var scoreRequest = new { Inputs = new Dictionary<string, StringTable>() { { "transactionInfo", new StringTable() { ColumnNames = new [] { #region Column name list }, Values = new [,] { { #region Column value list } } } }, }, GlobalParameters = new Dictionary<string, string>() }; client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", ConfigurationManager.AppSettings["FraudPredictorML:ServiceApiKey"]); client.BaseAddress = new Uri("https://ussouthcentral.services.azureml.net/workspaces/<workspace_id>/services/<service_id>/execute?api-version=2.0&details=true"); watch.Start(); HttpResponseMessage response = await client.PostAsJsonAsync("", scoreRequest); if (response.IsSuccessStatusCode) await response.Content.ReadAsStringAsync(); statistics.TimeToResponse = watch.Elapsed; statistics.ResponseStatusCode = response.StatusCode; watch.Stop(); } return statistics; } 

We get the following request / response:
Listing 2.1. Request to Azure ML web service

 POST https://ussouthcentral.services.azureml.net/workspaces/<workspace_id>/services/<service_id>/execute?api-version=2.0&details=true HTTP/1.1 Authorization: Bearer <api key> Content-Type: application/json; charset=utf-8 Host: ussouthcentral.services.azureml.net /*   */ { "Inputs": { "transactionInfo": { "ColumnNames": [ "PartitionKey", "RowKey", "Timestamp", "CardId", "CrmAccountId", "MCC", "MerchantId", "TransactionAmount", "TransactionCreatedTime", "TransactionCurrency", "TransactionId", "TransactionResult", "CardExpirationDate", "CardholderName", "CrmAccountFullName", "TransactionRequestHost", "PartitionKey (2)", "RowKey (2)", "Timestamp (2)", "CardsCountFromThisCrmAccount1D", "CardsCountFromThisCrmAccount1H", "CardsCountFromThisCrmAccount1M", "CardsCountFromThisCrmAccount1S", "CardsCountFromThisHost1D", "CrmAccountsCountFromThisCard1D", "FailedPaymentsCountByThisCard1D", "SecondsPassedFromPreviousPaymentByThisCard1D", "PaymentsCountByThisCard1D", "HostsCountFromThisCard1D", "HasHumanEmail", "HasHumanPhone", "IsCardholderNameIsTheSameAsCrmAccountName", "IsRequestCountryIsTheSameAsCrmAccountCountry", "TransactionDayOfWeek", "TransactionLocalTimeOfDay" /*    */ ], "Values": [ [ "990", "f31f64f367644b1cb173a48a34817fbc", "2015-03-15T20:54:28.6508575Z", "349567471", "10145", "32", "990", "136.69", "2015-03-15T20:54:28.6508575Z", "840", "f31f64f367644b1cb173a48a34817fbc", null, "2015-04-15T23:44:28.6508575+03:00", "640ab2bae07bedc4c163f679a746f7ab7fb5d1fa", "640ab2bae07bedc4c163f679a746f7ab7fb5d1fa", "20.30.30.40", "990", "f31f64f367644b1cb173a48a34817fbc", "2015-03-15T20:54:28.6508575Z", "2", "1", "0", "0", "0", "0", "1", "2", "0", "0", "true", null, "true", "true", "Monday", "Morning" /*    */ ] ] } }, "GlobalParameters": { } } 

Listing 2.2. Azure ML web service response

 HTTP/1.1 200 OK Content-Length: 1619 Content-Type: application/json; charset=utf-8 Server: Microsoft-HTTPAPI/2.0 x-ms-request-id: f8cb48b8-6bb5-4813-a8e9-5baffaf49e15 Date: Sun, 15 Mar 2015 20:44:31 GMT { "Results": { "transactionPrediction": { "type": "table", "value": { "ColumnNames": [ "PartitionKey", "RowKey", "Timestamp", "CardId", "CrmAccountId", "MCC", "MerchantId", "TransactionAmount", "TransactionCreatedTime", "TransactionCurrency", "TransactionId", /*    */ "Scored Labels", "Scored Probabilities" ], "Values": [ [ "990", "f31f64f367644b1cb173a48a34817fbc", "2015-03-15T20:54:28.6508575Z", "349567471", "10145", "32", "990", "136.69", "2015-03-15T20:54:28.6508575Z", "840", "f31f64f367644b1cb173a48a34817fbc", /*    */ "Success", "0.779961256980896" ] ] } } } } 

Stress Testing

For the purpose of load testing, we will use Azure's IaaS capabilities — let's raise a virtual machine (Instance A8: 8x CPU, 56Gb RAM, 40Gbit / s InfiniBand, Windows Server 2012 R2, $ 2.45 / hr) in the same region (US Central South) our Azure ML web service. Let's run a task on ~ 20K queries on a VM and look at the results.

Listing 3. Service client code and tasks

 /// <summary> /// Entry point /// </summary> public void Main() { var client = new FraudPredictorMLClient(); RequestsStatistics invokeParallelStatistics = client.InvokeParallel(1024, 22); LogResult(invokeParallelStatistics); RequestsStatistics invokeAsyncStatistics = client.InvokeAsync(1024).Result; LogResult(invokeAsyncStatistics); } private static void LogResult(RequestsStatistics statistics) { Contract.Requires<ArgumentNullException>(statistics != null); Func<double, string> format = d => d.ToString("F3"); Log.Info("Results:"); Log.Info("Min: {0} ms", format(statistics.Min)); Log.Info("Average: {0} ms", format(statistics.Average)); Log.Info("Max: {0} ms", format(statistics.Max)); Log.Info("Count of failed requests: {0}", statistics.FailedRequestsCount); } /// <summary> /// Client for FraudPredictorML web-service /// </summary> public class FraudPredictorMLClient { /// <summary> /// Async invocation of method /// </summary> /// <param name="merchantId">Merchant id</param> /// <exception cref="ArgumentOutOfRangeException"><paramref name="merchantId"/></exception> public async Task<RequestsStatistics> InvokeAsync(int merchantId) { Contract.Requires<ArgumentOutOfRangeException>(merchantId > 0); IEnumerable<TransactionInfo> tis = null; IEnumerable<TransactionStatistics> tss = null; // upload input data Parallel.Invoke( () => tis = new TransactionsInfoRepository().Get(merchantId), () => tss = new TransactionsStatisticsRepository().Get(merchantId) ); var inputs = tis .Join(tss, ti => ti.TransactionId, ts => ts.TransactionId, (ti, ts) => new { TransactionInfo = ti, TransactionStatistics = ts }) .ToList(); // send requests var statistics = new List<RequestStatistics>(inputs.Count); foreach (var input in inputs) { RequestStatistics stats = await InvokePredictorService(input.TransactionInfo, input.TransactionStatistics).ConfigureAwait(false); statistics.Add(stats); } // return result return new RequestsStatistics(statistics); } /// <summary> /// Parallel invocation of method (for load testing purposes) /// </summary> /// <param name="merchantId">Merchant id</param> /// <param name="degreeOfParallelism">Count of parallel requests</param> /// <exception cref="ArgumentOutOfRangeException"><paramref name="merchantId"/></exception> /// <exception cref="ArgumentOutOfRangeException"><paramref name="merchantId"/></exception> public RequestsStatistics InvokeParallel(int merchantId, int degreeOfParallelism) { Contract.Requires<ArgumentOutOfRangeException>(merchantId > 0); Contract.Requires<ArgumentOutOfRangeException>(degreeOfParallelism > 0); IEnumerable<TransactionInfo> tis = null; IEnumerable<TransactionStatistics> tss = null; // upload input data Parallel.Invoke( () => tis = new TransactionsInfoRepository().Get(merchantId), () => tss = new TransactionsStatisticsRepository().Get(merchantId) ); var inputs = tis .Join(tss, ti => ti.TransactionId, ts => ts.TransactionId, (ti, ts) => new { TransactionInfo = ti, TransactionStatistics = ts }) .ToList(); // send requests var statistics = new List<RequestStatistics>(inputs.Count); for (int i = 0; i < inputs.Count; i = i + degreeOfParallelism) { var tasks = new List<Task<RequestStatistics>>(); for (int j = i; j < i + degreeOfParallelism; j++) { if (inputs.Count <= j) break; var input = inputs[j]; tasks.Add(InvokePredictorService(input.TransactionInfo, input.TransactionStatistics)); } Task.WaitAll(tasks.ToArray()); statistics.AddRange(tasks.Select(t => t.Result)); } // return result return new RequestsStatistics(statistics); } /* other members */ } 

Call InvokeParallel ():
Best Response Time: 421.683 ms
Worst time: 1355.516 ms
Average time: 652.935 ms
Number of successful requests: 20061
Number of failures: 956

Call InvokeAsync ():
Best Response Time: 478.102 ms
Worst Time: 1344.348 ms
Average time: 605.911 ms
Number of successful requests: 21017
Number of failures: 0

Limitations (potential)

The bottleneck of the system being developed, at first glance, will be Azure ML. Therefore, it is extremely important to understand the limitations of Azure ML in general and Azure ML web services in particular. But on this issue there is very little of both official documentation and the results received from the community.

So the question remains with the throttled policy of the Azure ML web service endpoints: the maximum value of parallel requests for the Azure ML web service is not clear (the figure of 20 parallel requests per endpoint is empirically tested), as well as the maximum size of the received message (actual for packet mode service work).

Less relevant, but there is a question with the maximum size of input data (Criteo Labs placed dataset on 1 TB of data), the maximum number of predictors and use cases that can be input to the machine learning algorithm in Azure ML.

It is critical to reduce the response time of the web service FraudPredictorML, as well as the time to retrain the model to a minimum, but so far there are no official recommendations on how this can be done (and is it even possible at all).

Recommendations to customers

Antifraud-service does not limit customers in any way in the preliminary verification of payments, and in the subsequent interpretation of the prediction results. Preliminary checks specific to the business process, as well as the final decision on accepting / declining a payment, are tasks that clearly fall outside the area of ​​responsibility of the anti-fraud service.

Regardless of the role of the client - an online store, a payment system or a bank - the following recommendations exist for customers:

• perform a preliminary check of payments, using technologies accepted in the industry (fingerprint, etc.), and using your own knowledge of the client (order history, etc.);
  
• interpret the result using the following practice: fraud probability is less than 0.35 - accept payment without 3D-Secure, probability from 0.35 to 0.85 - accept payment with 3DS enabled, fraud probability - more refuse;
  
• choose the levels proposed in the previous paragraph based on your own analytics and regularly review them (minimize lost profits and fraud penalties).
  

Conclusion

In this cycle, consisting of 4 articles, we conducted an experiment on the design and development of a highly scalable, fault-tolerant, reliable antifraud-service operating in near real-time mode, with open for external REST / JSON API clients.

The use of machine learning algorithms (decision tree, neural networks) allowed us to create an analytical system capable of self-learning both on accumulated history and on new payments. Through the use of PaaS- / IaaS-services, it was possible to reduce the initial financial costs of infrastructure and software to almost zero. The fact that the developer has competencies in the subject area, data science, and distributed systems architecture has helped dramatically reduce the number of participants in the development team.

As a result, in less than 60 man-hours and with minimal initial infrastructure costs (<$ 150, which were covered from an MSDN subscription), it was possible to create an antifraud system core.

The resulting service, of course, requires more careful verification (and subsequent correction) of the main modules, more fine-tuning of the classifier (s), the development of a series of auxiliary subsystems, interest, and (no secret) investments. But despite the above-mentioned defects, the service is an order of magnitude (and more) more efficient than similar developments in the industry, both in terms of development costs and in terms of the cost of ownership.

Other parts of the article

If it remains unclear to you what the problem is (part 1).
If you didn't miss, why fraud problem long and expensive difficult to solve (part 2).
If you are interested in how it looks from the point of view of software architecture (part 3).


Dmitry Petukhov
Software Architect & Developer, Big Data Enthusiast, Microsoft Certified Professional
architect, developer, enthusiast, tireless researcher and coffee lover