Beautiful parsing email alerts from the bank

We continue to delight you with interesting technical solutions.

Today on the queue email alerts from the bank, which look like this:

  Pokupka, SHELL AZS OLGINO 1133, karta * 347788, 07.23.11 12:09, 300.25 rub.  Dostupno = 421.61 rub 

or so

  1000.00 RUR was charged from your account ** 77876.
 Point of sale: ZAO GAMMAEKSPER
 Date: 07/12/2011
 Available balance: 12344.11 RUR 

What do you think the beautiful handling of such alerts in the personal finance service should look like?

')

The king is dead

I'll start with the story. Email processing appeared in Zen-mani more than a year ago. But then she had a bunch of flaws:

• only one bank was supported;
• each Zen-Mani account had its own email-address and, if you had several accounts in one bank, the payments to the accounts were not shared;
• if the bank sent any notifications, they did not reach you, because the service intercepted all the letters.

The inconvenience of working with email alerts and the success of our overseas counterpart Mint (for which you do n’t need to enter expenses and incomes at all (he gets all the data from a bank card and kindly sorts them into food, gasoline, entertainment ...) prompted us to rewrite the processing email alerts.

Let's start small, increase the support of banks to four: VTB24, Citibank, Bank Saint Petersburg and Bank24.ru. This is done by smart regular expressions and hard testing :)

Then the first interesting solution, leave one email address (dropbox). In general, one for all accounts and all banks (for one user). The difference in accounts occurs in the last four digits. The user enters them at the stage of creating an account:



The last inconvenience with the loss of information letters, we decided by sending all messages to the user's personal email. And it was a very difficult decision for us. Before this function, we did not ask for e-mail in order to preserve the privacy of such sensitive information as personal finances as much as possible. Now you have a choice to specify the mail or not.

These three improvements are not enough for a beautiful solution. The user still has to determine what each transaction was spent on: gasoline, training, or something else. I suggest you unravel what we came up with :) The answer is below under the heading “Zest”, but for now some code.

Parser mechanism

The message parser works in two stages:

• definition of letter format;
• directly parsing.

We will parse this letter here:

  Dear Customer,

 1000.00 RUR was charged from your account ** 77876.
 Point of sale: ZAO GAMMAEKSPERT
 Date: 07/12/2011
 Available balance: 12344.11 RUR
 Please note that the available balance on the current ruble account includes a credit limit.

 If you do not agree with the information received, please contact a CitiPhone employee at (495) 775 75 75 (in Moscow), (812) 336 75 75 (in St. Petersburg), 8 (800) 700 38 38 (in other . cities of the Russian Federation) for eight days.

 For more information about our services, visit our website on www.citibank.ru. 

To determine from which bank the letter came, and what to expect inside we use the following sign:

  id foreign_format check_order marker_location marker_pattern datatype
 43 33 0 10 /www\.citibank\.ru/mail
 44 33 1 2 / ^ \ d / mail 

For each letter, we check each of the formats in turn in the check_order order and look for marker_pattern in the lines. As it is easy to check the letter from Citibank above gives us 33 format.

From the second table select field parsers for 33 formats:

  id column_name value_offset foreign_format
 49 outcome 2 / ^ ([\ d \.] +) / "33
 50 datedmy 4 # (\ d {2} / \ d {2} / \ d {4}) $ # 33
 51 payee 3 /[^::+:(.*)$/ "33 

The number before the regexp means the line number of the letter. We go through the letter again and get the data we need.

Zest: Autodetect categories

So we got to the most interesting - our implementation of automatic comparison of categories of spending on the map.

First, another example:

  We inform you that on 07/21/2011 at 11:12:05 your bank card VTB24 ... 7780 made a payment transaction in the amount of 1959.00 RUR.
 Available for use: 2274.35 RUR.  Payment details: HITZONA.  Authorization code 222635 

And now look, in each payment we have a store code:

• SHELL AZS OLGINO - Gasoline.
• HITZONA - entertainment (music, movies, games).

That is, it is enough for us to compare store codes to our categories. But where to get this data? Crowdsourcing comes to the rescue.

Each Zen-mani user contributes to the common cause by comparing a payment on a card of one of the categories. And we get something like this (users themselves enter the category names).

  10 SHELL AZS Petrol
 3 SHELL AZS Cars
 1 SHELL AZS Sweets 

When a user receives a notification about a payment in SHELL AZS, we look for the intersection of the base with its categories and select the most popular one. There is a category “Gasoline” - choose it, if not choose “Cars”.

In other words, we in Zen-mani are doing everything to facilitate your home accounting. Today, it is receiving data from six banks: Alfa-Bank, Raiffeisenbank, VTB24, Citibank, Bank Saint Petersburg, Bank24.ru and Yandex.Money systems.

Try it today :)