Photo from Google, this is not the author's mom

My mother started working in one of the largest EU banks before I was born, and I was always partial to her specialty, especially in recent years, since I myself became a programmer. I was asked many times to interview her, and I finally decided to do it.

I will explain a little

The world of banking software is another universe. It is very different from the one that most of us are used to. I publish this post on HN and on Reddit . I publish an interview not in the form of questions and answers, but in the form of a story. I added some questions and answers to the bottom of the post.

1991

The year when she began internal training at Nordea Bank, then known as Nordbanken (Northern Bank). In 2001, it was renamed Nordea. During her studies, she had to undergo various tests, primarily an IQ test, to show that she had enough intelligence to work in this area. The psychological stability test is that she has enough nerves for this specific job and a multitasking test, which she filled up with a 22/100 rating. The rest of the tests she passed successfully and took one of the 16 available positions.

The position sounded “like an IBM mainframe programmer in the COBOL language”, and until now, for 25 years, my mother works in this position in the same bank.

This position in the bank is the most important, at least from a technical point of view. If, say, a mother and her team members quit a job at the same time, the bank will break down, at best, within a few weeks. Her team works in shifts with round-the-clock availability. I remember when I was younger, she took a taxi and drove to work in the middle of the night, on Sunday, to fix the interlocking.

My brother (he is 1 year old, he is on the left), my mom and me (on the right)

COBOL ...

... it is not such a cool programming language as a functional Haskell or Golang for parallel computing. COBOL is an imperative, procedural language, and since 2002 an object-oriented language. There is nothing bad in the language itself, the problem is that almost nobody knows it, at least in the context of mainframe programming. My mother is the second of the youngest team members, and she was born in 1964. The youngest of the staff has a difference with her in two years. This is a global problem, since almost all the largest banks in the world work with IBM and COBOL mainframes - their main programming language. Banks have a smaller situation better, they usually work, for example, in Java, without mainframes.

Mom used to ask me if I wanted to learn this language, but then I already worked with more advanced technologies like Postgres, Redis, Node, Crystal , PHP and always answered “Yes, for nothing!”. I am still interested in what it does, but these types of systems seem to cause the worst corporate feeling that I can imagine and which I would like to avoid.

True, I can only imagine the fat salary of the 20-year-old programmer of the old mainframe, because age is invaluable in this case.

Database

Their main database is called IMS . This is a hierarchical database created by IBM for the Apollo program. Between themselves, they call it DL / 1, Database Language One (Database Language One). They are trying to migrate to DB2 , a relational database that understands normal SQL. But, given the huge amount of data stored by Nordea Bank, this task will take several years. You need not just to move data from IMS to DB2, in addition you need to update modules for loading and saving data from DB2 instead of IMS, but thousands of modules, many of which were developed by programmers who either died or retired.

Each transaction is stored in DB2. They avoid recordings in the IMS as much as possible, and only read data from it until their new purchased system is fully integrated and they cannot store data there.

IMS is extremely old and very slow (for some tasks).
Data retrieval may take several hours. Ha, and here we argue that MySQL has a two millisecond faster query performance than Postgres. A bit ironic.

They also use a flat file structure for a number of tasks, which themselves are divided into several subspecies supported by IMS. One example is GSAM , because of which my mother often had to rush to work by taxi. And all because of the modules working with the same GSAM file at the same time, creating interlocks.

Let's talk about the size of their databases. DB2 only stores transaction data, and transactions vary in size, depending on the type of accounts between which the transaction flows. Private accounts, like my personal bank account, are much simpler than corporate accounts. Each transaction weighs anywhere from 500B to 2Kb, on average about 1Kb.

Now their DB2 database stores 11 billion transactions, and the law requires each transaction to be stored for 10 years, and in fact, 11 years. At the moment, transactions are only 7 years old, and their number is expected to grow by 5-8% every year, and so on until they can wait for 11 years, when it will be possible to destroy records older than 11 years.

It turns out, today, DB2 stores about 10TB of data, and this is only a transaction. After four years it will turn into approximately 13-14TB.

Most of the remaining information is stored in IMS. This data is very different in size, so it is difficult to estimate the size, but I assume that it is much more than in DB2.

They also store some data on tapes, and their programs tell the machine to load or save data from a particular tape. This is pretty cool!

IDE

Everyone has something like an IDE or a text editor, right? They, here, have. Their IDE is called ISPF and resembles the operating system. ISPF can be extended, and the part that can be called an IDE is an ISPF extension called Endevor.

ISPF is directly connected to the mainframe, and they have no such thing as a local development environment.

ISPF interface found on the Internet

Batch processing

Approximately 80% of their systems are batch processing. These are tasks that are started at a specific time or interval of time, performing some kind of data processing or sending data to other banks, agencies or somewhere else. For example, when I buy a Coca-Cola bank, the money is withdrawn from my account, but in fact the money does not go anywhere until one of their batch tasks completes this operation. These tasks are usually performed at night, so it usually takes a day to complete the operations between banks. Transactions within the bank are usually instantaneous, since they are executed immediately.

Batch jobs work with gigabytes, sometimes terabytes of data, and in some cases can take several hours. I would like to see how IBM mainframes suddenly go to full capacity in their data centers at the second when it is 00:00. That would be awesome!

Problems faced by banks

Banks that operate on mainframes have many problems to solve, but, unfortunately, they do not have enough time for this.

Programmers are getting older, not many people want to learn, and the time after which a new employee can confidently perform his functions is 2-3 years, and even after that he can get into an atypical situation.

There are programs for decades: no one understands how they work, and the person who wrote them is long gone.

Many banks want to buy new cool systems to replace the old ones, and all their programmers say with one voice: “This will not work, our monster cannot be replaced with anything”.

In part, the reason for the enormous size of their systems is that they initially made a mistake and began to adjust their system to other systems. A good example: we create RESTful services, in which programs that need to get data from us, adjust their systems to ours. In some banks, this is not the case, and they send data to other systems, for example, the tax authorities, and the tax authorities may at any time update their systems, which, in turn, will break the ones that banks use to send data.

Banking systems are also extremely advanced. A personal bank account is very different from a corporate bank account. In addition, there are at least 50 different types of bank accounts for each type. And in the case of Nordea, there are also Swedish government accounts, which differ from personal and corporate accounts. I think that there are Finnish government accounts and, perhaps, part of the Danish, which are also somewhat different.

Finally

It will be very interesting to hear what Nordea and other banks are going to do in the coming years, and what new technologies they are going to implement.

Q & A

Why did you decide to program on the IBM mainframe on COBOL?
- I always wanted to “work with computers”, but I did not look for this particular job. I saw an ad in the newspaper, filed a resume and got a job. True, at that time I already had some experience in the field of computer systems.

What shocked you the most?
“My colleague once forgot to add a dot at the end of the instructions for a software module in the most important part of our system, which we call the cash register.” She is responsible for handling all the money. As a result, the work of the entire bank stopped at 16 o'clock, solely because of the module that continued to be executed, although it had to stop after that instruction. It literally hung our system, arranging, we can say, DoS-attack itself.

What do you think, what future awaits banks operating in the same infrastructure as Nordea?
“Most banks have recognized the need to replace older mainframes with more modern ones. Nordea bought a new system, which promised to replace the current one within 4 years, but the bank serves several countries, and in reality, 4 years will go to just one country, so a total of 16 years.

Banks and financial systems that do not understand this will be in a very difficult situation. Although it still seems to me that we will use the IBM infrastructure.

What problems did you encounter as a female programmer who started working in the 90s?
- Absolutely no problems. In my team there are several women, but more men. It doesn't bother me much.

You have been working on one module and, possibly, on one code base for more than 20 years. Does it bother you at all?
- Yes, definitely, as in other areas of activity. But while working here, I created a lot of absolutely new systems in different financial areas, and it was always interesting. Unfortunately, from now on, we will never create anything new, just keep the current one and wait for the new system that we bought for replacement.

How scary to write code for the bank?
- It is very scary, especially when we make changes on the main server on Sundays. When we change something there, large blocks of the system need to be transferred to offline. One of these blocks is the entire IMS. In this period of time, anything can happen, but we have a very reliable test system, so, as a rule, everything goes without problems.

Have you ever made serious mistakes in your work?
- Definitely, I made one rather serious mistake back in 1997, when my youngest son (this is me, the author) just went to kindergarten and my maternity leave ended. We have a funded pension system. This type of bank accounts at that time was not closed, but by law it was impossible to withdraw money from this account until it turned 55 years old. Since the accounts were not closed, it was possible to withdraw money if you had a bank account number. Therefore, the solution was simple - do not give customers their bank account numbers.

I managed to screw it up and changed the module that began to add the account number to the letter sent to the client by mail. Customers began to withdraw money (from which they have not yet written off taxes!) From their retirement accounts earlier than they were supposed to by law.

This led to a huge internal investigation. The Swedish government intervened, we were attacked by the financial inspectorate and the media. It's all me.

What is your working atmosphere?
- We recently moved to a more “fashionable” place. We used to have personal desks, and now “choose any available place” - an open space. I don't like it at all.

(Translation by Natalia Bass )