2 years ago we started a blog on Habr, starting with a
review text about what we do, what technologies we use and where we are going. Since 2017, much has changed, and today we will tell how we are making our decision - the global management platform of Connected Cars, which is used by many users and companies of different levels. The material is broken down by process, from problem statement to deployment.

The flagship product Bright Box -
Remoto is a technologically sophisticated, feature-rich platform for Connected Car, which includes equipment, software for dealers and automakers, plus a mobile application for the user. According to the Bright Box primary analytics among car owners, it turned out that first of all they need remote control of door locking, climate control and vehicle search with alerts about shocks or evacuation. The latter is already a classic. Now the Remoto block provides the user with the following services: remote control of vehicle functions, the ability to receive data from hardware and CAN, GPRS, SMS, Bluetooth and control of output power to the electronic ignition unit. The user receives this information in the mobile application.
And such a user can be not only the owner of the car. Accessible information can be a useful tool for many participants in the car market. For example, car sharing. Today, car sharing companies are the most active players in the automotive market. Moscow has become the number one city in the total number of cars involved in car sharing. By 2020, car sharing should reach 40 thousand cars in Russia. Carsharing companies become owners of the following data: mileage, GPS-coordinates, speed, door status and fuel level. The key to all this is a smartphone, which is a cheaper and safer option.
')
Andrei Kuprikov, co-founder and director of business development for YouDrive and one of the Bright Box clients:“Carsharing without a telematics solution is hard to imagine. Our platform is obliged to collect all possible information about the car, what and how it happens. Otherwise it will affect the business. It is telematics that gives information about the cost of repairs and spare parts, the cost of downtime of a car that is being repaired due to the speed fan. If you have a telematics device on board, you can build a unique user loyalty program. ”
Since the beginning of this year, Remoto has become a solution provider for two large car sharing companies, YouDrive and EasyRide, with 1,000 cars in its fleet. Using a carsharing solution is not only convenient, but also effective from a security and financial point of view - in the form of a reduced risk of accidents and in a loyalty program. With the development of car sharing, user data is accumulated, and now car sharing, like banks, has a certain client scoring system. We wrote twice about the logic of scoring algorithms for users of car sharing, first analyzing
the scoring algorithm based on sharp accelerations and decelerations, and then
the driving style analysis algorithms based on the values ​​of speed, engine speed and accelerometer indicators .

But we are actively working not only on the Russian market, but this is an additional challenge. With the expansion of the geography of work, it became clear that the correct and effective restructuring of the engineering vertical is a key moment of development.
They say Vitaly Baum, Chief Product Officer
and Vyacheslav Sokolov, Chief Engineering Officer:Our system consists of a set of components. In engineering, dedicated teams are responsible for them. In fact, Engineering activity includes 3 business processes and a set of supporting services.
Inside the engineering division can distinguish the following business processes:
- Development of telematics devices with embedded software for integration with cars. Engaged in the department of HW engineering.
- The production process of devices for a specific client on request of the business unit. Manufacturing department responds,
- Development of Remoto Cloud Services, responsible for the interaction of the client, user and telematics device. It is a set of backend services with a set of portals for various users, client mobile applications, Data lake. Staging involved in product management department. The development of the entire software part - Product Development Department, releases and support - the RCS operations team.
It is worth noting that the task of defining the functional requirements for a particular feature lies with the Product Management department, which, in addition to productologists, also includes analysts and designers. Next, the requirements go to the Product development department, which is faced with the difficult task of decomposing features into system components, including the device firmware. This task is solved by the Product Development Architect with a team of system analysts.
What does product planning look like? Recently, product management has become part of the engineering team. And this organizational structure is reflected in the way we started working. The PM-team determines which product should be in general, which functions it should have, regardless of the system components. It turns out the brief - a superficial description of what needs to be done in the task. After that, functional specifications are prepared, which we call FSD, or a set of job stories - for example, the ability to send an application to “sign up for THAT” in a product. Every feature we have is described by a set of similar job stories.

PMs also do technical design. They perform technical analysis of the functional specification and create a technical design - TDD (Technical Design Document), discuss this technical design with the developers, and ground it under their understanding. After the functional requirements and technical design are written, we start working on the interface - this is the user experience interface.
Thus, product specialists form a certain set of “units” of utility for a client (“writing on THAT” can be such a unit of utility) and transfer it to a specialist describing the logic in this set. The usefulness in the application for TO means that the customer can fill out a form with the necessary information, which is exactly what dealers expect when submitting an application for maintenance. The product analyst also analyzes the market, examines what should be in the product and what value it gives customers.
Our productologists today communicate more with the business within the company or with customers directly. The roadmap is formed by the roadmap committee, which includes the top managers of the company, in order to take into account all directions of the company's development. The committee meets once a quarter.
This is done in order to coordinate the overall agreement and ensure the integrity of the product, so that the features smoothly fit into the current vision of the product.
There is a separate service - cyber security, which interacts with people working with devices and specialists from the backend department in order to identify vulnerabilities, close them and assess risks, which these risks can lead to. Today this division is assigned by the Chief Engineering Officer, who also heads the product team, and she in turn communicates with the client and understands what is needed now in order to comply with all cyber security standards. All this is included in the release plan, vulnerabilities are closed, certificates are obtained, and a security gap is generally eliminated.
After the functionality has been developed by engineering and evaluated by the security department, its specification goes to the product development team, where the working group decomposes the functions into system components - what is related to the backend, what is the device, what the mobile application should be. Product development team and HW engineering team agree on interaction, it all comes down to a joint plan and diverges by team.
How do we deploy
At the end of the development, the collected result passes integration testing and rolling out to the release on the cloud platform. On a cloud platform where we are hosted (Azure), environments for clients are placed. The environment is the responsibility of the operating team, in which engineers, DevOps and support work.
Comment from Vladimir Glazkov, Senior DevOps Engineer:Our entire infrastructure is described as a code. All changes we make only through the code. This approach reduces the risk of human factors with updates. It also allows you to quickly deploy an additional instance of the environment for some temporary urgent needs. In the event of a computing power failure (VM / VMSS), you can quickly deploy a new instance.
About CI / CD - at the moment we are using TeamCity / Octopus Deploy. TeamCity is in the process of building .net projects, running Unit tests, after which a release is created in Octopus and deployed to the appropriate targets (VM / VMSS / K8S). After successful deployment, acceptance tests are run. If one of the tests fails, the development team will be notified.
Initially, separate sets of resources were created for each business project, including CI / CD tools. It was quickly realized that with an increase in the number of projects, this approach was doomed to fail — to administer such a zoo effectively is simply impossible. Two years ago, a unification project was launched, which ended 4 months later. In its process, core components of the system were identified, for them the process of assembly and deployment is the same for all environments. The possibility of adding additional components specific to a particular business project was also described and implemented. When creating new environments, individual instances of TeamCity & Octopus are no longer required. Scripts were written that create and configure all the necessary things for assembly and deployment through the API.
We have come to the following use of environments: for development, each team uses two environments:
- the first for, in fact, the development of new functionality, verification of features by the author, etc .;
- second to stabilize.
There can be a lot of such sets of environments. It is rather simple to maintain them in the type of unification performed.
There is also an environment for acceptance of release by the team responsible for the combat environment. It is the final test before the deploem in production.
We have an agreement with the developers about the device transformations of configuration files. In each project there is a file that contains a set of parameters that have different values ​​in different environments. The developers fill the file with the necessary parameters (lines of connection to the database, connection keys, etc.), the values ​​of these parameters are variables. For each environment, the values ​​of these variables are individual. With this approach, developers do not interfere with collecting locally and checking with them. Variables are stored in Octopus Deploy.
For monitoring, we use Azure Monitor, Application Insights and Log Analytics. Zabbix is ​​living its time, probably in the future it will be assigned the honorable role of external checks.
When I joined the company, creating a new environment took three weeks. There were almost no instructions, changes were made manually and were not recorded anywhere. Our journey to IaaC began with simple automation, which reduced the process to 1 week. Now creating a new environment takes 4 hours. About 95% of actions are automated.
Our backend is written in .net (4.6 / 4.7 and core), the front is JS. For hosting use
Virtual Machine Scale Sets and K8S. Accordingly, it is very easy to scale under load changes.
How the system is arranged inside
Says Ivan Stolet, Head of Platform Development Bright Box:
You can always find the layout of the current architecture
on the site .
All data in the system is stored distributed. There are separate databases for the storage of personal data with reference to the region and organized in accordance with local legislation. There are databases in which the content part of customer retention services is stored, the storage of news, applications, data from various systems of integration of dealers and automakers. The processed telemetric information is collected separately, as well as the settings and other data necessary to ensure the performance of the Remoto services and our devices. We collect cold telemetry data separately using databases intended for storing huge amounts of information. Aside, separate data warehouses are built to ensure the operation of Remoto AI systems. With the help of so-called crawlers, all the necessary statistical information is collected, on its basis, artificial intelligence selects user groups and builds “predictions”.
Data collection from devices is carried out using Microsoft's IoT solution. Devices are connected to the platform, the platform collects all telemetry and puts it into an intermediate temporary data storage event hub. Our workers have already connected to the event hubs, process telemetry, record cold data and processed data, such as routes and traffic events, execute commands. A separate service can query the device for diagnostic data, analyze the condition of the car and build custom reports.
For user-defined mobile applications, an API is implemented through which the user gains access to the processed telemetry, as well as the ability to execute commands for the device installed in the vehicle. The same API is used to gain access to customer retention service data, the user receives news, special offers from dealers and automakers in his mobile application, has the opportunity to use services, for example, fill out an application for a test drive or loan. Through the mobile application, the user can set the settings for the device, activate telematic services, such as pushing on impact, speeding or leaving the zone, as well as setting the engine to automatically start according to schedule or temperature.
Dealers, in turn, using the provided portals have the opportunity to run diagnostics on the user's device, block the remote engine start, for example, for technical work or service, create a special personal offer, and also process user requests. Communication with the user in such cases is most often carried out using push notifications.
Also, the dealer has the ability to customize the mobile application, the dealer or car maker can paint the application in the colors of its brand, change key icons, determine the set of functions available in the application and some other cosmetic functions, a separate portal has been created for this.
To provide customer support, there is a technical portal in which you can validate the current settings of users and their devices, diagnose the device’s performance, and if necessary, correct the data at the request of the customer, for example, if the user chose a different car model when configuring, the support specialist can fix it. The portal also provides the ability to FOTA (firmware over the air) to update the firmware of a device or group of devices in the event that a new version of the firmware is released with new features or bug fixes.
And a few words about security
Comment from Artem Neroba, CISO:Today, the company's security team is in active dialogue with the business.
We strive to comply with legal requirements: personal data law and GDPR. It is more important than ever to establish a safe development cycle process, i.e. adding to the current development procedures some control points in the form of checking the code before the release of the application, additional third-party code analysis, and increasing the awareness of developers in terms of how to write secure code initially. Global practices and standards are highly recommended to take care of security during development, and not after it. Because after the release, the cost of fixing vulnerabilities is 30% higher. We periodically check product safety by customers, i.e. penetration tests. Given the increased information security, we are now passing these tests quite successfully, and there is not a single vulnerability in the product from release to release with Critical or High risks.
Today we have a team to carry out penetration tests, and we consider it as a team that will help us in the development process to do some kind of security code review for consideration in future releases. These will not be full penetration tests, but simply a review that will be built into our business development process, which is extremely correct from the point of view of a secure code development cycle.
In addition,
we have confirmed the ISO 27001 certificate , the information security management standard according to BSI’s audit.
How do we live and what's next?
Here, in the Bright Box, we are constantly looking for ways to develop the Connected Car Remoto platform.
And our technologies have already helped manufacturers, regional offices, importers and large dealer networks not only increase revenue, but also significantly improve customer retention and, most importantly, reduce operating costs. Over the past two years, such companies as Honda, Motor Car, MINI have become our customers. At the end of 2017, the company itself became part of the insurance group Zurich.

Employees of the company joke that the Bright Box works in an "atmosphere of harassment and envy." Of course it is not. But the reasons for the envy of those who do not work with us in the team, definitely have:
Flexible social package: each employee has the right to choose what to spend on the allocated budget in the amount of his salary, but not exceeding the average salary in the company.Here we provide a wide range of “cafeteria” choices:
- LCA (any medical expenses ranging from one-time visits to doctors to medicines and medical preparations), life insurance, dentistry (including implantology);
- Any sports activities (from climbing to yoga);
- OSAGO / KASKO, parking on the territory of the business park;
- Learning any foreign languages;
- Co-financing of personal travel (visas / tours / tickets / hotels);
- Kindergarten.
It is worth noting that employees of the company can use the social package for their family members, be it medicine, fitness or vacation.
Flexible working hours and remote work days?It seems that many companies offer flexible working hours, but this is far from the case.
We do not limit our engineers and developers to the allocated number of days of remote work per year, but provide every week on Tuesdays and Thursdays the opportunity to work remotely. We are familiar with the concept of work-life balance and we will not prevent the child from being taken from kindergarten or visiting the 1st of September at school, nor will we interfere with visiting exhibitions and other events. Employees can complete urgent tasks from home.
Business trips around the world and work in the company's offices in Dubai, Budapest, Zurich, New York at will.Projects are launched quite often and our engineers and developers go on business trips around the globe. And if the position held does not imply business trips, then you can ask your manager about remote work in one of the company's offices.
Many dream of a cold season to spend in a warm climate - at your service the opportunity to go to Dubai and work from the Dubai office remotely.
Want to live in Europe? Do not question, coordinate remote work from the office of Budapest.
Relocation to BudapestFor those who want to work in Europe or at the call of the service, we find a place in our office in Budapest, help with paperwork for employees and families, moving and finding housing. This is a fairly long process, which takes an average of 3 months. So have patience.
BonusesEmployees of the company are divided into three types, based on their tasks. Each of them has its own bonus system:
The creators are those who create the product. Creators have a decent salary in the IT market, so in their case, bonuses can be for “over” the result. And it is determined by the immediate supervisor, who works closely with each of the employees and can evaluate the work done.
Vikings are those who conquer new territories and make a company profit. There are Vikings who are engaged in the introduction of the product - they participate in the bonus pool, which is determined for each project individually, depending on the complexity. There are Vikings who are engaged in sales - they receive bonuses in the amount of 1-2% of the cost of the realized project.
Farmers are those who serve the Creators and the Vikings. They have bonuses up to a maximum of 1 salary per quarter, and the total amount is calculated on the basis of personal achievements and team goals.
So if you always wanted to be a Viking, but over time and the country did not work out - maybe you should try to become one within our company.
Future parentsOn the occasion of the birth of the child by the employee - the company provides a pleasant bonus in the form of a gift of the pram or in the form of payment of a sum of money.
And if you are ready to withdraw from a decree that was previously laid, then you can safely count on a bonus payment to your salary, which is set by company policy.
Bright Box is in search of talent. If it seemed close to you what you read, and you love machines and artificial intelligence,
come .

Subscribe to regular news, articles and analytics from the world of Connected Cars
here . There is also the official blog
Driving to the future on Medium .