SOC is people. How to build a team in terms of personnel hunger
They say that in the 70s, Gennady Zaitsev, the first president of the legendary Leningrad Rock Club, formulated the principle of selection of musicians in the following way: “Finger fluency is a new thing. Had a good man . ”
Perhaps, if Gennady Borisovich and I had decided to complain to each other about the difficulty in recruiting over a glass of something to the fire, we would have found many common points. When we assembled the first Solar JSOC team, the market for monitoring services and countering cyber attacks actually did not exist, therefore, there were no ready-made specialists for these tasks, or even clear search criteria. We had to put together a team of unique ones, but before that - to try new things many times, make mistakes and try again. It is about the team today and let's talk: how it was at the very beginning and what we came to as a result.
I do not know for certain how the idea of ​​creating a JSOC was born - maybe just an accidental combination of circumstances, or maybe the guys got together and firmly decided: “We are creating SOC for the commercial market of Russia!” as well as incredible adventures in the wild world of customer infrastructures.
')
We will start from the moment when JSOC from synthetic plans and estimates is ripe for entering the market and running the solutions developed in battle. And for this, technology, ideas, ambitions and gurus of SIEM and analytics are not enough. There is no way without the first line of analysts - the team that provides the front for the main SOC processes.
Our first line of monitoring is different from the first line in the classic sense. This is not a call center and not girls with a script routing application. The role of the classic first line in our paradigm is assigned to the SIEM with the pie of the profiling, aggregating and enriching rules. And the task of the first line of JSOC is to conduct full investigations of IS incidents, including FP filtering, forming analytical information and recommendations on counteraction. A free review is also welcome, especially in situations where unusual events are seen in the vicinity of the incident.
“Any adventure has to start somewhere ...
When the first version of the team was formed, we did not quite represent the exact profile of the analysts' activities, since the development of events in a completely new market is difficult to predict. Therefore, to compensate for possible turns of fate and the vicissitudes of technological trends, we decided to look for guys with a broad technical background.
To assemble a team of such specialists turned out quickly enough, and this greatly helped us at the initial stage. We felt ready for a winding and difficult way of bringing SOC to the market.
However, after some time, we realized that we had miscalculated something: the majority of the hired specialists were clearly overqualified for the position of the first-line engineer, and the v.1.0 team was rather unstable.
Despite the high dynamics of the development of Solar JSOC and the constant emergence of new tasks, the ambitions of the guys were even higher, and the final list of responsibilities was significantly lower than their experience and expectations. Therefore, colleagues quickly grew from the first line - someone within the JSOC, and someone, alas, outside of it.
However, the “virus” of IB, obtained in JSOC, turned out to be very persistent - despite the fact that the v.1.0 team was almost 100% from pure IT, almost all of them continue to work on the information security field.
Nevertheless, this first attempt to assemble a team was very useful for us.
We “on the move” decided what we ultimately do and what we expect from our employees. We are used to the fact that in the new market multiple growth and the attendant permanent training of personnel are the norm and routine. Found that the market can not offer ready-made specialists for work on the first line of SOC. It became clear that in these conditions it is simply utopia to simply pass with a frequent rake on the summary.
“Learning a scientist is only spoiling.”
So, we find ourselves in a situation in which there are practically no ready-made specialists on the market, and the number of customers begins to grow, which requires scaling the team. At a certain point, we simply came into conflict with the capabilities of the regional personnel market (the first line of Solar JSOC is located in Nizhny Novgorod).
Salvation for us was working with students. At this point, we had little experience of such work in the Moscow team of Solar Security, but it was not aimed at working with the university as a platform, organizing the flow of young professionals. We felt that in the case of Solar JSOC, a more systematic approach would be required.
And here our colleagues, graduates of the Nizhny Novgorod University, who helped us to the UNN student career center, a vigorous student organization interested in the professional employment of students, helped us a lot. We began to regularly participate in the Days of the Employer, telling students and recent graduates of technical universities about the activities of Solar JSOC.
Then we launched an internship program. Initially, it was aimed at students of the specialty of information security, but at present it is extended to students of other IT specialties.
Looking ahead, I will say that this work has given excellent results. For the third year now, young people come to us by "gravity", ready to get tested, train, study and become first-line engineers. And many graduates of our internships really found their first job in Solar JSOC.
- This is too difficult task for my little brain ...
We understood what the ideal team of an ideal SOC should ultimately end up with. And we are ready to reveal this secret: the team should be first and foremost a team, no matter how trite it may sound. And the phrase of Gennady Zaitsev was given at the beginning of the article not by chance. We approach the selection of personnel, guided by the same principle: “Fingers fluency is a new thing. Had a good man. ” After repeated trial and error, we came to the conclusion that when searching for engineers of the first line, the main focus should be not on the technical background and previous experience. (This, however, does not mean that previous experience is completely unimportant. Of course, good positioning and linux knowledge is welcome for a job seeker in a GIS maintenance team.) We are trying to find guys with “live brains”. Guys who can see a fan of solutions that can reason with good analytical skills.
When there was an influx of potential candidates for internships, and subsequently work, we were faced with the task of evaluating the learning ability of each candidate. Therefore, during the interview we try to bring the applicants to logical reasoning, give logical tasks and change their conditions in order to see how a person adapts and selects the best solution. This approach helps us to select people with whom it is pleasant to work and who are very quickly integrated into the team. Well, the "fluency of the fingers" - it can be acquired. About how we train it - just below.
As an entrance test for an internship, we suggest the guys to deal with this kind of tasks:
The internship program includes:
During internships, the most attention is paid to test incidents, most of which are created based on real events from the Solar JSOC practice. In addition to the template, automatically generated incidents, interns regularly receive incidents that are multi-layered with tackles for analysis. Incidents that require a special approach, because as a result of a template investigation, students, for example, come to the conclusion that malicious activity was initiated from their own hosts from under their own uchetku.
Such people significantly increase their involvement in the process of learning and investigating incidents, teach them to pay attention to various “trifles”, recheck their conclusions, speculate and prove or disprove them in the future.
Involvement in the investigation process is the most important at this stage. If we are interested in safety and letting them “feel” incidents with real history or incidents of which they are allegedly involved, we motivate them to study the “boring” theory and develop non-standard and non-described approaches to solving the problem.
At the exit from the internship, we get guys half ready to work on the first line. Thus, we form a personnel reserve.
Regardless of whether we take a person to work following an interview or after an internship, at the beginning of the professional path of the Solar JSOC first-line engineer, a training program awaits you, which takes 2 to 4 months to complete. It can be divided into the following blocks:
According to the results of the evaluation of the curator and the final control slice, conducted by the commission, the engineer is allowed to “independent swimming”, night / weekend duty.
At first, most engineers work in the paradigm “For an incident of this type, I use such and such tools, they give the necessary data about it.” This methodology allows you to train a person in a short time and bring him to the level when he is able to bring benefits on the line.
As they gain experience, the children realize the commonness of IS incidents, begin to better understand what information and in what section will be the most useful for the customer to respond to a specific incident. Gradually, they move to work in the paradigm "I need such and such information, it can be obtained in many ways, in this situation, such and such is preferable."
A similar restructuring of the approach, coupled with the pumping of the IB background and immersion in content, SIEM prepares children for steps to the next steps, where they are waiting for deep investigations of highly critical incidents and chains of IS incidents or immersion into SIEM systems with all the variety of custom logic.
This ends the formalities and begins a full-fledged life on the first line: for administration - full of complex quests to satisfy the desires of customers and compliance with information security requirements; for monitoring - consisting of many fascinating stories reflected in the logs. Sometimes a day on the first line may resemble a puzzle on several boards of a duty shift: the collected pictures flash before the engineer, changing not only their texture, but also the ticket number :)
But, although the theater begins with a hanger, and the operating life of the SOC - from the first line, it does not end with it. People grow, projects scale, and we have a great many roles and tasks within us. We are very encouraging in the first-line engineers to strive for development, therefore we have worked several “professional elevators”. A certain percentage of engineers who are ready to move to other lines are even included in KPI team lead lines :) But the story of "sublimating expertise" or further growth of personnel in Solar JSOC will become part of other articles. To be continued ...
Perhaps, if Gennady Borisovich and I had decided to complain to each other about the difficulty in recruiting over a glass of something to the fire, we would have found many common points. When we assembled the first Solar JSOC team, the market for monitoring services and countering cyber attacks actually did not exist, therefore, there were no ready-made specialists for these tasks, or even clear search criteria. We had to put together a team of unique ones, but before that - to try new things many times, make mistakes and try again. It is about the team today and let's talk: how it was at the very beginning and what we came to as a result.
I do not know for certain how the idea of ​​creating a JSOC was born - maybe just an accidental combination of circumstances, or maybe the guys got together and firmly decided: “We are creating SOC for the commercial market of Russia!” as well as incredible adventures in the wild world of customer infrastructures.
')
We will start from the moment when JSOC from synthetic plans and estimates is ripe for entering the market and running the solutions developed in battle. And for this, technology, ideas, ambitions and gurus of SIEM and analytics are not enough. There is no way without the first line of analysts - the team that provides the front for the main SOC processes.
Our first line of monitoring is different from the first line in the classic sense. This is not a call center and not girls with a script routing application. The role of the classic first line in our paradigm is assigned to the SIEM with the pie of the profiling, aggregating and enriching rules. And the task of the first line of JSOC is to conduct full investigations of IS incidents, including FP filtering, forming analytical information and recommendations on counteraction. A free review is also welcome, especially in situations where unusual events are seen in the vicinity of the incident.
“Any adventure has to start somewhere ...
corny, but even here it is true. "
Lewis Carroll
When the first version of the team was formed, we did not quite represent the exact profile of the analysts' activities, since the development of events in a completely new market is difficult to predict. Therefore, to compensate for possible turns of fate and the vicissitudes of technological trends, we decided to look for guys with a broad technical background.
To assemble a team of such specialists turned out quickly enough, and this greatly helped us at the initial stage. We felt ready for a winding and difficult way of bringing SOC to the market.
However, after some time, we realized that we had miscalculated something: the majority of the hired specialists were clearly overqualified for the position of the first-line engineer, and the v.1.0 team was rather unstable.
Despite the high dynamics of the development of Solar JSOC and the constant emergence of new tasks, the ambitions of the guys were even higher, and the final list of responsibilities was significantly lower than their experience and expectations. Therefore, colleagues quickly grew from the first line - someone within the JSOC, and someone, alas, outside of it.
However, the “virus” of IB, obtained in JSOC, turned out to be very persistent - despite the fact that the v.1.0 team was almost 100% from pure IT, almost all of them continue to work on the information security field.
Nevertheless, this first attempt to assemble a team was very useful for us.
We “on the move” decided what we ultimately do and what we expect from our employees. We are used to the fact that in the new market multiple growth and the attendant permanent training of personnel are the norm and routine. Found that the market can not offer ready-made specialists for work on the first line of SOC. It became clear that in these conditions it is simply utopia to simply pass with a frequent rake on the summary.
“Learning a scientist is only spoiling.”
Russian proverb
So, we find ourselves in a situation in which there are practically no ready-made specialists on the market, and the number of customers begins to grow, which requires scaling the team. At a certain point, we simply came into conflict with the capabilities of the regional personnel market (the first line of Solar JSOC is located in Nizhny Novgorod).
Salvation for us was working with students. At this point, we had little experience of such work in the Moscow team of Solar Security, but it was not aimed at working with the university as a platform, organizing the flow of young professionals. We felt that in the case of Solar JSOC, a more systematic approach would be required.
And here our colleagues, graduates of the Nizhny Novgorod University, who helped us to the UNN student career center, a vigorous student organization interested in the professional employment of students, helped us a lot. We began to regularly participate in the Days of the Employer, telling students and recent graduates of technical universities about the activities of Solar JSOC.
Then we launched an internship program. Initially, it was aimed at students of the specialty of information security, but at present it is extended to students of other IT specialties.
Looking ahead, I will say that this work has given excellent results. For the third year now, young people come to us by "gravity", ready to get tested, train, study and become first-line engineers. And many graduates of our internships really found their first job in Solar JSOC.
- This is too difficult task for my little brain ...
- What matters is not size, but the ability to use.
From Solar JSOC folklore
We understood what the ideal team of an ideal SOC should ultimately end up with. And we are ready to reveal this secret: the team should be first and foremost a team, no matter how trite it may sound. And the phrase of Gennady Zaitsev was given at the beginning of the article not by chance. We approach the selection of personnel, guided by the same principle: “Fingers fluency is a new thing. Had a good man. ” After repeated trial and error, we came to the conclusion that when searching for engineers of the first line, the main focus should be not on the technical background and previous experience. (This, however, does not mean that previous experience is completely unimportant. Of course, good positioning and linux knowledge is welcome for a job seeker in a GIS maintenance team.) We are trying to find guys with “live brains”. Guys who can see a fan of solutions that can reason with good analytical skills.
When there was an influx of potential candidates for internships, and subsequently work, we were faced with the task of evaluating the learning ability of each candidate. Therefore, during the interview we try to bring the applicants to logical reasoning, give logical tasks and change their conditions in order to see how a person adapts and selects the best solution. This approach helps us to select people with whom it is pleasant to work and who are very quickly integrated into the team. Well, the "fluency of the fingers" - it can be acquired. About how we train it - just below.
As an entrance test for an internship, we suggest the guys to deal with this kind of tasks:
- Why does UDP continue to be used when TCP guarantees delivery of data in unchanged form, sequence and lossless?
- You need to write TK on keylogger. Where do you start? By what criteria would you aggregate information for convenient work with the results?
- List the fundamental differences of the Remote Administration Tool such as the Microsoft Remote Desktop Connection and TeamViewer in terms of controlling access to critical hosts?
- At which stages of the APT attack can one encounter the use of these utilities, and how can an attacker use them?
c: \ windows \ system32 \ whoami.exe
c: \ windows \ syswow64 \ netstat.exe
c: \ windows \ syswow64 \ nslookup.exe
The internship program includes:
- Learning the basics of network technology.
- Study of the peculiarities of logging information security events by various sources.
- HPE ArcSight skills.
- Analysis of simple test incidents on the training stand.
- And much more.
During internships, the most attention is paid to test incidents, most of which are created based on real events from the Solar JSOC practice. In addition to the template, automatically generated incidents, interns regularly receive incidents that are multi-layered with tackles for analysis. Incidents that require a special approach, because as a result of a template investigation, students, for example, come to the conclusion that malicious activity was initiated from their own hosts from under their own uchetku.
Such people significantly increase their involvement in the process of learning and investigating incidents, teach them to pay attention to various “trifles”, recheck their conclusions, speculate and prove or disprove them in the future.
Involvement in the investigation process is the most important at this stage. If we are interested in safety and letting them “feel” incidents with real history or incidents of which they are allegedly involved, we motivate them to study the “boring” theory and develop non-standard and non-described approaches to solving the problem.
At the exit from the internship, we get guys half ready to work on the first line. Thus, we form a personnel reserve.
Regardless of whether we take a person to work following an interview or after an internship, at the beginning of the professional path of the Solar JSOC first-line engineer, a training program awaits you, which takes 2 to 4 months to complete. It can be divided into the following blocks:
- The theoretical block includes basic knowledge, starting with information about the company and its organizational structure and ending with an introduction to regulatory documentation, the latest trends in information security, APT, Threat Intelligence.
- The practical block consolidates the knowledge gained at the new level and teaches specific skills of working with the main tool of the first line - SIEM. In addition, the practice includes such mandatory areas as tracking information security incidents in the ticket system, the subtleties of communication with customers, reproducing and analyzing the most interesting incidents on the stand, and much more.
- Next, newcomers are given progressively more complex tolerances to the whitelist-based production environment: the engineer is involved in investigating “combat” incidents according to the scenarios for which he successfully passed the exam and for which he received admission.
- Since our company, like the field of information security itself, is developing rapidly, it is impossible to cover all the details with the training program. Therefore, at the final stage, each future engineer of the first line of monitoring is assigned a curator from the most experienced engineers, so to speak, “practicing surgeons” to study the nuances and control the work performed on a productive environment.
According to the results of the evaluation of the curator and the final control slice, conducted by the commission, the engineer is allowed to “independent swimming”, night / weekend duty.
At first, most engineers work in the paradigm “For an incident of this type, I use such and such tools, they give the necessary data about it.” This methodology allows you to train a person in a short time and bring him to the level when he is able to bring benefits on the line.
As they gain experience, the children realize the commonness of IS incidents, begin to better understand what information and in what section will be the most useful for the customer to respond to a specific incident. Gradually, they move to work in the paradigm "I need such and such information, it can be obtained in many ways, in this situation, such and such is preferable."
A similar restructuring of the approach, coupled with the pumping of the IB background and immersion in content, SIEM prepares children for steps to the next steps, where they are waiting for deep investigations of highly critical incidents and chains of IS incidents or immersion into SIEM systems with all the variety of custom logic.
This ends the formalities and begins a full-fledged life on the first line: for administration - full of complex quests to satisfy the desires of customers and compliance with information security requirements; for monitoring - consisting of many fascinating stories reflected in the logs. Sometimes a day on the first line may resemble a puzzle on several boards of a duty shift: the collected pictures flash before the engineer, changing not only their texture, but also the ticket number :)
But, although the theater begins with a hanger, and the operating life of the SOC - from the first line, it does not end with it. People grow, projects scale, and we have a great many roles and tasks within us. We are very encouraging in the first-line engineers to strive for development, therefore we have worked several “professional elevators”. A certain percentage of engineers who are ready to move to other lines are even included in KPI team lead lines :) But the story of "sublimating expertise" or further growth of personnel in Solar JSOC will become part of other articles. To be continued ...
Source: https://habr.com/ru/post/350474/
All Articles