What tests and interviews test
In a previous article, habr.com/ru/post/450810, I looked at 7 ways to quickly test the competencies of IT specialists that can be applied before a large, lengthy and time-consuming technical interview. There I examined the essence of these methods and my practice of their application, as well as the reasons why I like or dislike them.
In this article, I want to talk about the modern concept of human decision-making, how it relates to the testing of work skills, and what methods of testing competencies, such as interviews and tests, are actually testing.
For many centuries scientists have been concerned with the question - how and why does a person make certain decisions? In each epoch, this question was answered differently - for millennia belief in the fate and will of the gods prevailed, then for a long time the opinion was popular that man is a rational being, which basically acts rationally and prudently. The scientific revolution led to the fact that in the second half of the 20th century, a lot of research was conducted on the behavioral responses of "reasonable man". And at the moment the most modern and scientifically recognized concept is the hybrid model of human behavior, about which psychologist Daniel Kahneman wrote very well in his scientific articles and popular science books. Daniel received the Nobel Prize in Economics due to the fact that his work was refuted by many economic theories based on models of rational decision-making by man. Daniel Kahneman convincingly showed that human behavior in most situations is due to automatic behavioral responses that are formed on the basis of life experience.
According to the concept of Daniel Kaneman, human behavior is controlled by two interacting decision-making systems. System 1 - fast and automatic, ensures the safety of the body and does not require significant effort to form a solution. The accuracy of the decisions of this system depends on experience and training, and the speed depends on the characteristics of the individual's nervous system. System 2 is slow, requiring effort and concentration. It provides us with complex reasoning, logical conclusion, reasonable forecasting. The speed of decision making by this system is tens and hundreds of times lower than the speed of System 1. It is during the operation of System 2 that the full potential of human intelligence is manifested. However, during the operation of this system, resources are strenuously consumed - both physical (energy) and attention, which is derived from a multitude of resources. Therefore, most decisions are made by System 1.
I think that each of you noticed that he could not think hard and solve complex problems for more than a period of time in a row. Each person has a different interval. Someone can think hard only half an hour a day, and someone can solve the most difficult tasks for 3 hours in a row. This ability can be developed, but it is given by very hard work and efforts on oneself and still the resource of attention will be limited.
Both systems work together. Information coming from the senses is first processed by Quick System 1, which recognizes dangerous situations and reacts immediately in case of threats. Also, System 1 recognizes unfamiliar situations and either decides to ignore them or activates System 2.
How it works? Have you ever watched professional chess players play - at what speed do they make moves at the beginning of the game? For a person who rarely plays chess, it seems that it is impossible to make such difficult decisions so quickly. However, you can correct trainee errors on the full machine when you make a code review. Your System 1 is able to recognize typical mistakes of novice programmers and automatically correct them, just like a professional chess player reads the situation on the board and knows how to walk, practically without straining the conscious System 2.
Numerous experiments have shown that in familiar situations, decisions are almost always made by the automatic System 1, and this is quite rational from the point of view of the survival of the organism and the expenditure of energy. In this regard, we act very rationally and optimally, but not in the sense of deliberation and optimality of the decisions themselves, but in the sense of a balance between the result and the expenditure of resources of our body. When you drive a car in the city on the way to work, your turning trajectory and the number of accelerations and decelerations may not be optimal, but from the point of view of the task of getting you from your home to your work, everything is very good. If you are a race car driver and drive a race car on a race track, your decisions about the trajectory, acceleration and deceleration will be much more calculating.
In unfamiliar situations that we are interested in or which could not be avoided, we are forced to act consciously, connecting attention and System 2. After several repetitions of very similar situations, the result of the work of System 2 is stored in the form of signs and reactions and then you will not have to spend strength and time for logical conclusions - System 1 will already be trained on this task and next time will provide a solution automatically. Some automatic reactions are lost over time if they are not periodically claimed. Skills that we do not train are lost.
Numerous experiments have shown that for the first time in a new job a normal, mentally healthy person adapts and tries to accept the rules, conditions and working processes of a new job. However, after some time, each of us relaxes and begins to work as we can. Efforts and diligence give way to automatic reactions and patterns fixed in System 1. Moreover, even during a trial period in stressful situations when a quick solution is required, we react with the help of automatic System 1 and not always the way we were taught on this new work place.
In general, we can say that our core value as an employee is largely determined by our experience - that is, the training of our System 1 in solving certain tasks needed by the employer. Therefore, employers very often want an employee not with outstanding intelligence, but with experience in some field. Experience is valued more than intelligence. This is due to elementary calculations. If there is enough time, then any employee with sufficient intelligence will be able to understand the topic and solve the tasks. However, he will have to spend time on learning and gaining experience, and only then he will be able to qualitatively solve the tasks. His System 2 will have to solve a lot of training tasks before its System 1 can quickly and efficiently solve real problems. This takes time, which the employer is often not ready to pay at the high professional rate. Another employee who has already solved similar tasks will do the work much faster, because most of the solutions will be given to him by System 1, trained to solve problems in the required field. An experienced employee will produce high-quality solutions not only faster, but less straining at the same time. This means that unused resources of attention can be directed to solving new challenges and getting new experience.
What to choose - experience or intelligence - the employer decides in each case individually. Where rapid response to a typical task and a quick solution is required, experience is often chosen. If you have to solve many different tasks, but the solution time is still highly valued, then choose the experienced and smart. If time is not very critical, then you can give preference to an intellectual without experience. As you understand, in the real world there are few jobs where time is uncritical.
')
Experience - that is, the training of System 1 - is often an important, perhaps even decisive, criterion when an employer selects a new employee. How can we evaluate the candidate’s experience most effectively and accurately? Let's look at the popular ways of assessing competencies in terms of what they value.
This format implies the conversation of the candidate and evaluator. Mostly, the questions are asked by the evaluator, but the candidate has the opportunity to read non-verbal signs, ask clarifying questions and, as they say, change his answer “on the fly”. This is an oral examination that is familiar to all of us. As a rule, the interview takes place according to a standard plan and many questions are also typical, which means you can prepare for them. That is, train your System 1 for successful interviewing.
The success of the candidate's assessment depends on the communication skills of both participants. A candidate experienced enough in interviewing may well make a good impression. However, this result is not due to work experience, but due to the experience of communication and interviewing. A prepared candidate who responds well to typical questions influences the examiner and he becomes more loyal to the candidate.
This method basically tests System 1 candidate, although often not the kind of experience that will be needed in the work. It is well suited for evaluating professionals who will have to communicate a lot on official duties and quickly adapt, but for my technical skills assessment, in my opinion, this method is poorly suited. It is possible to improve the accuracy of the assessment due to non-standard questions and interview scenarios, as well as through participation in the interview of several evaluators, which leads to a higher cost of this event.
The candidate receives the task, which is solved independently and then demonstrates the result of the decision. In fact, this is a familiar “written exam”. The candidate has enough time, the opportunity to ask clarifying questions, and also to search for information on the network and even use the help of friends. If the task is complex and time is sufficient, then this method tests System 2 rather than System 1, that is, intelligence, not experience. If, however, to reduce the time to complete the task, then there is a high probability that candidates will refuse to perform a complex test task. If at the same time simplify the task, give several tasks and reduce the time - then this method becomes quite a working tool, familiar to us from school. It tests System 1 quite well. However, its disadvantage is that the verification of results requires considerable effort on the side of evaluating specialists, since each solution can be unique and the verifiers must penetrate into the essence of the solution.
The candidate receives an uncomplicated task, which is solved under the supervision of an evaluator. This method is often used in the interview process - when evaluating specialists first talk and then propose to solve the problem. For introverted candidates who have not been interviewed for a long time, this method is often psychologically uncomfortable, and they show not very good results. In my opinion, this method should be offered to candidates as an alternative to the test task. That is, either 3-4 hours of independent work, or 1-1.5 hours of interviews and solving problems online. If the candidate is ready, this method allows you to test the basic skills of System 1 on typical tasks that are part of more complex work tasks. That is, elements of real work tasks should be chosen as test tasks. You should not offer abstract tasks that your employee will never meet later in work.
As you probably know, the final exams in Russian schools are now in the form of tests (GIA and EGE). At the time, this caused a heated debate. Citizens in general, negatively evaluated this decision of the Ministry of Education. Personally, I think that if we do not take into account new opportunities for corruption, then replacing written exams with tests is a good solution. Verification of test results does not require much time and attention, it is easily automated. At the same time, the subjectivity of the assessment of knowledge is minimized. Tests allow for 1-2 hours to qualitatively test the knowledge and experience gained during several years of study or work. The novice driver learns the rules of the road for several months, and on the exam he must answer 20 questions within 20 minutes. The practice of decades of application of this type of exam shows that this is sufficient if the test questions are correct and there are a lot of them.
In the modern world, most of the decisions of a person are reduced to the choice of one of the existing options, the most appropriate to the situation. You are unlikely to need a specialist who will reinvent the wheel. But then a specialist who is well aware of the advantages and disadvantages of different types of bicycles and similar types of transport will be useful to you, will help you quickly choose the right model and customize it to meet your needs. To solve logistical problems usually it is necessary quickly and there is no time to invent an innovative bicycle. Sometimes (very rarely) there are situations when you still need a new bicycle, which so far does not exist and needs to be created. However, in this case, a person who is well versed in the construction of bicycles will be more useful than the universal inventor.
One more example. If a programmer can implement several sorting algorithms - then he is certainly well done, but in real life he will be more interested in knowing the basic methods of the base class library of the language - there are probably several sorting options already implemented, you just need to call the desired function.
It is important that when choosing the method of testing competencies you turn on your System 2 and choose the appropriate method intelligently, and not according to tradition - “because we have always done so.” When choosing a method of checking competencies, I advise first of all to decide on what will be more important for you as an employer in the daily activities of your employee. Will it be the ability to quickly solve a certain range of typical tasks, or will have to solve complex, original, atypical tasks.
In most cases, time-limited tests are perfect for you as a first test for candidates. I recommend small tests, which take no more than 15-20 minutes to complete. During this time, you can ask 30-40 questions and test the candidates' knowledge in sufficient detail. Then you can hold an interview, during which to disassemble the mistakes made by the candidates. The test can also serve as a reference plan for the interview, during which you can clarify with the candidate the reasons why he answered the test questions in the same way and how he would answer if the question were different.
If it is important to you how the future employee works independently on sufficiently large and isolated tasks, then it will be appropriate to start with an interview and then offer to perform the test task. It is worth remembering that only 20-25% of candidates agree to perform test tasks before the interview, and in this case, you greatly reduce the selection funnel.
In my next article I will discuss in more detail the features of creating tests to test the competencies of candidates.
In this article, I want to talk about the modern concept of human decision-making, how it relates to the testing of work skills, and what methods of testing competencies, such as interviews and tests, are actually testing.
A bit of theory
For many centuries scientists have been concerned with the question - how and why does a person make certain decisions? In each epoch, this question was answered differently - for millennia belief in the fate and will of the gods prevailed, then for a long time the opinion was popular that man is a rational being, which basically acts rationally and prudently. The scientific revolution led to the fact that in the second half of the 20th century, a lot of research was conducted on the behavioral responses of "reasonable man". And at the moment the most modern and scientifically recognized concept is the hybrid model of human behavior, about which psychologist Daniel Kahneman wrote very well in his scientific articles and popular science books. Daniel received the Nobel Prize in Economics due to the fact that his work was refuted by many economic theories based on models of rational decision-making by man. Daniel Kahneman convincingly showed that human behavior in most situations is due to automatic behavioral responses that are formed on the basis of life experience.
According to the concept of Daniel Kaneman, human behavior is controlled by two interacting decision-making systems. System 1 - fast and automatic, ensures the safety of the body and does not require significant effort to form a solution. The accuracy of the decisions of this system depends on experience and training, and the speed depends on the characteristics of the individual's nervous system. System 2 is slow, requiring effort and concentration. It provides us with complex reasoning, logical conclusion, reasonable forecasting. The speed of decision making by this system is tens and hundreds of times lower than the speed of System 1. It is during the operation of System 2 that the full potential of human intelligence is manifested. However, during the operation of this system, resources are strenuously consumed - both physical (energy) and attention, which is derived from a multitude of resources. Therefore, most decisions are made by System 1.
I think that each of you noticed that he could not think hard and solve complex problems for more than a period of time in a row. Each person has a different interval. Someone can think hard only half an hour a day, and someone can solve the most difficult tasks for 3 hours in a row. This ability can be developed, but it is given by very hard work and efforts on oneself and still the resource of attention will be limited.
Both systems work together. Information coming from the senses is first processed by Quick System 1, which recognizes dangerous situations and reacts immediately in case of threats. Also, System 1 recognizes unfamiliar situations and either decides to ignore them or activates System 2.
Please multiply 65 by 15 and note for yourself how long it took you to do these calculations.
How it works? Have you ever watched professional chess players play - at what speed do they make moves at the beginning of the game? For a person who rarely plays chess, it seems that it is impossible to make such difficult decisions so quickly. However, you can correct trainee errors on the full machine when you make a code review. Your System 1 is able to recognize typical mistakes of novice programmers and automatically correct them, just like a professional chess player reads the situation on the board and knows how to walk, practically without straining the conscious System 2.
Please multiply 65 by 15 again and note for yourself how long it took you to do this calculation.
Numerous experiments have shown that in familiar situations, decisions are almost always made by the automatic System 1, and this is quite rational from the point of view of the survival of the organism and the expenditure of energy. In this regard, we act very rationally and optimally, but not in the sense of deliberation and optimality of the decisions themselves, but in the sense of a balance between the result and the expenditure of resources of our body. When you drive a car in the city on the way to work, your turning trajectory and the number of accelerations and decelerations may not be optimal, but from the point of view of the task of getting you from your home to your work, everything is very good. If you are a race car driver and drive a race car on a race track, your decisions about the trajectory, acceleration and deceleration will be much more calculating.
In unfamiliar situations that we are interested in or which could not be avoided, we are forced to act consciously, connecting attention and System 2. After several repetitions of very similar situations, the result of the work of System 2 is stored in the form of signs and reactions and then you will not have to spend strength and time for logical conclusions - System 1 will already be trained on this task and next time will provide a solution automatically. Some automatic reactions are lost over time if they are not periodically claimed. Skills that we do not train are lost.
Please multiply 65 by 15 again. Have you noticed progress compared to the previous attempt to solve this problem?
How does all this relate to work and competency testing?
Numerous experiments have shown that for the first time in a new job a normal, mentally healthy person adapts and tries to accept the rules, conditions and working processes of a new job. However, after some time, each of us relaxes and begins to work as we can. Efforts and diligence give way to automatic reactions and patterns fixed in System 1. Moreover, even during a trial period in stressful situations when a quick solution is required, we react with the help of automatic System 1 and not always the way we were taught on this new work place.
In general, we can say that our core value as an employee is largely determined by our experience - that is, the training of our System 1 in solving certain tasks needed by the employer. Therefore, employers very often want an employee not with outstanding intelligence, but with experience in some field. Experience is valued more than intelligence. This is due to elementary calculations. If there is enough time, then any employee with sufficient intelligence will be able to understand the topic and solve the tasks. However, he will have to spend time on learning and gaining experience, and only then he will be able to qualitatively solve the tasks. His System 2 will have to solve a lot of training tasks before its System 1 can quickly and efficiently solve real problems. This takes time, which the employer is often not ready to pay at the high professional rate. Another employee who has already solved similar tasks will do the work much faster, because most of the solutions will be given to him by System 1, trained to solve problems in the required field. An experienced employee will produce high-quality solutions not only faster, but less straining at the same time. This means that unused resources of attention can be directed to solving new challenges and getting new experience.
What to choose - experience or intelligence - the employer decides in each case individually. Where rapid response to a typical task and a quick solution is required, experience is often chosen. If you have to solve many different tasks, but the solution time is still highly valued, then choose the experienced and smart. If time is not very critical, then you can give preference to an intellectual without experience. As you understand, in the real world there are few jobs where time is uncritical.
Please multiply 65 by 15 again and note for yourself how long it took you to do this calculation. Have you noticed how you got the result?
')
Ways of testing competences from the point of view of testing "Systems 1" and "Systems 2"
Experience - that is, the training of System 1 - is often an important, perhaps even decisive, criterion when an employer selects a new employee. How can we evaluate the candidate’s experience most effectively and accurately? Let's look at the popular ways of assessing competencies in terms of what they value.
Job interviews
This format implies the conversation of the candidate and evaluator. Mostly, the questions are asked by the evaluator, but the candidate has the opportunity to read non-verbal signs, ask clarifying questions and, as they say, change his answer “on the fly”. This is an oral examination that is familiar to all of us. As a rule, the interview takes place according to a standard plan and many questions are also typical, which means you can prepare for them. That is, train your System 1 for successful interviewing.
The success of the candidate's assessment depends on the communication skills of both participants. A candidate experienced enough in interviewing may well make a good impression. However, this result is not due to work experience, but due to the experience of communication and interviewing. A prepared candidate who responds well to typical questions influences the examiner and he becomes more loyal to the candidate.
This method basically tests System 1 candidate, although often not the kind of experience that will be needed in the work. It is well suited for evaluating professionals who will have to communicate a lot on official duties and quickly adapt, but for my technical skills assessment, in my opinion, this method is poorly suited. It is possible to improve the accuracy of the assessment due to non-standard questions and interview scenarios, as well as through participation in the interview of several evaluators, which leads to a higher cost of this event.
Test items
The candidate receives the task, which is solved independently and then demonstrates the result of the decision. In fact, this is a familiar “written exam”. The candidate has enough time, the opportunity to ask clarifying questions, and also to search for information on the network and even use the help of friends. If the task is complex and time is sufficient, then this method tests System 2 rather than System 1, that is, intelligence, not experience. If, however, to reduce the time to complete the task, then there is a high probability that candidates will refuse to perform a complex test task. If at the same time simplify the task, give several tasks and reduce the time - then this method becomes quite a working tool, familiar to us from school. It tests System 1 quite well. However, its disadvantage is that the verification of results requires considerable effort on the side of evaluating specialists, since each solution can be unique and the verifiers must penetrate into the essence of the solution.
Live doing
The candidate receives an uncomplicated task, which is solved under the supervision of an evaluator. This method is often used in the interview process - when evaluating specialists first talk and then propose to solve the problem. For introverted candidates who have not been interviewed for a long time, this method is often psychologically uncomfortable, and they show not very good results. In my opinion, this method should be offered to candidates as an alternative to the test task. That is, either 3-4 hours of independent work, or 1-1.5 hours of interviews and solving problems online. If the candidate is ready, this method allows you to test the basic skills of System 1 on typical tasks that are part of more complex work tasks. That is, elements of real work tasks should be chosen as test tasks. You should not offer abstract tasks that your employee will never meet later in work.
Answer Tests
As you probably know, the final exams in Russian schools are now in the form of tests (GIA and EGE). At the time, this caused a heated debate. Citizens in general, negatively evaluated this decision of the Ministry of Education. Personally, I think that if we do not take into account new opportunities for corruption, then replacing written exams with tests is a good solution. Verification of test results does not require much time and attention, it is easily automated. At the same time, the subjectivity of the assessment of knowledge is minimized. Tests allow for 1-2 hours to qualitatively test the knowledge and experience gained during several years of study or work. The novice driver learns the rules of the road for several months, and on the exam he must answer 20 questions within 20 minutes. The practice of decades of application of this type of exam shows that this is sufficient if the test questions are correct and there are a lot of them.
In the modern world, most of the decisions of a person are reduced to the choice of one of the existing options, the most appropriate to the situation. You are unlikely to need a specialist who will reinvent the wheel. But then a specialist who is well aware of the advantages and disadvantages of different types of bicycles and similar types of transport will be useful to you, will help you quickly choose the right model and customize it to meet your needs. To solve logistical problems usually it is necessary quickly and there is no time to invent an innovative bicycle. Sometimes (very rarely) there are situations when you still need a new bicycle, which so far does not exist and needs to be created. However, in this case, a person who is well versed in the construction of bicycles will be more useful than the universal inventor.
One more example. If a programmer can implement several sorting algorithms - then he is certainly well done, but in real life he will be more interested in knowing the basic methods of the base class library of the language - there are probably several sorting options already implemented, you just need to call the desired function.
Conclusion
It is important that when choosing the method of testing competencies you turn on your System 2 and choose the appropriate method intelligently, and not according to tradition - “because we have always done so.” When choosing a method of checking competencies, I advise first of all to decide on what will be more important for you as an employer in the daily activities of your employee. Will it be the ability to quickly solve a certain range of typical tasks, or will have to solve complex, original, atypical tasks.
In most cases, time-limited tests are perfect for you as a first test for candidates. I recommend small tests, which take no more than 15-20 minutes to complete. During this time, you can ask 30-40 questions and test the candidates' knowledge in sufficient detail. Then you can hold an interview, during which to disassemble the mistakes made by the candidates. The test can also serve as a reference plan for the interview, during which you can clarify with the candidate the reasons why he answered the test questions in the same way and how he would answer if the question were different.
If it is important to you how the future employee works independently on sufficiently large and isolated tasks, then it will be appropriate to start with an interview and then offer to perform the test task. It is worth remembering that only 20-25% of candidates agree to perform test tasks before the interview, and in this case, you greatly reduce the selection funnel.
In my next article I will discuss in more detail the features of creating tests to test the competencies of candidates.
Source: https://habr.com/ru/post/455088/
All Articles