We at Triplebyte have a lot of interviews. In reality, over the past two years, I interviewed more than 900 engineers. How much is effective use of my time - one can argue here (sometimes I wake up in a cold sweat and doubt it). But regardless of my feelings, the main thing is that we are trying to improve the interview procedure. To do this, we conduct interviews without viewing the summary (background-blind inrterview), we determine the programming skills, and not evaluate the merits and recommendations. After the engineers have passed our interview, they are sent for the final interview directly to the companies we work with (including Apple, Facebook, Dropbox and Stripe). We interview engineers, without knowing anything about their biographies, and then we look at how they manifest themselves in various major IT companies. In my opinion, this is the best test of the effectiveness of the interview.

In this article, I'm going to show what we have been able to understand by now. Technical interviews are largely incorrectly organized. This is easy to say, and many articles talk about it. It is more difficult to correct these shortcomings. My task in this article is to cope with this task and to provide specific advice for hiring managers and technical directors. Interview is a difficult thing. But I think that many problems can be solved if you carefully consider the process. Here I write about the assessment of technical skills. In future articles, we will talk about cultural relevance, behavioral interviews and the assessment of non-technical qualities.

Status quo

Most interviews include two main steps:

• Selection of candidates.
• Personal final interview.

The purpose of the selection is to filter out candidates at the first stage and save engineers time for interviews. In the selection process, the recruiter usually scans the candidate’s resume (in about 10 seconds), after which he talks to him by phone from 30 minutes to an hour. 80% of the companies we work with give the candidate more homework (either instead of or in addition to a telephone interview). Interestingly, at the selection stage, the vast majority of candidates are eliminated. In fact, among all the companies we work with, more than 50% of candidates are eliminated after scanning resumes, and another 30% are eliminated after a telephone interview and homework. Pre-screening is the stage where the hiring procedure is especially capricious. Recruiters are overwhelmed by a huge number of resumes, they have to make instant decisions. Here the merits of the candidates and the principle of comparison with the sample come into force.
')
Personal final interviews almost always consist of a series of 45-60-minute sessions, each with a new interviewer. These are mainly technical sessions (plus one or two for each company for cultural relevance and personal qualities). The final decision on hiring a candidate is made at the general meeting after the candidate’s departure, with the participation of the HR manager and everyone who interviewed the candidate. In fact, for a positive decision, it is necessary that someone alone insists on it, while the others do not have any particular objections ^[1] .

Many final interviews expand the standard format and are very different from each other.

39% of the companies we work with use a whiteboard with a marker.

52% allow you to work at your own computer (the remaining 9% are inconsistent in choosing a board or computer).

55% allow the interviewer to choose the question himself (the remaining 45% use the standard question bank).

40% want to see academic knowledge in computer science to make an offer to the candidate.

15% of companies do not want to see academic knowledge in computer science (and they think that academic talk about computer science is a sign that the candidate will not work effectively).

80% allow you to use any programming language at the interview (the remaining 20% ​​require the use of a specific language).

5% evaluate in detail every little thing about using a programming language during an interview.



Among all the companies we work with, 22% of the final interviews end with a positive hiring decision. (This figure was obtained from the internal statistics of the companies themselves based on their own procedure for selecting candidates. For candidates who pass through Triplebyte, 53% of cases make a positive decision). About 65% of proposals are accepted by candidates (and they get a job). After one year of work, companies express satisfaction with about 30% of new employees and lay off about 5% ^[2] .

False negative and false positive results

So what's wrong with the current state of affairs? At least, the proportion of laid-offs does not seem significant. To understand the problem, you need to consider that the interview is considered unsuccessful for two reasons. Or hire a bad engineer who will later have to be fired (false positive result). Or an interview disqualifies a candidate who could do a good job (a false negative). Unsuccessful hiring is well noticeable and costly for the company (salary, management expenses and morale of the whole team). Unsuccessful hiring sucks energy from the team. But candidates who could do a good job, but they were not given a chance, on the contrary, remain inconspicuous. Every case is always ambiguous. Because of this asymmetry in controversial situations, companies are strongly inclined to failure.

This effect is exacerbated by the “noise” in the interview procedure. It is extremely difficult to evaluate programming skills in one hour. Add to this the principle of pattern matching and subjective “instinctive” feelings, as well as the complex mix of features of each company discussed above - and you will get a very noisy signal.

To keep the level of false positives low, with such a noisy signal, companies have to incline even more to failure. As a result, they are deprived of good engineers, they still value merit more than real skills, and often seem capricious and disappoint those who take part in interviews. If everyone in your company will have to re-interview for the current position, what percentage of it will pass? This is a frightening question. The answer will definitely be a figure much less than 100%. Candidates suffer from the fact that they are rejected by companies where they could work fine, and companies suffer from the fact that they cannot find talent.

To clarify, I am not campaigning to lower the bar on interviews. Failure is the essence of the interview! I do not even say that companies in vain are afraid of false-positive results much more than false-negative ones. Unsuccessful hiring is expensive. I only argue that the noisy signal, coupled with the fear of unsuccessful hiring, leads to a really high level of false negatives, and this is bad for everyone. The solution is to improve the quality of the signal.

Specific ways to reduce noise during the interview

1. Determine what skills you are looking for.

There is no single set of skills that define a good programmer. On the contrary, there is a whole ocean of diverse skill sets. No engineer can be strong at once in all areas. In fact, we at Triplebyte often see excellent, successful programmers with completely incoherent skill sets. Thus, the first step to conducting a good interview is to determine which particular skill set is appropriate for the position. I recommend asking yourself a few questions (these are questions that we ask when we accept a new company for service at Triplebyte).

Do you need fast, iterative programmers or attentive and meticulous?

Do you need someone who is motivated to solve technical problems or create a product?

Do you need skills in a particular technology or a smart programmer who will learn as you work?

Are the academic knowledge of computer science, mathematics, algorithms important?

Is knowledge of concurrency, memory model C or HTTP important?

There are no correct answers to these questions. We work with successful companies that choose different options in each case. But the most important thing is to make a conscious choice based on your needs. It is necessary to avoid random selection of questions for the interview (or allowing the candidate to choose). When this happens, a company’s programming culture may slip into a direction where more and more developers have a certain skill or approach to development that is not particularly important to the company, and engineers without this skill (but with other important skills) are rejected.

2. Ask questions as close as possible to real work.

Professional programmers are hired to solve large, growing problems over weeks and months. But interviewers have no weeks or months to evaluate candidates. Each of them usually has 1 hour. And the interviewers check how quickly candidates solve small problems under duress. This is another skill. Yes, it is relevant to the case (the interview is still not completely random). But it is not perfectly correlated with what is needed. Minimizing this difference is the goal of developing interview questions.

This can be achieved if the questions at the interview are as close as possible to the work that you are expecting from the candidate (or to the skill you are trying to assess). For example, if you are looking for a programmer for the backend, ask the candidate to develop an endpoint API and add functionality to it - this will almost certainly be better than solving the problem of traversing the graph by searching for width. If you are concerned about the knowledge of algorithms, then ask them to apply an algorithm to solve a specific problem (for example, to build a simple search index, possibly with BST and HashMap support for better removal performance) - this is almost certainly better than the task of determining whether this point belongs concave polygon. And in debugging tasks it is almost always better to give the candidate to work with the real code base than to ask him to solve a small problem on the board.

In this case, there is an argument in favor of using the board with a marker. As an interviewer, it does not matter to me that the candidate remembers all the iterators from the set of itertools in Python. It matters to me whether he can find a way to use these iterators to solve a problem. When a candidate draws on the board, he is free from the limitations of exact syntax and can concentrate on logic. But ultimately, I think this is an unfortunate argument, simply because there are not enough rationales for using a different format. In fact, you can get all the same benefits by allowing the candidate to work on your own computer, but by removing the condition for the mandatory launch of this code (or better yet, by conducting an open book interview, that is, with the ability to search for help information in Google).

There is an important caution when developing questions that reflect real work. It is important that the question is free from external dependencies. For example, please write a simple Ruby web scrapper seems like a good problem from the real world. However, if a candidate needs to install Nokogiri (Ruby parsing library, which can be difficult to install) and he will have to spend 30 minutes fighting native extensions, then this is a terrible interview. Not only time is wasted, but the candidate’s stress level rolls over the roof.

3. Ask composite questions that cannot be answered quickly.

Another good rule for interview questions is to avoid questions for which there is a short and complete answer, that is, there is some kind of magic piece of information that a candidate could read to Glassdoor in advance - and this will allow him to easily answer the question. Obviously, this immediately excludes from the number of questions a puzzle or any questions requiring insight. Moreover, questions should consist of several steps, where each subsequent step is based on the previous one, and not one-time questions about the central problem. Why else it is useful - ask yourself how you can help a candidate who makes a good overall impression, but is stuck on a question. In the case of one-time questions, if you provide significant assistance to the candidate, then the interview failed. In a multi-part compound task, you can help in one step, and the candidate will successfully complete the remaining steps on their own.

This is important not only because your questions can get on Glassdoor. More importantly, composite questions reduce noise. Good candidates may get stuck at some point due to stress. The ability to help them and observe the recovery process is very essential. There is considerable noise in how well a candidate copes with a specific piece of programmer logic, depending on whether he has recently encountered a similar problem, this is just a random probability. Multi-part composite tasks eliminate some of this noise. They also give candidates the opportunity to observe how the effect is increasing. Efforts at one stage often help solve a problem in the next stage. This is an important dynamic of real-world development processes, and studying it during an interview reduces the noise level.

If examples are needed, then the request to the candidate to realize the game “Four in a row” in the terminal (a series of multiple steps) is likely to be a better task than a request to rotate the matrix (the only step, with some tricks that ultimately simplify the task). And k-means clustering (several operations based on each other) is likely to be a better task than determining the largest rectangle that is placed under the histogram.

4. Avoid hard questions.

If a candidate really well solved a difficult problem, it says a lot about his abilities. But since the question is complex, most candidates will not be able to answer it well. In this case, the expected amount of information received from the question depends on its complexity. We concluded that the optimal level of complexity is much lower than most interviewers believe.

The effect is enhanced by the fact that during the interview the candidate has two sources of information: 1) whether he gives the “correct” answer to the question and 2) the process of thinking, that is, how easily he thought of this answer. We at Triplebyte collect these indicators (an assessment for the correctness of the answer and an assessment for the amount of effort that he needed to do this, and then we make a prediction of what estimates correlate with the success of the employee in different companies). What we have found is a kind of compromise. If the candidate answers more complex questions, then we get a stronger signal directly from the answer. For simpler questions, on the contrary, the process and how much the candidate suffered was more important. Taking into account both signal sources, the optimum is closer to simple questions.

In practice, such a rule has developed: the interviewer himself should solve the problem for 25% of the time that he expects from the candidate to solve this problem. So if I develop a new task for an hour-long interview, then my colleagues (without warning) should give the correct answer in 15 minutes. Given the fact that we propose to solve multistage problems from the real world, the optimal task for an interview should really be quite clear and simple.

To clarify, I do not campaign for lowering the bar in the sense of lowering the passing score. I only argue in favor of simple questions and further evaluation of how the candidates respond to them. I stand for simple questions and a very strict assessment of the answers. This is what optimizes the signal best in our experience. Here, there is an added benefit of reducing the level of stress for most candidates.

I will give examples. Asking the candidate to create a simple command line interface with commands to put into storage and extract key-value pairs from there (and add additional functionality if the candidate is doing well) is probably better than asking for a parser for arithmetic expressions. And the question of the most common data structures (lists, hashes, maybe trees) is probably better than the question of lists with passes, duchi (tree + heap) or other little-known structures.

5. Ask each candidate the same questions.

The essence of the interview versus candidates. The goal is to sort out candidates for those who are able to help the company and those who are not capable (and in the case of filling a single vacancy, choose the best one). Given this, it is completely illogical to ask different questions to different candidates. If you evaluate candidates for the same job in different ways, you add extra noise.

I think the reason that interviewers often choose different questions for different candidates is in the interviewer's personal preferences. IT engineers usually do not like to conduct interviews. This is something that they do only sporadically, and it distracts from the main work. In order to standardize questions for each candidate, the interviewers will have to spend more time studying these questions, internalizing the rating system and reporting. And they will have to undergo this procedure again with each change of question. And to always ask the same question is a little more boring.

Unfortunately, the only way out for interviewers here is to make more effort. Uniformity is a key element in conducting good interviews, which means asking each candidate the same question and standardizing reports. There simply is no other option.

6. Consider the possibility of multiple paths.

Although this is contrary to the previous paragraph, but consider the possibility of several completely different versions of the interview. From the start, when designing the interview, think about what candidate’s skills matter. But some of the answers may be contradictory! For example, a completely normal situation, if companies need engineers who are well versed in mathematics, but they also need very productive / iterative engineers (maybe even for the same position). For such cases, consider several ways to conduct an interview. The key factor here is that the scale of the task should allow fully standardize each of the paths. That's what we do at Triplebyte. We concluded that you can simply ask the candidate what type of interview he prefers.

7. Do not allow candidate resumes to influence you.

The summary is not entirely meaningless. MIT alumni and Stanford graduates or with experience at Google and Apple are really better as a group than others. The problem is that the vast majority of engineers (including me) do not belong to this group. So if a company relies too much on these signals, then most qualified candidates will pass by. Taking into account a resume to some extent at the stage of preliminary degree may have some meaning. We do not do this at Triplebyte (all our assessments in blind tests are done completely without taking into account the experience and merit of the person). But it makes sense to give some weight of the summary at the stage of preliminary selection.

But to take into account the summary at the interview stage is completely meaningless. And we have evidence that this is actually happening. With the same scores in our “blind tests”, candidates with a diploma from a top university successfully pass interviews with companies 30% more often than candidates without a brand name in the resume. If interviewers know that a candidate has a diploma from the Massachusetts Institute of Technology, they are more likely to turn a blind eye to the shortcomings during the interview.

This is noise and should be avoided. The easiest way is to simply remove the names of universities and companies from the resume before giving them to the interviewers. Some candidates may mention the university or company during a call, but we always conduct interviews without taking this information into account, and candidates very rarely pay attention to these facts during technical tests.

8. Do not mock candidates.

One of the nastiest ways to fail at an interview is when it ends in a mocking way. Interviewers no longer only assess a candidate’s abilities, they also become members of a group or team that accepts a new member to their group. In this second capacity, they conduct something like a rite of passage. Yes, the interview itself is stressful, but we all went through this stress, so that candidates will have to go through it. The situation is aggravated when the candidate can not cope with the task. As an interviewer, you may experience a strong disappointment by watching the candidate beating his head against the wall, solving a problem where the answer is so obvious! You can flare up and break. Of course, this only increases the stress for the candidate - and it becomes even harder for him to solve the problem.

It is better not to allow such situations in any case. The solution will be to discuss the problem and train the interviewers on how to overcome it. Here we use one trick: when a candidate fails at all, you need to switch from the evaluation mode, where the goal is to assess the person, to the training mode, where the goal is to make the candidate understand the answer to the question. Mental switching from one mode to another helps a lot. When you are in training mode, there is no reason to hide information or show any other emotions other than friendliness.

9. Make decisions based on maximum skill, not average or minimum.

So far, I have been talking only about individual issues, and not about the decision at the final interview. Here my advice is to make a decision based on the maximum skill that the candidate has demonstrated (among all the skills that are important to you), and not at an average or minimum level.

Probably you are already doing this, intentionally or not! The final decision is made at the general meeting of all interviewers, and the proposal is made to the candidate if at least one interviewer strongly recommends doing so, while the others do not strongly object. In order to be impressed by one interviewer, a candidate just needs to flash on one section of the interview. According to our data, the maximum skill is an attribute that is strongly correlated in order to show itself well in at least one section of the interview in the company. However, in order to get a job offer, the candidate also needs not to turn any of the other interviewers against himself. Categorical objections from them may appear if the candidate looked really stupid when answering a question.

Here we are dealing simply with a lot of noise. There are so many areas of activity of a highly qualified engineer that almost no one is able to achieve perfection in all areas at once. This means that if you ask the right (or wrong) question, then any engineer can look stupid. After that, candidates receive offers in the event that at least one interview affected a zone of strong knowledge (maximum skill) and none affected areas of significant weakness. The problem is that it increases the noise level. The same engineer who failed one interview, because he looked stupid after the question of network technologies, brilliantly passes the rest of the interviews where this topic was not addressed.

In my opinion, it is best for companies to concentrate on maximum skill and not resist hiring employees who have performed poorly in certain sections of the interview. That is, to seek strong arguments in favor of the candidate and not worry so much about the technical areas where he is weak. Here I do not want to be definitely categorical. Of course, there are technical issues that are simply very important to the company. And your decision that all employees must have a certain level of knowledge in these areas makes perfect sense. But the concentration on the maximum skills of the candidate really reduces the noise at the interview.

Why do interviews at all?

The last question I would like to ask is: why do I have to have interviews at all? I am sure that many readers already gritted their teeth, saying: “What is the point of discussing the inefficient system in such detail? Just use your homework! Or just take people for a trial period! ”In the end, some very successful companies actually use a trial period (where the candidate works with the team for a week) or completely replace personal interviews with homework. The trial period in many ways makes sense. Spending a week side by side with a new employee (or watching a team with him carry out an important project) almost always gives a better assessment of a candidate’s skills than monitoring for just one hour, how he solves problems. However, there are two problems that do not give a trial period to replace standard interviews:

1. The trial period is expensive for the company. No company can take for a week everyone who wants to find a job. To decide who to take on probation, will have to hold interviews.
   
2. The probationary period (and large projects for home execution) cost the candidate dearly. Even if the work is paid, not everyone has time. For example, if an engineer works full time and he has no opportunity to take time off. And even if there is an opportunity, many do not want to take it. If an engineer already has a job, then he is not in the mood to accept the uncertainty of the probationary period. We see this well with the example of the Triplebyte candidates. Many of the best candidates (already having offers from other firms) simply will not carry out large projects and will not agree to a trial period.

It turns out that the trial period is a great option for some candidates. I think if the scale allows you to use different ways of the interview, then adding a trial period to them is a great idea. However, it is impractical to completely replace the interview with them.

Discussions of work experience on past projects are also often viewed as a substitute for technical interviews. The logic is this: to make sure that the candidate will successfully complete the work in the future, just see what he did in the past. We in Triplebyte tried this technique. Unfortunately, it did not show good results. It turned out that the communicative ability (ability to sell oneself) turned out to be a stronger signal than the candidate’s technical abilities. It’s just too often a situation arises when a candidate who is able to speak beautifully exaggerates his role (appropriates the result of the whole team) or when a modest person understates his merits. If there is enough time and enough questions, you can get to the bottom of it. However we found outthat with standard time limit discussion of past experience is not a universal replacement for the interview. This is a good way to break the ice at the beginning of a conversation, to understand the meaning of human interests (also assess communication skills and cultural relevance). But the discussion of past experience is not suitable as a complete replacement for the interview.

Something good about programming interviews!

I want to finish this article on a more pleasant note. Along with all the shortcomings of interviews, they still bring great benefits.

Interviewing is a direct assessment of ability. I have friends who work as teachers. They say that teacher interviews are mainly an assessment of the communicative ability (ability to sell themselves) and a summary / recommendation. Apparently, this situation in many, many professions. Silicon Valley is not an ideal meritocracy. But at least we are trying to directly measure the skills that matter, and adhere to the idea that anyone with such skills can become a good professional, regardless of his education and work experience. Resume evaluation often gets in the way. But we at Triplebyte have managed to largely overcome this problem and help a large number of people with non-standard resumes to get an excellent job in IT. I doubt that a company like Triplebyte could work, for example, in the legal sphere.They rely too heavily on summaries and recommendations.

Programmers also prefer interviews. Although this is a very controversial topic (there will certainly be programmers with a different point of view), but when we experimented with alternative methods for evaluating candidates, most programmers still chose standard interviews. And we found out that only a small part of programmers are interested in companies that offer probation or homework. Good or bad, but apparently, interviews for programmers will not go anywhere. Other ways are a good addition, but they are unlikely to replace the interview as the main way to evaluate programmers. If Churchill is wrong to quote: “Interviewing is the worst way to evaluate an engineer, if not to consider other methods that we sometimes try.”

Conclusion

Interview - not an easy thing. Human beings are hopelessly complex. In a sense, an attempt to assess the capabilities of a person in 4 hours of interviews seems stupid. I think it is important to maintain modesty. Any interview process is doomed to frequent failures. People are just too complicated.

---

[1] Of course, there are various options. At the opposite end of the spectrum are companies that require a unanimous positive decision from each interviewer, and a company where the HR manager alone makes the decision. ↑

[2] These are figures from the internal statistics of the companies themselves about their own candidates. And they differ greatly in different companies (for example, the share of laid-offs varies from 1% to 30%). Triplebyte candidates have better numbers. To date, our candidates receive job offers after 53% of interviews, and the rate of dismissals is 2%. ↑