tl; dr: Questions and tasks at programming interviews seem overly complex. Sometimes it does that adds stress. This is not the only argument against them. Our data shows that more complex tasks actually predict a worse outcome than simpler ones.

It is difficult to program under the pressure of time. Especially at the interview. An assignment that seems simple under normal conditions somehow causes huge problems in the bright light of the interview room. Stress hormones cloud the brain (unfortunately, neither a fight nor a flight will be an effective answer to a coder problem). There is a feeling that the questions seem to be specially designed with perverted complexity. I think these feelings do not arise from scratch.

In fact, the tasks for the interview are intentionally complex . Since a mistake in hiring is more expensive than a failure to a good engineer, this pushes companies to set the bar high. So, ask hard questions. Intuitively, it makes sense, because more complex questions seem to better screen out the weak and identify the strong. But intuition brings here. Our data shows that more complex questions actually produce a less predictable result than relatively easy ones.

Simpler questions

Difficult tasks filter out bad engineers, but also filter out good ones (that is, a lot of false negative results). On the contrary, simple questions have less false-negative, but more false-positive results (as more engineers answer correctly, including some bad ones). These two signals need to be somehow balanced when choosing the optimal level of complexity of the issues. If a company seeks to avoid false alarms at all costs, it is more likely to lean towards the tougher end of this spectrum.
')
However, the correct answer is not the only signal source during the interview . You can evaluate the whole process: how long it takes to solve, how clean the code is, and how much effort the solution required. Our analysis shows that this second signal source (process) is almost as important as the first one (correctness).

But here is an additional compromise . The questions that carry the greatest signal of the process are much simpler than the questions that carry the greatest signal of correctness. The reason becomes clear if we reduce the process to how much effort the solution required (the aspect of the process most directly related to the difficulty of the task). If the question is rather complicated, then all the candidates will have difficulties (even those who ultimately answer it correctly). So this aspect carries almost no signal when evaluating the effort.

Conversely, tasks with a strong process signal are easy enough for most candidates, that is, the correct answer in itself says little. The optimal complexity is when the signals of the process and the correctness of the response are balanced to achieve the maximum signal, and the signal is not necessarily the peak for both factors.

We interviewed thousands of engineers and evaluated their responses in many ways, including process and accuracy, and compared these estimates with subsequent performance. And after the regression analysis (looking at both the process and the signal of the correct response), our data showed that the greatest correlation is given by questions that are actually much simpler than we expected (and simpler than the questions asked by many companies). ).

It turned out that the more difficult tasks ultimately filter out too many qualified candidates. Thus, if you want to make the hiring process more accurate, you should probably ask simpler questions.

But I want to clearly state that this does not mean lowering the bar and accepting more candidates. Setting simpler tasks doesn’t mean making the interview easier . The level of complexity of the issues and the decision threshold are independent of each other. The hiring process can still be very strict if asked relatively simple questions, but it is demanding to evaluate them. Our conclusion is that simpler questions give more signals. And what you will do with this signal depends only on you.

Simpler questions also reduce stress, which is an important advantage. Stress makes candidates underestimate their capabilities. When a candidate feels more comfortable, he shows the maximum potential, which makes the interview more predictable. It seems to me that interviewers tend to underestimate the effect of stress on candidates, while at the same time overestimating their own abilities. When you set a task, it is easy to forget how much code in reality you can write in 30-60 minutes. To counter this bias, we at Triplebyte made the rule: interviewers should give 3 times more time to solve a problem than the time in which they themselves can solve the problem. Usually this corresponds to a more or less correct assessment.

Longer tasks

Simpler questions are useful for another important reason. They allow you to add more content to the task, that is, you can use longer, composite tasks, which has additional advantages in terms of forecasting. You can ask questions that become more complicated over time - and such longer, real questions are more predictable than short and hard ones . It is logical that the easiest way to predict the performance of the candidate, if you offer him to work on a task that is similar to what he will do at work .

This is partly due to the fact that longer questions are closer to real programming. Real programming involves working with a relatively large code base over a long period of time, and longer questions are simply closer to this reality.

In addition, longer questions allow you to give hints when a candidate is stuck. I think this is vital, because even strong engineers can stumble at some point in the decision. Longer tasks give a chance to recover from mistakes and later demonstrate their skills. One wrong step should not spoil the whole interview . Finally, the ability to help relieve stress on the interview, which again leads to more accurate results.

Conclusion

By challenging candidates, the company falls into a vicious circle . Questions initially too difficult and short, which leads to sub-optimal decisions about hiring. These decisions are based on noisy signals. When a company sees poor results, it tries to rectify the situation, making the selection process even more stringent, which usually means even more complex issues . But this only reduces the accuracy of interviews, and so on. Both companies and candidates suffer from such a vicious circle. I see that companies are missing out on talent, and candidates are missing out on jobs — and they are straining more than they need.

I hope the interviewers will draw conclusions. They can not only improve the accuracy of selection, but also simplify their work. It is much easier to come up with simple, multi-step problems than perversely complex short questions.

So here is our advice. If you really want to make interviews more accurate, then you need to start asking simpler programming tasks. This does not mean lowering the bar. It simply means improving the signal to more effectively weed out candidates and hire the most talented.