An Interview Process That Works For Me

“What is your interview process?” is a question I have answered many times. I have conducted technical interviews for over twenty years and I am proud of the process I have refined in the past decade. It has been extremely effective for hiring high-quality software engineers that are a fit for the nature of the work and the culture of the team. It is not perfect—no interview process is—but it treats candidates with respect, it is enjoyable, and it gives candidates a lot of insight into the team. I will provide a description of the process, along with my motivations and takeaways for each step in the process. I hope this will be a useful guide for people interviewing on my team. If you are an employer, I hope it provides a positive influence on your technical interview process.^[1]

Screening

When screening a candidate, I want to see three things:

1. An example of their work, usually a resume.
2. A compelling narrative for why they want to work on this team.
3. A summary of their career highlights, focused on the value and impact.

First, the quality of the resume remains the single best indicator of the candidate’s work.^[2] If the resume communicates poorly, or lacks refinement and attention to detail, why would you expect code, tickets, documentation, pull requests, presentations, or emails to be any different? Aline Lerner found that typos, grammatical errors, and formatting matter more than anything else:

The most significant feature by far was the presence of typos, grammatical errors, or syntactic inconsistencies.
—Aline Lerner

Asking for a resume may seem a bit old fashioned, but evaluating LinkedIn or Github is rarely as effective.^[3] A personal website or technical presentation are great examples of the candidate’s work, but very few candidates share these.

Second, I want to know why the candidate wants to work on this team. The fact that they are curious about technology or want to keep learning is important, but not sufficient. I want to hire people who resonate with the purpose and value of our work, people who can innovate and improve the product, people who want to work with these teammates, in this culture, solving these specific problems.

Finally, I want a concise summary of the candidate’s career accomplishments. The best candidates can summarize why their work was important, why it was challenging, and how they contributed. They communicate purpose and mastery. They communicate this on their resume. Returning to Aline Lerner’s study:

You get a very clear idea of what the product is, what the candidate’s contribution was in the context of the product, and why that contribution matters.
—Aline Lerner

These screening steps can be conducted by a recruiting team without a deep understanding of engineering. It is an efficient process yielding high-quality candidates.

Introductory Conversation

Before investing in an interview, I have a brief, informal, exploratory conversation with the candidate over video conference. I highlight the work and the culture of the team, understand what the candidate has been doing, and what they want to do next.

Many hiring managers are hesitant to have a conversation, viewing it as a large investment.^[4] They prefer to screen candidates by giving them a programming assignment. Personally, I would never apply to a job where the first step was to complete a programming assignment before speaking to the hiring manager or an engineer. As you will see below, I am more interested in the candidate’s motivations and the thinking behind their work, which will not be revealed by a trivial programming assignment. Furthermore, by the time you review an assignment in any level of detail, you have already invested the same amount of time as a brief introductory conversation—and having a conversation is more personal and can reveal much more information.^[5]

Another reason to have an introductory conversation is that you will not understand the candidate from the screening alone. For example, you may look at the candidate’s resume and conclude that they are an excellent systems programmer, but you are looking for someone to focus on operations and cybersecurity. However, if you actually talk to the candidate, you might hear something like, “I have really enjoyed my contributions as a systems programmer, but now I want to focus less on programming, and evolve my career to concentrate on operations and cybersecurity.” Don’t make assumptions.

Another way to look at it is: The screening is about quality, the introductory conversation is about fit.

Follow-Up Email

After a promising introductory conversation, I send the candidate a follow-up email. I have the text saved as a template so I can send it instantly, without having to compose an email.^[6] It has a link to a conference presentation that motivates our work,^[7] information about the products we support, links to articles in the press demonstrating the impact of our work, and some references to my personal writing on this blog.

The most engaged candidates will respond to this email with questions or comments. Engaging with the candidate over email provides me yet another opportunity to assess communication skills, as well as interest and aptitude for the role. Sharing articles from my blog, both technical and non-technical, gives the candidate great insight into me and the team. They are personal essays and they generally do not mention my employers, but if you read between the lines, they are reasonably transparent in terms of my approach to engineering and management, and the resulting culture of the team.^[8]

Programming Assignment

By this point, I assume everyone is a capable programmer—their experience, skill, and focus will vary, but everyone is capable.^[9] Therefore, I’m not interested in assessing how well someone rehearsed or memorized questions focused on algorithms or data structures, so common in technical interviews, and largely irrelevant to our day-to-day work.^[10] I’m also not going to time them, supervise them so they don’t cheat, or force them to use an online tool like this is an academic exam. Engineers who have been working professionally, especially ones with strong professional networks, will simply not submit themselves to this type of interview process.

I want to treat people professionally. I want the candidate to be comfortable, to use the tools they are accustomed to, and to have the assignment be a reflection of the quality of their professional work. Consequently, the assignment is practical and modelled on the types of distributed systems and IoT problems we work on every day. Almost every candidate finds the assignment interesting and enjoyable.^[11] If they don’t, this team is not the right fit. The assignment itself is not difficult, but it does involve many trade-offs that demonstrate expertise and engineering judgement.

If you only evaluate candidates with live coding or pair-programming during an interview, you are creating a bias for fast, automatic, instinctive, and emotional thinking. A take-home assignment gives the candidate some time with the problem to demonstrate their slow, effortful, deliberative, and logical thinking, much more like our daily work. It is important to assess both, but if you bias for fast thinking, you are biasing towards the candidates who have seen the problem before, and failing to assess how they approach novel and professional problem solving.^[12]

Programming Assignment Review

The goal of the assignment review is to appreciate the thinking behind the solution. This is the most critical part of the interview and it provides the most reliable evaluation of the candidate. Many interviewers evaluate candidates by looking at the quality of the code or by objectively testing it against a set of sample inputs. While this is an important part of my evaluation, it doesn’t tell me how the person thinks in systems, evaluates trade-offs, or navigates challenges that have no correct answer. I want to assess engineering skills and this is best done through a discussion. In addition, I asked the candidate to spend their time completing the assignment, I owe them a discussion. As I wrote in Interviews for Programmers Should Involve Code Review:^[13]

A programming test in an interview places too much emphasis on the employer. The candidate carefully considers and completes the assignment. The employer delivers judgement. The programming assignment can be a valuable part of the interview, but if you require it, review the assignment with the candidate. It says: I value your time, your ideas, and your work.

I start the assignment review by asking the candidate if they enjoyed the assignment. Then I ask questions about their approach, as one would in a code or design review. I may disagree with their approach, or it may have problems, or their approach may be unique and unfamiliar to me. It is important to keep an open mind because the candidate may be coming from a different perspective and know something you don’t.^[14]

The majority of the time is spent changing the requirements of the assignment and talking through how to adapt the solution to meet these requirements.^[15] This is what happens with most systems: the first version meets the initial requirements and then people ask for additional features, or the system must adapt to meet growing demand, resulting in changes for scalability, reliability, performance, cost, compliance, or security. This should be an interesting and enjoyable discussion focused on programming, distributed systems, and IoT. There are no “right” answers, only trade-offs and engineering judgement.^[16] To ensure we incorporate more than one perspective, I usually complete the first half of the assignment review and someone else completes the second half.

Panel Selection

By this point, I have seen some of the candidate’s work, we have interacted over video conference, and corresponded over email.^[17] I have a good understanding of how the person communicates, what motivates them, and how they will integrate and contribute to the team. The next step is to design the interview panel.

I design the panel to provide a diverse set of perspectives, a rigorous assessment, answer questions, collect concrete examples, and support the candidate through the process. The panel usually includes five people, and I try to balance the panel to include years of experience, men and women, areas of expertise, among other perspectives.^[18] A diverse panel helps us make good decisions and it helps the candidate experience a cross-section of the team to understand the work and the culture. To ensure we keep a high bar, I often select one panellist from outside our team. Input from a trusted leader outside the team can be particularly helpful when hiring senior candidates or candidates in unique roles. For some candidates, it is clear we want to hire them, and the panel is more about confirming this opinion and convincing the candidate to take the role.

I always want one junior engineer on the panel, someone in their first few years in the industry. For a junior candidate, like a new graduate, this interviewer is the most relateable. For a more senior candidate, I want to see how they treat a junior engineer. Can they listen and take direction from someone less experienced? Are they patient and understanding? Can they modulate their communication for someone with less expertise? Would they be a good mentor? Similarly, I always want more than one senior engineer on the panel—leaders on the team who have years of experience interviewing and have seen people successfully on-board, contribute, and develop their careers. When interviewing a more senior candidate, it is critical that the panel include at least two or three senior engineers who are willing to champion the candidate.

Before the interview, the panel meets to prepare. I review the position we are hiring for, set expectations, and review the feedback collected so far from the recruiter, the introductory conversation, and the programming assignment review. We discuss open questions. The panel needs to cover a wide range of topics, including design, architecture, programming, testing, operations, troubleshooting, and team culture, in addition to the candidate’s motivations. To achieve some balance, we assign topics so that the panellists do not all focus on the same thing. At the same time, we try to gather more than one opinion on each topic.

Most organizations have training that encourages diversity, equity, and inclusion in the hiring process by applying exactly the same process for each candidate—ask exactly the same questions, in exactly the same order, and expect exactly the same responses. I understand the perceived equity from this approach. However, it is paradoxical to apply a prescriptive process to achieve diversity. I’m willing to be more open minded and design the panel to accommodate the candidate’s unique needs and talents.

Presentation

The interview begins with a presentation by the candidate. Some candidates are understandably nervous about the presentation, but most use it to differentiate themselves by highlighting their work, their engagement, and their ability to communicate. The presentation is scheduled for thirty minutes—twenty minutes for the presentation and ten minutes for questions. The presentation is attended by the five people on the interview panel, anyone else who has been involved in screening the candidate, and sometimes a Senior Manager, a Director, or a Product Manager.

The presentation must be technical and demonstrate the candidate’s ability to communicate outside of code, tests, pull requests, and so on. The candidate should target the presentation at peers, assuming they are intelligent and technical, but perhaps not familiar with the domain, as well as people who are less technical, but appreciate clear communication about the objectives, value, and outcomes of the work.

My most succinct prompt for the presentation is:

Describe the most challenging problem you have solved. Why was it challenging? Why was it important? What exactly did you do to solve it?

Interview Panel

After the presentation, the candidate has an individual 45-minute interview with each person on the interview panel. The interviewer will ask questions about design, architecture, programming, testing, operations, and troubleshooting, as well as team and organizational culture. Similar to the programming assignment, the interview questions are practical and reflect the nature of our work. For almost every topic of conversation, there is no “right” or “best” answer. I want to see candidates recognize and navigate trade-offs with good engineering judgement. I want to see the candidate think clearly, state and test assumptions, and work through problems from first-principles.

Software teams want to hire people with aptitude, not a particular skill set.
— Joel Spolsky

One or two of the interviews will involve programming, but they will avoid academic or trick problems. It is not very representative of our work to present a problem saying: “Here, code it!” I like to refactor code because it provides a context for the candidate to work from, it is an activity that comes up daily in our work, and it demonstrates the candidate’s ability to think in systems, evaluate trade-offs, and use engineering judgement. For the architecture and design questions, we often ask questions about challenges we are currently facing.

I avoid generic behavioural interview questions. We are not psychologists and we are not very good at interpreting the responses or understanding how our biases or behaviours are influencing the responses. All I can think of is the classic “Can you describe your weaknesses?” followed by “Well, I’m a bit of a perfectionist.”^[19] I prefer to approach behavioural subjects more indirectly and in context. For example, “Tell me about a time your work was blocked by a bug in another team’s code base? How did you approach the situation?”

When the interview is complete, the interviewer submits written feedback highlighting concrete examples along with a hiring decision. To avoid bias, the interviewer should write this feedback as soon as possible and without talking to anyone else.^[20]

Round Table

After the panel interviews are complete, we have a round-table discussion. Each person on the panel quickly summarizes their written feedback and provides a hiring recommendation. Others who have been involved in screening may also comment in relation to the panel feedback and provide their hiring recommendation. Then we discuss questions and concerns and make a hiring decision. Even if the panel is unanimously positive or negative and you know the conclusion, it is important to take the time to hear from each person—it ensures everyone’s input is considered and the discussions will refine the interview process.

Occasionally, the panel will raise a concern it is unable to resolve. For example, they may conclude the candidate is an excellent engineer, but failed to sufficiently evaluate their ability to also assume management responsibilities. When this happens, we will do one more follow-up interview selecting an interviewer to directly answer the open questions. In other cases, when we want help championing a candidate, we will have a follow-up interview with a Senior Manager or Director. This is useful when we are establishing a new role, hiring someone from a non-traditional background, or hiring an exceptional candidate.^[21]

Executive Write-Up

The final step is executive approval. We prepare a detailed write-up—written in prose, not bullet points—that summarizes the interview and highlights why we want to hire the candidate. The write-up includes a role justification; the candidate’s work history; two or three career highlights, a paragraph for each; and a summary of the feedback from the interview panel, including concrete examples from the programming assignment, presentation, or panel interviews where the candidate demonstrated excellence. We ask for the candidate’s help in drafting the career highlights because we want them to be accurate and we find they are most compelling when written in the first person.^[22]

There is a lot of scrutiny paid to the write-up, it is not just a formality. Drafting the write-up takes a significant amount of work, but the process brings a discipline that selects for excellent candidates. Executives often have feedback that leads to healthy discussions and a better understanding of the needs and objectives of the team. The writing process is also a good test and affords a lot of reflection. If we can’t draft a compelling write-up after this rigorous interview process, it makes us question our conclusions. There have been a few examples where the panel is largely positive, but when it comes to drafting the write-up, we realize we cannot make a strong argument.^[23]

After executive approval, we extend an offer to the candidate and conclude our interview process. If all goes well, the candidate accepts our offer and we have a great new teammate.^[24]

Conclusion

This interview process is lengthy and involved.^[25] It selects for people who are mission-driven, and want to work on these problems, in this team culture. It selects against people who are casually interested.^[26] It attempts to evaluate all aspects of the role, especially engineering judgement, not just programming skills. It also treats candidates with respect and tries to give them ample insight into the nature of the team and the work.