What I would copy from the Pivotal interview process

Hey there! I'm Nat Bennett, you're reading Simpler Machines, and while this newsletter isn't always specifically about how Pivotal made software that's what it's about again this week.

So last week I wrote up a description of how the Pivotal interview process worked for engineers. This week I want to get into a little bit more detail about what was important about that process and what I think most companies could profitably borrow from it.

Figure out who does unusually well at your company and design a process to find those people

One of the things the Pivotal interview process was really good at was taking a pool of Java engineers with unfashionable resumes and filtering for the folks who were really good at – or could become good at – test-driven development and pairing. This gave Pivotal a big advantage over a lot of similarly-sized companies because it was using a different (and better) set of signals than many similar companies. It didn't have to rely on weak signals like school and former employer – or compete on price with the many companies that were using those signals.

Interview separately, decide together

You're probably going to have multiple people interview each of your candidates. You should have each of those people interview the candidate and fill out a scorecard independently, but then, schedule a synchronous discussion with the hiring manager and the other interviewers to make the hire-or-not decision.

If you can swing it, I actually think its even better to invite everyone who interviewed that week to this meeting, and let interviews who are interested listen in on the discussion about candidates they didn't interview.

There are probably other ways to get similar benefits but this was a pretty good way of onboarding new interviews. You want to have folks fill out the scorecard separately so if people have wildly different first impressions of a candidate they don't immediately collapse into groupthink, but figuring out the why behind those disagreements and coming to a consensus decisions helps new interviewers onboard and understand the criteria more experienced interviewers are using to evaluate candidates.

(If your existing interviewers are bad you will then have the problem of new interviewers copying their bad methods – but then you've probably got bigger problems.)

Interview with realistic sample work

This is something that's become more common over the past decade – you're much less likely these days to encounter a leetcode kind of problem than you were 5 or 10 years ago, I think. An interview works better on both sides if you use a problem that's either directly pulled from your company's work or is very similar to a real one you had to solve, instead of a generic problem. It gives interviewees a clearer picture if they'll enjoy working with you, and it gives you a more accurate assessment of their actual work-related skills.

Pair hiring managers and recruiters

In other large companies I've worked with the pipeline of potential hires was sourced by a centralized recruiting team that had either no contact or very minimal contact with the managers they were supporting. I don't think this works nearly as well as putting recruiters in close, regular contact with the specific hiring managers they're recruiting for.

Only look at resumes at the beginning of the process

Engineers who are conducting technical screens or post-screen interviews shouldn't look at resumes. Someone else has already looked at the resume and decided the person it represents is worth continuing to interview. All looking at the resume will do past that is introduce irrelevant information that will distort your decision as an interviewer – it'll bias you in favor of hiring the people who everyone else is hiring.

Treat interviewing as the most important thing an engineer can work on

Management, and the senior engineers that everyone else looks up to, should treat interviewing as their most important priority. Engineers who don't want to interview shouldn't have to – but it should be at least a little weird. Engineers who are in the interviewing pool should reschedule basically everything except customer meetings in order to take interviews.

I don't know exactly how to operationalize this if you don't already have a culture like this, but I do know for sure: if it's hard for recruiters and to get interviews scheduled you are going to have very bad problems very soon. You are going to get outcompeted for the candidates you want most by companies that are able to move faster.

Use a standard technical screen

I've interviewed at places that let engineers kind of come up with their own exercise for the technical screen and I don't think this works very well. It's not even so much that individual engineers are bad at designing technical screens – though I think they are – but that if everyone's using their own screen you're not benefiting from improvement from everyone in the group that's using it. So treat it like a shared codebase and incrementally iterate on it.

Having a standardized screen also gets you a subtle – but really important – morale effect, which is that everyone knows that every engineer that's there "passed" a particular test. There's less "well do they really deserve to be here? or did they just get lucky with the interview?" when everyone basically knows the shape of the technical screen because they've taken the same one themselves.

You might not need a technical screen if you're not hiring at scale

That said, you probably don't even need a technical screen until you've got a large enough candidate pipeline that your "regular" interview would be taking up all your interviewing engineer's time. The technical screen is an optimization and it has costs – it makes the process longer and harder to co-ordinate. Pivotal used it extensively and pretty early, because Pivotal had developed out of a consulting company, which is a kind of business that's likely to hit the "scaled hiring" point faster than product companies.

Remember that interviewing is a sales process

If you talk to Pivots about why they decided to join Pivotal often they'll explicitly mention the interview process and how much they enjoyed it. If you've interviewed as a software engineer you know that this is very weird – interviewing is usually miserable!

And its miserable in large part because companies tend to treat it entirely as an evaluation process, and forget that it's a sales process. You're selling the candidate on the idea that they will have a good time working at your company. And "how it feels to be interviewed" is going to utterly dominate any other information they have at the offer stage. If that experience is bad it's going to be much harder to get candidates to say "yes."

Tell interview candidates what you're evaluating them for

Want a quick way to make your interview process more pleasant for your candidates? Give them the rubric up front. Tell them what you're looking for and why.

(Do you know what you're looking for and why? If not– go back to the top of this article.)

This will relax a lot of candidates and stop them from making terrible guesses about what you're looking for.

Make sure the candidate learns something during your interview

The biggest reason candidates liked the interview process was the unusually realistic picture it gave them of what it would be like to work at Pivotal. The second biggest reason was that they almost always learned something – due to the nature of pair-programming on a "real" problem. Unless you're doing something you've done a bunch of times with someone you've worked with before, you're almost guaranteed to learn something interesting in 3-4 hours of pairing.

Software engineers tend to be people who like learning. So if they learn something during the interview, they're going to think, "Wow, I'm going to learn things all the time here." This is a powerful incentive to be able to offer convincingly.

Next Time

I have a few more parts of Pivotal I want to do deep dives on (allocations, ask, the Cloud Foundry release process, maybe IPM?) but I've been on this topic for a minute now and have not had a ton of time to write during the week, so you might get something a bit "lighter" next week.

As always, thanks for reading and sharing.

- Nat