Interviewing and Evaluating Candidates
Move beyond resumes to assess high-fidelity signals like agency, first-principles thinking, and actual craft.
The Guide
5 key steps synthesized from 14 experts.
Standardize the Evaluation Rubric
Establish a universal rubric that evaluates both technical skills and soft skills consistently across all interview loops. Group your questions by specific themes like agency, first-principles thinking, and resilience to ensure all essential competencies are explored. This structure prevents talent silos and allows for fair comparisons between different candidates by reducing reliance on gut feelings.
Featured guest perspectives
"I think this is something I saw when we would have some pockets of analytics folks embedded is having a consistent bar for talent in terms of what we're looking for, what are the technical skills, what are the soft skills? And being able to evaluate candidates with that same bar, using our same rubric."— Jess Lachs
Implement Realistic Work Trials
Replace abstract case studies with almost-real-life assignments based on historical challenges your team has actually faced. Design a one to five day paid work trial where final-stage candidates work on real projects to evaluate their actual judgment, taste, and collaboration style. Ground the project prompt in a situation your team has already solved so you can better calibrate the candidate's output against known variables.
Featured guest perspectives
"Job candidates go through a paid work trial. Theyjoin the team for 1-5 days and work on a real project with the team."— Lenny Rachitsky
"The project was typically from a real-world situation we faced and solved in the past. This allowed us to know the ins and outs of the problem, which is useful when assessing and guiding candidates."— Lenny Rachitsky
Test for High Agency and Clock Speed
In rapidly evolving environments, prioritize candidates who show a history of solving problems without waiting for permission. Look for indicators of clock speed and energy by asking for specific examples of past initiatives where the candidate defined what would not exist without them. Screen for the ability to move past luck when explaining success to reveal the candidate's level of self-awareness and reflection.
Featured guest perspectives
"Finding people who are high agency and work with urgency, if I was hiring five people today, those are some of the top two characteristics that I would look for in people because you can take on the world if you have people who have high agency and not needing to get 50 people's different consensus."— Logan Kilpatrick
"I saw some of the highest performers just being people that had very high agency, had that clock speed, had that energy, but they didn't necessarily need to have deep experience on that matter."— Albert Cheng
Test Humility with Direct Feedback
During the interview, provide blunt or even jarring feedback on a specific response to see how the candidate reacts. A high-signal candidate will respond with curiosity and a desire to improve rather than defensiveness or ego. Additionally, ask candidates to describe their biggest product flops to evaluate if they possess the humility to own their mistakes and the resilience to handle the fallout.
Featured guest perspectives
"It's just like when you're interviewing someone, giving them tough feedback after the interview if there's an area of concern, but you otherwise like them, can be a really great way to see how they would interact with you personally as well as how they take feedback. And so that was my learning from the situation, is if you really enjoy an interview, except for maybe something on the culture side, to give the direct feedback and see how people engage."— Keith Yandell
"You can learn the most about how a person operates, thinks, and collaborates by exploring times when things didn’t go as planned. If they get hired, you can guarantee they’ll face unexpected challenges, so you’ll want to know how they’ll handle these moments before they have to tackle them on your team."— Lenny Rachitsky
"It tells me you have some humor, you’re humble, and you can point out when you’ve made a mistake. You’ve done enough to be able to confidently say, of course I’ve made a mistake. Because none of us are perfect. And you know how to spot those mistakes and you can learn from them."— Lenny Rachitsky
Conduct Rigorous Reference Stacking
Treat reference checks as your primary source of truth rather than a final check-the-box step. Seek out blind references who have worked with the candidate for multiple years and ask pointed questions about whether they were in the top 1 percent of performers. Use triangulation by asking the candidate to predict exactly what their references will say about their specific superpowers and weaknesses.
Featured guest perspectives
"I think CEOs and everyone dramatically overrates their ability to interview, and overrates their gut feeling, and underrates a really high quality blind reference."— Brian Halligan
"I generally value the reference check over interview signals. If I had to stack rank in interviews, what is the best signal? The reference check is the top of the list. Those people, they worked with this person sometimes for years, their knowledge, what you're going to get out of 30 minutes of artificial scenarios it's just like never going to compare what a good reference check will give you."— Shishir Mehrotra
Get this guide as an AI skill for Claude Code
Install This Skill
Add this skill to Claude Code, Cursor, or any AI coding assistant that supports Agent Skills.
Quick Install (Recommended)
Install this skill directly using npx:
npx skills add RefoundAI/lenny-skills --skill interviewing-evaluating-candidates Or install all 76 skills:
npx skills add RefoundAI/lenny-skills Manual Installation
Download the skill
Download Skill (.zip)Add to your project
Create a folder in your project root and add the skill file:
.claude/skills/interviewing-evaluating-candidates/SKILL.md Start using it
Claude will automatically detect and use the skill when relevant. You can also invoke it directly:
Help me with interviewing and evaluating candidates Or receive these install instructions in your inbox:
Guest Perspectives
Deep dive into what 13 podcast guests shared about interviewing and evaluating candidates.
Albert Cheng
"I saw some of the highest performers just being people that had very high agency, had that clock speed, had that energy, but they didn't necessarily need to have deep experience on that matter."
- Prioritize hiring candidates with high agency who can solve problems without relying on established playbooks.
- Screen for 'clock speed' and energy to ensure team members can keep pace with shifting market conditions.
- Look for talent that is willing to intentionally discard outdated habits in favor of learning new, AI-accelerated workflows.
Brendan Foody
"How can they leverage this technology to do so much more? We'll give people interviews where we say, 'Use whatever tools are available to build a website and let's see what product you're able to build in an hour.'"
- Allow candidates to use any available AI tools during the technical interview process.
- Set time-constrained tasks that require high output through tool leverage.
- Evaluate the quality of the final product and the candidate's speed of execution.
Brian Halligan
"I think CEOs and everyone dramatically overrates their ability to interview, and overrates their gut feeling, and underrates a really high quality blind reference."
- Conduct blind references and ask if the candidate was in the top 1% of employees or if the reference would enthusiastically rehire them.
- Have candidates review a real board deck under NDA to see if they provide meaningful challenges rather than just being a 'yes person.'
- Prioritize 'spiky' candidates with significant strengths over 'well-rounded' candidates who have fewer weaknesses but less impact.
Geoff Charles
"It's also a great way to have positive selection in terms of talent because talent wants to join companies that ship fast. And a lot of people who join Ramp, I ask them, 'Why are you interested in joining the company?' And they often say, 'Well, it's because you guys are actually building things and shipping things and I want to know what that feels like.'"
- Use the company's shipping velocity as a primary recruiting value proposition.
- Identify candidates who prioritize the ability to build and ship over corporate status.
- Ask candidates directly why they are drawn to high-velocity environments.
Interview Q Compilation
"Ambiguity is a big one for me, because at the end of the day, the PM job is really ambiguous. It's really hard to describe on a piece of paper all the things that you're going to encounter. Good answers are people who put structure and a way forward through the ambiguity."
- Ask behavioral questions focused on how candidates have navigated ambiguous situations in the past.
- Look for candidates who proactively seek help and external inputs rather than claiming to have all the answers.
- Evaluate whether the candidate can create a structured path forward and define milestones for success.
""Describe to me a time when you were part of a controversial product decision. What did you do?" All those things. I think it's really revealing, because if they can set up this conflict and understand why this problem was really important and represent both sides and such that you can understand why that conflict existed in the first place, and they can do it in this kind of even-keeled way where you realize that they can take on these different perspectives, you start to learn a lot about that person, I think."
- Ask candidates to describe a time they were involved in a controversial product decision.
- Require the candidate to represent both sides of the conflict to see if they can understand the existential importance of the problem.
- Observe if they provide an "even-keeled" response that demonstrates their ability to take on different perspectives.
Jess Lachs
"I think this is something I saw when we would have some pockets of analytics folks embedded is having a consistent bar for talent in terms of what we're looking for, what are the technical skills, what are the soft skills? And being able to evaluate candidates with that same bar, using our same rubric."
- Implement a universal rubric to evaluate technical and soft skills consistently across all interview loops.
- Evaluate candidates on their ability to act as thought partners, not just technical service providers.
- Maintain a high bar for hiring centrally to ensure consistent data quality and methodology across the company.
Keith Yandell
"It's just like when you're interviewing someone, giving them tough feedback after the interview if there's an area of concern, but you otherwise like them, can be a really great way to see how they would interact with you personally as well as how they take feedback. And so that was my learning from the situation, is if you really enjoy an interview, except for maybe something on the culture side, to give the direct feedback and see how people engage."
- Provide blunt feedback on a specific cultural concern to see how the candidate responds.
- Look for curiosity and a desire to improve rather than defensiveness or ego.
- Use informal settings like dinner to probe personality traits that felt 'off' during formal interviews.
Kevin Yien
"When you get to offer stage, I send an email and I say all the terrible things that are probably going to reinforce their fears. If you can tell them that upfront and they can read that whole email and still be equally excited to join you, find yourself a A+ hire."
- Send an 'unsell email' at the offer stage detailing the company's biggest challenges and flaws.
- Directly address the specific fears a candidate might have about the role or culture.
- Prioritize candidates who maintain high excitement after learning the negative aspects of the position.
Logan Kilpatrick
"Finding people who are high agency and work with urgency, if I was hiring five people today, those are some of the top two characteristics that I would look for in people because you can take on the world if you have people who have high agency and not needing to get 50 people's different consensus."
- Screen candidates for high agency by looking for histories of solving problems without waiting for permission.
- Prioritize urgency as a core hiring trait to ensure the team can outpace competitors and navigate crises.
- Identify individuals who prioritize solving customer challenges over seeking broad internal consensus.
Melissa Tan
"I think they looked for two main things. They looked for first principles thinkers, so not necessarily your experience, but how do you approach problems, how do you know the right questions to ask? And then create your own framework around that. Dropbox also hired for people that were just really humble, collaborative and team oriented."
- Assess for first-principles thinking by evaluating how candidates create their own frameworks rather than relying on experience.
- Look for humility and a team-oriented mindset to ensure the candidate can innovate across functions.
- Train the team to sell the company and specific roles during the interview process to close top talent.
Patrick Campbell
"And so that was the other thing is we would pull all of this forward into the interview process and talk about it and basically say like, 'Hey, if this is not how you think, that's okay. We're not better than you, you're not better than us, it's just this is how we do things here.'"
- Present a specific behavioral scenario, like handling a conflict, during the final interview.
- Be upfront about the difficult aspects of your culture to test for alignment.
- Filter for candidates who demonstrate judgment consistent with your established company values.
Peter Deng
"In 6 months, if I'm telling you what to do, I've hired the wrong person. It helps me and the person operate on a different level where the goal is not, did you hit this OKR? The Meta goal becomes, are we calibrating enough? Are we actually getting into a spot where in 6 months you're the one telling me what needs to be done?"
- Test for the ability to take full ownership of a domain within six months.
- Focus management on calibration and alignment rather than just OKR tracking.
- Hire experts who operate at a level where they define the work rather than receiving it.
Shishir Mehrotra
"I generally value the reference check over interview signals. If I had to stack rank in interviews, what is the best signal? The reference check is the top of the list. Those people, they worked with this person sometimes for years, their knowledge, what you're going to get out of 30 minutes of artificial scenarios it's just like never going to compare what a good reference check will give you."
- Stack rank reference checks as your #1 signal when evaluating candidates.
- Seek out references who have worked with the candidate for multiple years to gain deep historical context.
- Value real-world work history over performance in artificial, 30-minute interview scenarios.
Make your product team AI-native
These skills work best as a system. Get a free 8-email course: map how your team works today, wire agents into real workflows, and make it stick. From the team that does this for a living.
Related Skills
Other Team & Organization skills you might find useful.
Organizational Design for Product and Engineering
To prevent internal resentment and organizational rejection, innovation teams must be positioned as...
View Skill → →Coaching and Talent Development
Senior product management impact is defined by the ability to provide strategic clarity and prioriti...
View Skill → →Building High-Performing Team Culture
High-performing cultures require leaders to prioritize clear performance standards and organizationa...
View Skill → →Leading Organizational Change
Organizational innovation is most effective when leaders act as 'gardeners' who nurture emergent, se...
View Skill → →