Your recruiter should help you say "No":  The case for case studies

charles-barsotti-what-the-hell-we-could-use-an-idiot-new-yorker-cartoon_a-l-9184509-8419447.jpg

I was talking with a friend who runs an executive search firm recently. He told me a story about another recruiter he used to work with.  This recruiter hated the idea of testing candidates – in his mind, “testing just gives the client another reason to say no.”  I was taken aback, as was my friend.  That recruiter no longer works at his firm.  As a partner to companies looking to hire leaders, the job of executive search is to find the best, not close deals. 

A good search involves several key components.  At IAR, we believe the case study process used with all our clients to be among the most important.  This comes from years of practical experience.  Before starting IAR, I implemented a case study process at every company where I ran recruiting.  Everyone, from CEOs to interns, did a case study before they got an offer.  I want to share this process with you because it has proven to be a wonderful tool to assess candidates.  Try it out and let us know what you think.  We are here to help if you have any questions.

What is a case study? 

A case is any test that allows the hiring team to evaluate real world performance of a candidate on the practical skills needed for the job.  You may have seen this used on software engineering teams.  A hiring manager will give out a coding test before the first interview to make sure a candidate’s skills match the needs of the role.  Our process is essentially the same but scaled up for executives.  For individuals we recruit (C-level executives and their direct reports), the case study is more strategic in nature and used as a selection tool in the final stage as opposed to a filtering tool at the early stages.   

Why should I do it? 

The first reason is because it works.  I have several anecdotal stories that illustrate this but want to focus on the data.  While researching for this post I came across a meta-analysis done by Smith and Robertson.  A summary of their findings is below.  As you review the chart, keep in mind:  A validity of 1 means a perfect fit between the criteria and what it is predicting; 0 shows no relationship; 0.4 and above is respectable (thanks to the folks at Opp in the UK for helping translate this).  

Graph blog 3.png

The analysis shows that work sample tests (or as we call them, case studies), give the best predictor of performance.  They yield better results than personality questionnaires, references – even years of experience.

The second “why” is a more obvious - a good case makes the candidate put “skin in the game”  driving up engagement.  It doesn’t take much to say yes to an interview.  It takes real effort  to spend personal time solving a business problem.  Candidates won't do it unless they are serious.  At the same time, candidates who go through the process are more likely to say "yes" when presented with a fair offer.  They are invested, have a clear idea of what their job will entail, know the challenges, and have already thought through solutions. 

How do I develop a case study?

I am going to skip the initial phases of recruiting and get to the point.  Let’s assume IAR is recruiting for a Chief Human Resource Officer for a growth stage SaaS company.  We have vetted candidates, gone through several rounds of interviews and have identified our top three.   This is where the case comes in.  At the kick-off of every search we establish the core competencies that a candidate will need to be successful.  For a CHRO this may mean (1) being an effective coach to leadership, (2) hiring and retaining the best and (3) ensuring that the culture of the organization scales as the company grows.  Using this criteria as a guide, we will build a custom set of real world challenges that the candidate will likely face in their role (e.g. What type of system would you set up to scale recruiting?  What type of metrics would you use to measure the success of your team?).  As part of the case, we ask the candidate to develop a plan of attack using their prior experience and knowledge they have gained during the interview process – generally this plan covers the first 90 to 180 days.

Case studies can take many forms.  We suggest keeping it simple - a written prompt with relevant information will suffice.  For the process to be effective there are several things that must be done:

  • Reserve the case for your top two or three:  Your interview process should do the filtering.  The case study is meant for the very best candidates to see how they will act in the real world.     
  • Communicate the process early:  Preparing a case means a significant investment of time and energy.  If candidates make it past the first-round interview let them know a case study will be part of the final evaluation process.  Prepare them for the time commitment in advance. 
  • Ensure your questions reflect the needs of the role:  If you are not specific about the real-world challenges the candidate will face the less validity the test will have.  You will be wasting your, and the candidate’s, time.    
  • Make it transparent:  Let the candidate know how they will be judged.  If you have multiple questions let them know your priorities.  Communicate the process early - prepare them for the time commitment in advance (ideally once they pass the first round interview).  Finally, make it open book -  ensure the candidate knows that they can ask questions as they work through the problem.  Encourage them to reach out.  If the candidate gets hired they will have access to you and the team to ask questions – the test should be no different.
  • Let the candidate get creative:  Give candidates a minimum of 5 business days to come up with a solution.  The goal is to see their planning and execution in action, you can’t get that by giving them an hour to work on a problem in the office.  Also, leave the format up to them.  Visio, Word, PowerPoint, whiteboard, it really doesn’t matter as long as the delivery is coherent.  Limiting your candidates will place needless restrictions on people that are (hopefully) creative problem solvers. 
  • Time box candidate workload:  Let candidates know what you expect in terms of time commitment.  At the executive level, we ask candidates to spend no more than eight hours total working on the problem we present to them.  This serves as a good gut check as you are coming up with the question(s).  If the problem feels like it will take longer than eight hours to address, narrow the scope.
  • 50% presentation, 50% Q&A:  We recommend an hour total for the case presentation.  30 minutes for presentation and 30 minutes for Q&A.  Make sure the candidate knows they don't have an hour to fill.  For internal purposes, budget two hours total in case things go a little over and to ensure that you have time for you to have an…
  • Internal discussion immediately after:  This is key.  After the candidate has left, ensure that the selection committee spends time discussing the evaluation criteria and the candidate’s performance.  If you are the hiring manager, let the team go first to ensure you are getting unbiased opinions.
  • Follow up ASAP:  You owe timely feedback to candidates who have committed the time to a case.  This call should happen in no more than 48 hours after the case.  If you don't have a final answer for the candidate let them know the specific timeline for a decision.  
  • Do it – every time:  The case only works when you have all your final round candidates complete one.  An apples to apples comparison is key.
  • Have a pipeline of additional candidates ready:  Don’t pop the champagne and call it a day once you have locked in a case study schedule.  If done correctly the case will reveal candidate’s strengths and weaknesses.  You may come out of the case finding that your top candidate isn't a fit.  The last thing you want to do is start over completely.  The best way to avoid this is to have a group of “warm” candidates ready to introduce to the team without a significant delay.

We would love to hear your thoughts below!  What has worked for you?  What questions do you have for us?