The very best way to know what you want in an executive is to act in the role yourself—not just in title, but in real action: run the team meeting, hold 1:1s, set objectives. CEOs resist this because they think they lack the knowledge, but that worry is precisely why they should act—it's the only way to learn what your company actually needs from that role, today.

On preparing to hire an executive for a function you've never run.

Hire for strength, not lack of weakness. Everybody has serious flaws (including you); they're just easier to spot in some people. Optimizing for absence of weakness just means optimizing for pleasantness—instead, find someone world-class at the specific strengths you require and tolerate their weaknesses in less important domains.

Don't hire from central casting. The head of sales at Oracle in 2010 would have failed in 1989; the VP of Engineering at Apple might be exactly wrong for Foursquare. You're not hiring an abstract executive for an arbitrary company—you're hiring the right person for your company at this particular point in time.

Assemble the interview team with two distinct groups in mind: (1) people who can best assess whether the candidate meets the criteria, and (2) people whose support the executive will need to succeed once hired. No matter how great an executive is, they'll fail if the people around them sabotage them—so surface integration issues before the offer.

It's possible for an executive to be well-liked and totally ineffective, and it's possible for one to be highly effective, profoundly influential, and totally despised. The latter is far better. Effectiveness—not likability—is the criterion for team fit.

Front-door references are still valuable even though they're guaranteed positive. You're not looking for positive vs. negative—you're looking for fit against your specific criteria. Front-door references often know the candidate best and can speak to those specifics in ways backdoor references can't.

Write the interview questions in advance even if you never ask them. The act of writing forces a level of specificity about what 'good' looks like that you cannot reach any other way, and it aligns the interview team on a common bar.