Behavioral Science and Talent Evaluation
| 25 Jun 2015
I was listening to an episode of my favorite podcast: You Are Not So Smart the other day, and his guest was Google director of People Operations (Human Resources): Laszlo Bock. You should go listen to the whole podcast, because it was really fascinating. One part that caught my ear was when Mr. Bock was talking about hiring, and how managers hire within Google [20 minute mark]. Their research shows that college sophomores can make the same judgement on an interviewee that trained HR professionals can within just 10 seconds of watching the interview. That is astounding. What Google has found is that that first impression is so strong that the rest of the interview, the interviewer is confirming the biases they've formed within the first 10 seconds of the interview. They are still professional and get good answers and good results from the interview, but their personal bias is so subconsciously overwhelming that they interpret the information they've gathered to fit their bias.
"The single biggest thing, the most valuable thing you can do, is- whoever's doing the interviewing, and especially the hiring manager, do not let them make these decisions" -Lazlo Bock
He then goes on to talk about Google's hiring process. They have interviewers, and then they send it to a hiring comitee. So they solved it with a number of ways. Potential candidates go through a number of interviews. Their answers are compiled. It's given over to an HR comittee that has never met or seen the candidate does the hiring. They've found this to be an extrodinarly effective way of evaluating talent.
So the thought hit me: how can this apply to hockey?
To me, interviewers are very analagous to hockey scouts. That talent evaluators basically make up their minds on people within 10 seconds of seeing them is why I am extremely skeptical of scouts opinions on players. This is true for talent evaluators, and it's almost certainly true for hockey scouts as well, since they are, in fact, human. (This goes for people on the internet whose primary way of evaluating players is watching them too). Scouts aren't useless, and I want to make sure I don't shortchange what they do. There are still people whose inputs I greatly valiue, and who have a lot of insight. The draft vets this out. Better players are generally taken at the top of the draft. Just like the interviewers: Scouts are still good at their jobs and they still pick out talent. It's just that they also let their biases cloud their ability to finally judge them.
So how could hockey apply these principles for talent evaluation? How could hockey take advantage of a scouts knowledge and insight, but try and strip out the bias that clouds their judgement? A place to start, IMO, is follow Google's model as closely as possible. So here's how I'd apply these few things to hockey (ok. I'd do a lot more research before doing this, but this is a start based upon these interviews and other behavioral science principles):
• I'd send scouts to view prospects multiple times. Seperately. I wouldn't really let them converse with each other, and I'd attempt to keep them away from media coverage (and corrupt their own evaluations with others' biases). They would all be responsible on every prospect they've viewed at least 5 times. (obviously potential 6th round picks wouldn't have 20 scouts worth of data, but as much as possible) We'll call these scouts write-ups.
• I'd have an analytics department gather as much data on these prospects as possible. All the relevent stats possible. (and I'd try and see if someone's tracked the advanced ones for all these leagues) These would be a player's analytics write-up.
• I'd have a 3rd group of people combine the analytics and scouts write-ups together. Then, they would strip out as much of the biased language (big body, good motor, etc) as possible. I'd even have them strip out size (size bias is rampant). The player would be marked by "player A, position and handedness" This would be the final write-up, and hopefully it's stripped out the scouts AND analysts biases.
• I'd then send it on to a 4th comittee (maybe GM and Asst GMs, or maybe not). This would be analagous to Google's hiring comitee. They would take these full write-ups and rank the players based on those final write-ups. Then reveal the palyers names on the list.
There's your draft list. Choose the top one available.
There's probably a lot of tweeks to this that could be applied (there's a lot more behavioral science that could be applied). But this, to me, is a starting framework to start applying the very valuable stuff that's being discovered in behavioral science to hockey.
This would take a giant leap of faith because what I just laid out above lays out something that's almost crazy when it's said a different way: The guys deciding who to draft are not going to be watching those players play! That's a concept that's quite a bit different than what hockey does now. I don't see hockey culture being anywhere near ready for this. Hockey culture isn't really known for being on the innovative heels of Google. Any organization that did this would face intense scrutiny (especially from a hockey media that revels in the pride of it's own crustiness).
But, I think the behavior science in hockey is worth investigating, and the steps I laid out above, modeled after Google's hiring process, based on behaviorial science, is a way to get a more comprehensive evaluation process.