“How much should you charge to wash all the windows in Seattle?” “Explain the internet to someone who has never heard of it before?” “How many golf balls could fit in a school bus?”
Answer these questions in the correct way (since there was often not a correct or, at least, knowable answer), and you might get a job at some of the biggest technology companies in the world.
It seems a little strange.
Yes, the alleged brightest minds in the hiring world thought that these questions would reveal who could cut it at their big tech factories. These assessments, given in interview format by inconsistent and often unqualified graders, probably revealed something, but it isn’t a very good assessment of skills.
For recruiters outside of the tech industry, using these types of questions probably sounds a lot like the hiring manager who likes to ask, “If you were a tree, what kind of tree would you be?”
You wouldn’t be far off.
There should be a better way to assess candidates. While G2 lists over 100 technical skills assessment solutions of varying quality, one tech firm decided to take things into their own hands.
A Fairer Hiring Platform for a Better Game Platform
Roblox is a game platform where people can build and play each other’s games. For those of us raised on cartridges that you need to blow on to get to work, it’s a completely different gaming world.
Roblox is also unique as one of the few companies like it that is hiring during this tech winter. In a recent article in Fast Company, there is a comprehensive look at how Roblox is trying to create a better hiring process that is hopefully more fair to candidates. It’s a fascinating story on what this change can look like.
The result of this work was a game that also acted like a skills test. Or a skills test that acted like a game. All of which, of course, was built on the Roblox platform itself.
By some measures, it worked. For one, candidates liked the assessments better. While some might disregard that feedback as unimportant, an assessment is a strong opt-in/out point for candidates in any industry.
The other way it worked is that hires from so-called elite schools dropped and the share of “non-elite” hires grew by almost 50%. We’ll get back to that in a minute.
Before you download Roblox Studio and try your hand at doing this for your organization, it’s probably helpful to understand what you need to accomplish such a feat.
Roblox Spends More Than a Few Robucks on Hopefully Better Hiring
Roblox didn’t just launch an assessment based on what they thought they knew. Instead, it seems like they did their due diligence. That includes a long list of things you can do when you’re one of the most profitable game companies in the world.
- Approach a specialty assessment company about designing a game-based skills assessment and acquiring that company when they are up for sale, even though it’s not your primary business
- Spend two years researching this assessment, including hundreds of hours of interviews and observations with the level of talent they would be testing
- Hone down a list of critical skills (15 in total) and figure out which ones could be tested in a game-based assessment (two!)
- Build and test and rebuild and retest the two assessments, including figuring out how to score the assessments and balance the results
- Still have to use CodeSignal to complete your coding assessments and interviews to complete other assessments like empathy
Although not disclosed in the reporting, we can guess that millions of dollars were spent on a piece of technology that is only capturing a couple of the more than a dozen skills for which they are hoping to hire in their entry-level positions.
Outside of the hiring stats mentioned above, Roblox didn’t share any other diversity data and didn’t share if they had any data on who did well and who didn’t on their tests. It might just be reading between the lines, but it seems like they are still figuring out whether it is truly reducing bias in their hiring process.
When It Comes to Assessments, Don’t Be a Noob
Credit where credit’s due, it seems like Roblox is trying to be careful and systematic about their approach here. Most homemade assessments are slapdash and unscientific. Roblox, with their extensive resources, seemed to at least take the homework seriously. They didn’t come up with dumb questions they picked up off the internet for free and pass it off as some sort of objective cognitive assessment.
But Roblox’s story should give every talent acquisition leader some pause. Doing it right, both by your organization and by your candidates, is going to cost a significant sum of money. You are going to have to tap resources that your company likely doesn’t have on payroll and, if you’re not already a game designer, you’ll have to think differently about assessment design.
Most importantly, results can be difficult to isolate to prove that the investment in better assessments was more than a gamble. For folks like employment lawyers, they’ll want to make sure that anything new won’t be worse than whatever process that passes muster today.
All of those conditions on the results of this work are too bad because we really need more of what’s happening at Roblox and less questions about the shape of manhole covers. It shouldn’t take a vault full of cash and an army of I/O psychologists and game designers working for two years on a project to give us something that (hopefully) works much better.
Until then, we’ll have to find a happy middle ground between what can be confidently built in-house (often very little) and what can be bought or built elsewhere that can also be validated.