Inside A nugget challenge
Walking you through everything we consider when creating our challenges.
Walking you through everything we consider when creating our challenges.
The creation of each challenge and its components involves very rigorous generation and review processes. Let’s go over how we create each of these pieces (for more of the science behind each of these pieces, click here!).
Each rank-ordered question can be broken into two key parts: a scenario and its corresponding response options. The scenario outlines what is happening at a (imaginary) workplace, placing the user into a specific work context. The response options then provide the user with four options of what they could do in the scenario to allow them to outline what they would be most likely to do. When creating the scenario, we use the following considerations:
Taking into account all of these considerations, an example scenario from our challenge library is:
“You are a team lead running a meeting with Avery, the junior lead recently assigned to your team. The meeting is with your company’s most important client, Hayden. The stakes are high with this meeting, so you have been dreading the thought of it for weeks. It is the halfway point now and you call for a 15-minute break.
You exit the meeting room, with Avery following behind you.
Avery says, “Is it just me, or are we doing just absolutely horribly right now?” You reply, “Yeah, the client hates pretty much everything. It seems to me that Hayden’s just not interested in changing any part of this process and that the company just wants to stick to their old ways.”
Avery nods in agreement and says, “Yeah, I definitely agree with you. It doesn’t seem like Hayden is really listening anymore. They are just taking every opportunity to shoot down what we say. So, what do we do?”
You can see that the scenario is work-related, broad while providing key pieces of information required to make a decision and the language is not specific to any one type of job.
The next step in the process is to create the four response options for the rank-order question. Each response option reflects a different level of the target skill. Some additional considerations we take into account when creating the response options for the rank-order questions are:
With these considerations, response options for the scenario above are:
So with that we have our rank-ordered question laid out! Are we ready to move ahead? Not quite.
We always make sure we do a thorough review of the content before moving onto the next step. When reviewing the content, we revisit all the things we were meant to consider during the creation of the rank-order question to ensure each consideration was properly adhered to. Specifically, we use the following list:
Once the rank-ordered question has been thoroughly reviewed and we verify that it adheres to each of these criteria, we are able to move it into production.
For the open format questions, we must create a scenario for the person to respond to naturally and flexibly. We may ask the person completing the challenge to either write a message on a specific platform (e.g., Teams, Slack), write an email, write a script of what they would say to someone in person, write a brief action plan, or another work activity that is relevant in that scenario. When creating the open format questions, we make the following considerations:
An example open format question from our library is:
“After communicating your plan to Avery, you just finish filling up your water bottle and look down at your watch. You wince at the realization that the break is coming to an end.
You nudge Avery to start walking towards the meeting room. You both enter the room and see that Hayden is just getting off what sounded like an important call.
Hayden lets out a sigh and says, “What’s next?”
You take a deep breath and say….
Write a script of what you would say to Hayden.”
We will then review each work sample. When doing so, we ask ourselves the following questions:
Taking it all in, it seems like a lot, right? Although it is a time and resource intensive process, we believe our challenge creation process is totally worth it. This process helps to give us confidence in how we are measuring each of our soft skills and helps us to provide clients and users with accurate soft skill scores. We should note, however, that this is just the first step in ensuring we are doing the best things possible for our clients.
How do we make sure that we are accurately assessing our soft skills beyond the thorough challenge creation process? Check out this post to find out more!