3 IMPORTANT KEYS TO GET STARTED WITH MORE PURPOSEFUL AND EFFECTIVE EVALUATION
The Learning Gym recently hosted a virtual webinar, with HumanCap, based in Thailand, on an Introduction to the New World Kirkpatrick Model. Melanie Martinelli outlined 3 Important Keys to get started with more purposeful and effective evaluation.
Key 1: Begin with the End in Mind. In the Kirkpatrick program we begin with Level 4 – Results. It’s a little bit like using a GPS. When you get into a taxi or start a trip, if you don’t key in the destination before you start moving, you’re not going to know how to get there, or which is the best or fastest route to take. It’s the same with evaluation, if we don’t know the desired end destination, we won’t really be able to address why we’re doing the training or what we’re trying to achieve by doing it. We will also not know if we’ve reached the desired destination if we haven’t defined that at the start. We might indeed need a training to get there, but we need to first determine where “there” is.
Often a client approaches us, for example, for a Leadership Program, and there is the temptation to ask “What do you want covered? How many days should it be?” We need to move away from these “classical (content oriented) questions” and instead ask the stakeholder a few important questions to clarify what the desired end destination (in Kirkpatrick Language Level 4 Result) should be. To clarify the end destination, for example, we need to ask: what problem is this training supposed to solve? Why do we need a leadership program? What in the business is currently happening that tells us leadership development should be a priority? What outcomes would signify success – are there any key company metrics the client is hoping to improve with this initiative? Why leadership, why now, why is this topic keeping you up at night and why should we invest money and effort into it?” It is also very important always ask the so-called second question, “tell me more…” Get the stakeholders talking, engage them, dig deeper to find out what issues they are actually trying to resolve.
The first step of evaluation is to be really clear, where are we going and why? It is a lot harder to evaluate if we’ve reached our goal, after the training, if we haven’t defined it beforehand. More importantly there’s a big risk that we could design a program that has no impact, because we haven’t fully understood the real challenge. This approach requires a shift in mindset. Evaluation doesn’t happen after a program, when handing out surveys, it starts during planning and is about agreeing on the goals, end results and what success factors look like.
A simple way we like to think about the Kirkpatrick Model is that together with our stakeholders we want to agree on the so-called Return on Expectations (ROE). “What does a successful training initiative deliver to key stakeholders, demonstrating the degree to which their expectations have been satisfied”. These expectations can vary from program to program, they can be at different levels, but we want to make sure we are aligned on it from the start.
Here you can find an article about why at Kirkpatrick we prefer to talk about ROE rather than ROI (Return on Investment), because it’s not always about how much money you get back. There might be times when you conduct a program because you see there’s a big attrition rate in your organisation and so you’re investing in management development, leadership development, and employee well-being programs, so your ROE is linked to attrition and it’s not about the investment. Alternatively, you could run a program based on declining sales numbers or customer satisfaction, wherein the ROE might be to improve customer satisfaction. It’s very important to understand where the need for the program originated from. Next you might ask yourself “But how do you evaluate a program where the ROE is many months down the line?”
Often when you start evaluating, the journey looks a little bit like this in the fog. It’s easy in the early stages after the program to see the impact. It’s very tangible to see whether people liked the program or whether they learned what they were supposed to learn. But the further out we go after a program the blurrier it gets, especially if the ROE is very long term. You’re not going to improve sales numbers or improve attrition rates within one month, so how do you connect the training to a goal that is really far out in the future without losing track? This brings us to the second key.
Key 2: Use Leading Indicators to Monitor Progress. Evaluation is not a one-time affair. Evaluation is an ongoing process. Let’s use an example from the sports field to explore the concept and importance of Leading Indicators. You want to run a marathon within 3 hours. Completing the marathon within 3 hours is your Level 4, your ROE. You’ve been training regularly, eating healthy, and sleeping well so that you can finish the marathon in 3 hours. But, if your only goal is to complete the marathon in 3 hours, we’ll only know if you accomplished it when you cross the finish line. And if you cross that finish line in three hours and one minute, you’ve missed your mark and there’s no way to correct it. However, if you use Leading Indicators, in this example, you could monitor if you are on track every 5km and adjust your speed accordingly, you’ll have a better chance of completing the marathon in 3 hours. By setting yourself little milestones, you can check whether you are on track to reach that final goal.
Leading Indicators are in Melanie’s opinion one of the most important, but unfortunately, most neglected aspect of evaluation. We very often set that final goal, but if we don’t have these short-term milestones, it becomes a game of chance or luck, whether we reach our goal. But once at the end, we have no way to adjust if we missed the mark.
One thing most people struggle with is to connect the training to the ultimate goal. Especially if the training and ultimate goal are a significant time apart. You must bring these two together to show your stakeholders that you’re making progress in the right direction. The more indicators we have, the more equipped we are to see if we’re heading in the right direction. It is not about claiming that our training fixed the company’s problems, but it is about showing that our training indeed did have a positive impact on some of those key metrics that were disturbing us before.
Key 3: Focusing Resources on What Matters Most. This brings us to the ultimate question of how and where to focus our resources. Which of the 4 Levels needs the most attention? We would recommend spending a lot more effort on Level 3, as this is where the magic happens. We often think Level 3 – Behaviour – is the hardest to do, but it usually has the biggest payoff. Here the idea is to focus a lot more on making learning stick, versus pure evaluation. What we mean by this is, we need to think a lot more about how we support our learners once they have left the safe environment of the training room and are back to work. Once they’re back at work it’s no longer a case study, or a mock presentation, or a safe playground, it’s reality. This will be a different environment, as they have competing priorities and old habits tend to get in the way of applying what they’ve learned. This leads to obstacles that get in the way of reaching those Leading Indicators and the final Level 4 Results.
Robert Brinkerhoff, a thought leader in the field of Training Effectiveness has conducted many studies that came to the same conclusion. Typically, only one out of six people (15%), will have the inner desire, the volition, to commitment to applying what they’ve learned. Four of the remaining Five people, or 70%, will try, and these are the ones that are important. These are the people that we need to focus on. These are the people that after a training session are motivated, to a certain extent, to actually apply what they learned, but maybe lack confidence, practice opportunities, or maybe it’s just old habits getting in their way. We all know how hard it is to change behaviour, so this is where our focus needs to go if we want to produce evaluation data that allows us to congratulate ourselves and reach that ultimate goal. Offering a training is not enough, we need to find a way to work with those that need support afterwards. Yes, there will always be that one person that isn’t even going to try and that’s ok, it is possible to 85% of participants to make the necessary behavioural changes to reach the ultimate goal. If you don’t focus your resources on on-the-job support, you will spend a lot of money in training, but you will incur the risk of non-sustainable transfer success; meaning people might have had a great time in training, they might have learned a lot, but they are not going to be using those skills back on the job.
The good news is there’s a way that you can get closer to 85% of transfer success. It requires a shifting of mindsets and a shifting of where we spend our resources. We need to focus a lot more of our resources on Level 3. Not just evaluating behaviour, but focusing on building an ecosystem, so-called “driver package”, in the Kirkpatrick language, which is all about processes and systems that reinforce learning, monitor learning application, encourage it, but also rewards the performance of critical behaviours on the job.
We need to think about ways that we can further support the application of the learner, outside of the training room, so it leads to desired behavioural change and hence starts impacting the Leading Indicators.
So often we see people spending so much time and effort fine tuning the design elements of their training program, which is great, but the training program itself only has a small impact on the application afterwards. It is a starting point to build the skills and the knowledge needed, but training alone isn’t going to lead to the desired results. If you have an opportunity to make changes, this is where a lot more needs to happen to create an ecosystem, with the help of supervisors and business leaders, to build, so-called, tactical bridges, to support people on the job.
We hope you found these 3 Keys insightful and helpful. To know more about the New World Kirkpatrick Model click here.
