Ifyou’re responsible for deliveringhigh-qualitytraining, you’ll understand the importanceof measuring the effectiveness of the training you provide.Enterprisesof all sizes understand thatwithout formally assessing training effectiveness, they cannot calculate whether it was worth theinvestment. This is known as Return on Investment (ROI) and in today’spost,you’ll learn how to measure your ROI with the Kirkpatrick model.
What is Kirkpatrick’straining evaluation model?
In 1959, Don Kirkpatrickintroduced his learning evaluation model through a series of articles published in the Journal of the ASTD. He later expanded the model and, in 1993, published the Four Levels of Training Evaluation. This book revolutionized how businesses evaluate their training programs.
The Kirkpatrick model features four distinct levels:
Level 1: Reaction
Level 2: Learning
Level 3:Behaviour
Level 4: Impact
Here’s a brief overview of each level:
Level 1: Reaction
The starting point for the Kirkpatrick Model is to find out how the course participants reacted to the training. Werethey satisfied or dissatisfied?Why?
Level 2: Learning
Thesecond Kirkpatrick level investigates what learning took place. The most common approach is to ask the trainees to taketwo quizzesor tests, onebefore andoneafter the training.
Level 3: Behavior
The third level looks at whether any of the skills or learning from the training makes it into the workplace. It studiestheon-the-jobperformance of the trainees and sees whether they areapplyingwhat they’velearnedwhile at work.
Level 4: Results
The lastKirkpatricklevel examines whether the stakeholder’s expectations were met and if so, to what extent.
So, what do you notice?
The Kirkpatrick model doesn’temphasizethe measurement ofReturn on Investment (ROI).
What does the Kirkpatrick Modelactually measure?
The Kirkpatrick Model model primarily measures training effectiveness and impact based on shareholders' expectations. Don Kirkpatrick termed this metric ROE -Return on Expectations.
Here’s the deal:
The stakeholders in anyorganizationhave certain expectations of what any given training should deliver. TheKirkpatrickmodel emphasizes the need to accomplish a return ontheseexpectations.
Over time, Don Kirkpatrick’s original model has evolved. Don’s son James and Wendy (Don’s daughter in law) have expanded the model to create the New World Four Levels and the Kirkpatrick BusinessPartnershipModel. Interestingly, the latest versions of the Kirkpatrick Model still focus squarely onROE – Return on Expectations.
But doesn’t 'ROE' mean 'Return on Equity'?
Yes. This is where confusion about the Kirkpatrick Model comes from. ROE commonly means ‘return on equity’ which is calculated by comparing shareholders’ equity to the net incomeof a company. The Kirkpatrick Model uses‘ROE’to mean ‘return on expectations’ and describe these expectations – those of the shareholders – as the ultimate indicator of value.
So, what’s ROE in the Kirkpatrick Model?
ROE is a holistic measurement of all quantitative and qualitative benefits thata trainingcourseor programprovides and to what extent they meet shareholders’ expectations. Formal training is usually at the core of theKirkpatrickapproach but it can also include any initiative or program that comes about through intervention.
To measure ROE in the Kirkpatrick Model, those responsible for the learning must clarify the followingareas:
What are the expectations of the stakeholders?
How can these expectations be converted into business outcomes?
Are these outcomes measurable or observable?
Can the learning process be refined or improved to increase ROE?
If you notice, the Kirkpatrick Modelfocuseson determining whether training producesworkplace changes and whether these changes meet stakeholders’ expectations. There is no emphasis onfinancialformulas, assumptions or estimates.Crucially, ROI isn’t a staple of the Kirkpatrick approach.
How does ROI fit in with the Kirkpatrick Model?
The Kirkpatrick Model was already well established by the time that Jack Phillips published his own work on training evaluation in 1980. Phillips wanted to expand the Kirkpatrick Model beyond Level 4 and offer enterprises a way ofcalculatingthefinancial returnof a training program. This is known as‘return on investment’(ROI) and Phillips labeledhis methodology ‘Level 5’.
What is Level 5?
Level 5 isn’t featured in theoriginalKirkpatrick Model but isan additional levelof evaluation created by Phillips that uses cost-benefit analysis to determine the value of training programs. In other words, it helps companies calculate whether the money they spent on training produced measurable business results.
Look:
Let’s imagine that a company pays for an expensive, high-profile training program. The original Kirkpatrick Model would only evaluatethe effectiveness of the training, whether it translated into workplace changes and whether these changesmet the expectations of the company’s stakeholders.
Phillips’ Level 5 training evaluationgoes one stepfurther:
Level 5evaluationhelps a companydetermine whether the money they spent on the training translated into real-world benefits such as increased revenue or decreased operating costs.
What other changes did Jack Phillips make?
In addition to adding a fifth level, Phillips alsomakesubstantial changes to the original Kirkpatrick Model.
Level 3
Whilethe originalKirkpatrickModel looked at behavior in the workplace, Phillips expanded this level to coverApplication and Implementation. Thissubtledifference made it easier for companies to seewhytraining wasn’t being converted to on-the-job changes. Was it beingappliedincorrectly? Or implemented ineffectively?
Level 4
Phillips alsobroadenedLevel 4 and renamed it ‘Impact’.This expanded version of Level 4 helped identify whether processes other than training were responsible for bringing out learning and outcomes.
Introducing the Phillips ROI Methodology
Tounderstand how to measure ROI using the Kirkpatrick model, you need to understand what’s known as thePhillips ROI Methodology. This methodology helps to isolate the effects of any given training program.
This differs substantially fromthe original Kirkpatrick Model:
Don and his son James wrote in their 2007 revised book that “We are not isolationists”. What they meant was that they took a holistic approach to training evaluation and didn’t try to pinpoint specific data that led to results.
Phillips’ ROI Methodology aims toisolate the effectsof a program so that the cause of improvements identified in Level 3 and Level 4 can be correctly attributed.
How tocalculateROI using Phillips’ Methodology
Phillips’ methodology uses a variety of techniques to isolate the effects of a training program or course.
These include:
Estimates that have been adjusted for error
Control groups
Trend line analysis
Forecastingmodels
To measure ROI, additional information is required.
ROI compares the financial benefits of training to the costs of the training itself. The same scales or metrics can be used to measure both benefits and costs as they are both measuredwithmoney. To implement Phillips’ Level 5 ROI calculation, businessesmust essentiallyconduct atype ofcost-benefit analysis. This is known asROIDeterminationor Level 5for short.
How to measuretheROIof your training
ROIdeterminationhelpscalculate whether training has impacted business revenues or profit and how this compares to the cost incurred. Demonstrating the ROI of training is difficult as you are trying to link the training to a quantifiable factor, such as a process improvement or productivity improvement. This gives you a monetary figure that reflects the value of the training to the company’s bottom line.
Step #1.Decide how you will measure ROI
Therearevariousways to measure ROI and how you measure it will impact on which factors you measure, before and after training.
The standardformulafor measuring the ROI Of trainingisas follows:
ROI (percentage) = ((Monetary benefits – Training costs)/TrainingCosts) x 100.
Alternatively,you can measure ROI in terms ofthedecreasedcost or time to produce a product, either on a per-item basis or in bulk.
Step #2. Choose which factors you’ll measure
This step depends on what training is offeredand the expectations of the stakeholders. Which areas do they want the training to impact?
For example, let’s say that you want to calculatethe ROI of a training course for a pottery business. The company employs five potters and they make an average of 40 pots each over a typical 40-hour, 5-day week, or an average of one pot per hour.
The training course aims to teach the potters how to useanew, faster glazing technique to glaze the inside ofthe pots. The stakeholders’ expectation is thatthis newtechniqueshould increase the rate at which potters can make their pots.
Step #3. Calculate your company’s pre-training production costanalysis
Collect detailed financial information about your company’s costs including overhead, distribution, facilities,equipment, materials,andwages.Calculatethe company’s gross profits prior to the training taking place.
Let’s say that the pots cost the company $5 to make andthat each potcan be sold for a profit of $5 after all other costs (distribution, facilities,andoverheads) have been accounted for. Over a 50-work week year, each potter generates of profit of $10,000 for the business.
Step #4. Calculate the cost of the training andthenprovide the training
Yournext step is to calculate the cost of training the potters to use the newglazing technique. If it costs $1,000 to run the course for all five potters, the cost of training is $200peremployee.
Once you’d established this cost, you’d then run the training course as planned.
Step #5. Conduct post-training production costanalysis
Atsome point following the training, you’d run the same analysis ofthe company’s costs as you did before the training. In our exampleaboutthe glazing training course, you should be able to observe that the potters areable to complete morepotsin the same amount of time, using the newtechnique they havelearned.
You can compare pre- and post-productionratesand determine whether an improvement was made.
Step #6. Calculate the new benefit to your company
Youcan now compare employee productivity since receiving the training with their pre-training performance. You should see a clear increase in both employee productivity and the company’s profits. This helps you calculate the net benefit to your company as a result of the training.
For instance, following the introduction of the new technique, the pottersare able tocomplete an average of 45 pots per week. Over a 50-work week year, each potter will now be generating an averageannualprofitof$11,250 for the business. This is £1,250 more profit compared with before the training.
To calculate the increased profit that the employee now makes for the company, you should use the net benefit –theincreased profitof $1,250 per potter– and subtract the training costs ($200 per employee).
Apply the ROI formula noted above andcalculatethe percentage return on your investment.
ROI (percentage) = ((Monetary benefits: $1,250 – Training costs: $200)/Training Costs: 200) x 100.
This gives an ROI of 5.25%. ROI is usually reported as a percentage and represents the annual net benefit of the training beyond the initial investment.
This demonstrates that the training was worthwhile and resulted in increased profits for the company.
You will see a positive ROI whenthe impact onthecompany’sprofitsishigher than the cost incurredfrom offering the training.
How does the ROI statistic help?
Now that we’ve explored how to measure your ROI, you canuse it tocompare the effectiveness of the training to other programs either within your company or from other companies. ROI can also be used to makeinvestmentdecisions andforecastbenefits.
Do all training programs need an ROI study?
It’s worth noting that according to Phillips, hisROI Methodologyisn't requiredfor all training courses.As the highest possible evaluation level, a wide variety of factors such as the goals, costs,andvisibility of the training will determine whether an ROI study makes sense.
Listen:
A simple in-house professional development course may not require an ROI study.Whereas a costly course with far-reaching implications may warrant one.It depends on what Phillips termed the ‘Chain of impact’.
Phillips’‘Chain of Impact’vs Kirkpatrick’s‘Chain of Evidence’
Phillips’s guidingprinciple for deciding which training courses require a comprehensive ROI studyisknownas the ‘Chain of Impact’. This is quitesimilar toKirkpatrick’s ‘Chain of Evidence’. It can help you decide which training course requireswhich level of evaluation.
In her book,TheBottomlineon ROI,Patti Phillips’ suggested that between 90 and 100 percent of training courses should be evaluated at Level 1 (Reaction), yet this number drops as you climb the levelsof evaluation. According to Phillips, only five to ten percent of training courses require a Level 5 ROI evaluation.
Which approach should you use?
The choice between using the four-level Kirkpatrick Model or the five-level Phillips’ methodology depends on your stakeholders’ expectations.
Do they want to focus on simply getting a return on their expectations (ROE)?
Or, do they want to see a return on their investment (ROI)?
If it’s the former, then Kirkpatrick’s makes sense. If the latter, Phillips’ expanded version of Kirkpatrick is your best option.
Summary
Hopefully,this post has given you a good understanding of how to measure your ROI using an expanded form of the Kirkpatrick model. As we’ve shown, ROIcalculationsare unnecessary for the vast majority of trainingcoursesbut can be a helpful way tohelping an organization reach its program and business goals.
Need More?
For more tips on how to use the Kirkpatrick model to evaluate training and maximize business impact, download our free white paper.
Related posts:
The Noob Guide to Understanding Kirkpatrick model evaluation
How to Master Kirkpatrick model of training evaluation in 6 Simple Steps
Kirkpatrick levels of evaluation: Expectation vs reality
Kirkpatrick level 3: Free evaluation examples
