I have been acting as a Learning Consultant for various End-User Adoption programs for varied software implementations like SAP, Ariba, Oracle, Maximo, Microsoft Dynamics, and other custom technologies & platforms, for the clients across geographies.
Training Evaluation / Measurement is a critical area for the success of End-User Training. Typically, for training evaluation, we leverage the famous Kirkpatrick model with various levels of evaluation.
Based on my experience & research, I would like to share that we can also simply consider a set of KPIs (Key Performance Indicators) to carry out training evaluation. Here, I will explain the Training Evaluation, mainly for corporate end-user training programs in the context of technology implementation / transformation, for example, End-User Training for SAP implementation.
Based on my experience & research, I have tried to simplify & summarize the metrics as follows. You may want to consider these for training evaluation:
- Participation Metrics
- Performance Metrics
- Impact Metrics
- Usage Metrics
Let’s get familiar with these metrics, one by one.
Participation Metrics
For Participation Metrics, you may consider the following:
- Training Completion Rate: Here, we consider the percentage of employees who complete the assigned training.
- Attendance Rate: Here, we consider the percentage of employees who attend live or virtual training sessions, or complete the self-paced learning modules.
- Drop-off Rate: Here, we consider the number of employees who start but do not complete the training.
- Time Spent on Training: Here, we consider the average time employees spend on training modules.
Performance Metrics
For Performance Metrics, you may consider the following:
- Assessment Scores: Here, we consider the average pre- and post-training test scores to measure knowledge improvement.
- Certification Rate: Here, we consider the percentage of employees who earn certifications upon training completion.
- Knowledge Retention Rate: Here, we use the follow-up assessments (e.g., 30/60/90 days post-training) to measure the retained knowledge.
Impact Metrics
For Impact Metrics, you may consider the following:
- Employee Feedback Scores: Here, we consider the satisfaction ratings based on the post-training surveys.
- Trainer Effectiveness Score: Here, we consider the feedback on trainer performance through evaluations.
- Application of Learning: Here, we consider the percentage of employees who report applying new skills in their roles.
- Reduction in Support Requests: Here, we consider the decrease in IT or Support tickets related to training topics. A high volume of support requests relating to SAP usage suggests that users struggle despite training.
- Productivity Improvement: Here, we consider the changes in employee performance before and after training.
- Production Milestone Achieved: Here, we consider the number of units / products completed during a specific period after go-live, for example, number of orders processed in the first week.
- Error Rate Reduction: Here, we find out whether the training led to fewer mistakes or compliance violations over a period of time.
- Time to Proficiency: Here, we evaluate the speed at which employees reach full competency in their roles.
- Reduction in Process Cycle Time: Here, we consider the time taken to complete key processes, for example, order fulfilment, approvals, etc.
- Self-Sufficiency Rate: Here, we track how many users can complete tasks without additional support or helpdesk assistance.
- Customer Satisfaction Improvement: Here, we look at the increase in customer satisfaction scores post implementation as users start using the new system post-trainings.
- ROI of Training: Here, we compare training costs against measurable business improvements (e.g., increased efficiency, reduced errors).
Usage Metrics
Usage Metrics help determine whether users are actively using the new system (e.g., SAP) after training. For Usage Metrics, you may consider the following:
- Login Frequency: Here, we track how often users log in to SAP. A low frequency may indicate a lack of confidence or resistance to the system.
- First-Time Login Rates: Here, we measure how many trained users log in after training. A low rate suggests they may need additional support.
- Time to First Transaction: Here, we identify how long it takes for users to perform their first meaningful action in SAP. A long delay might indicate training gaps.
- Transaction Success Rate: Here, we track the percentage of completed transactions without errors. A high failure rate suggests training gaps.
- Average Time per Transaction: Here, we compare how long it takes users to complete transactions before and after training. If the time decreases, training was effective.
- Time Saved per Task: Here, we look at the reduction in effort required for various tasks.
- Tasks Completed per Employee: Here, we look at the increase in the number of tasks completed per user post-training.
- Error Rates per Transaction: Here, we identify how often users encounter system errors, such as data entry mistakes or workflow issues.
- Rework or Corrections Required: Here, we measure how often users need to correct data entries, which indicates training effectiveness.
Note: If you get low results for Usage KPIs, there could be several other issues in addition to Training, for example, e.g., system downtime, system glitches, technical issues, etc.
Data Sources
To collect data for various metrics, we can consider various data sources, such as:
- Surveys, Learning Management System, User Feedback Sessions
- Digital Adoption Platform Reports to check the consumption of In-app learning
- Departmental Reports, Business Intelligence Platform
- Data from Help Desk Tools (e.g., ServiceNow)
- Built-in System Logs & Reports
- Built-in Software Analytics to check page views, user interactions, tasks per user, etc.
- SSO & Authentication Logs to check login frequency and daily/weekly/monthly active users
- Various custom APIs enabled in the system
- ...and so on
An Example
Now, let’s consider one imaginary example of SAP Training Program to understand the application of these training evaluation metrics at a high level.
- Imaginary Company Name: Global Manufacturing Corp (GMC)
- Training Program Name: SAP S/4HANA Implementation for Finance & Procurement Teams
- Scenario: Global Manufacturing Corp is transitioning from legacy ERP systems to SAP S/4HANA. To ensure a smooth transition, they conduct a corporate training program for 200 finance and procurement employees. The training includes:
- E-learning modules (basic SAP navigation, finance, and procurement processes)
- Live Workshops / Instructor Sessions with SAP consultants (Trainers)
- Hands-on practice in a sandbox SAP environment
Now, let’s try applying the Participation, Performance, Impact, and Usage metrics for this scenario.
Participation Metrics
A few examples of Participation Metrics are mentioned below.
- Training Completion Rate: 92% of employees completed the assigned SAP training modules.
- Attendance Rate: 88% attend at least one live SAP session with Instructor.
- Drop-off Rate: 8% of employees start but do not finish the self-paced training.
Performance Metrics
A few examples of Performance Metrics are mentioned below.
Assessment Scores:
- Pre-training Test: Avg. score 50% (low familiarity with SAP)
- Post-training Test: Avg. score 87% (significant improvement)
- Certification Rate: 80% pass the final SAP proficiency test and receive internal certification.
Impact Metrics
A few examples of Impact Metrics are mentioned below.
- Employee Feedback Scores: 4.3/5 based on a post-training survey.
- Trainer Effectiveness Score: 4.6/5 based on feedback from workshop attendees.
- Application of Learning: 85% of employees report being comfortable using SAP in daily tasks.
- Reduction in Support Requests: IT helpdesk sees a 35% decrease in SAP-related tickets within 2 months.
- Productivity Improvement: Finance teams report a 30% reduction in time spent on purchase order processing.
- Error Rate Reduction: Invoice processing errors drop by 20%, reducing rework and delays.
- Time to Proficiency: Employees reach full efficiency in 4 weeks instead of the expected 6 weeks.
- ROI of Training: Organization recovers training costs in 5 months due to improved process efficiency and fewer errors.
Usage Metrics
A few examples of System Usage Metrics are mentioned below.
- Login Frequency (Number of times users log in after training): Example - 80% of trained users logged in within the first week.
- First-Time Login Rates (Percentage of users accessing SAP for the first time post-training): Example - 95% of users logged in within two days of training.
- Time to First Transaction (How quickly users complete their first task): Example - 70% of users completed their first purchase requisition within 24 hours.
- Transaction Success Rate (Percentage of transactions completed without errors): Example - 85% of purchase orders were created successfully on the first attempt.
- Average Time per Transaction (Time taken to complete key SAP processes): Example - Time to create a purchase requisition reduced from 15 minutes (pre-training) to 8 minutes (post-training).
- Error Rates per Transaction (How often users make mistakes): Example - Incorrect data entry errors decreased by 40% after training.
- Rework or Corrections Required (Frequency of users having to redo transactions due to mistakes): Example - The number of rejected purchase orders decreased by 25%.
- User Self-Sufficiency Rate (Percentage of users completing tasks without assistance): Example - 90% of users completed procurement transactions without supervisor help.
Conclusion
We can creatively use the above KPIs as necessary in the technology / system transformation programs to evaluate the success of End-User training programs.
