Evaluation Report for Electronic Customer Reporting (eCR)



Evaluation Report for Electronic Customer Reporting (eCR)Eric J. EricsonPurdue UniversityEDCI 577Evaluation Report for Electronic Customer Reporting (eCR)Electronic Customer Reporting (eCR) is a report generating division of the Ericson Employer and Consumer Services Company (EECSC). Annually, the company trains newly hired employees to use the specialized systems to assist clients who call with questions, problems, or technical guidance requests. One particular system is brand new and slated for launching in the 3rd quarter of 2013. This system is named after the division: eCR. It replaces an older, outdated, system: Manual Customer Reporting (mCR). The Design Strategy Planner is in charge of ensuring new training materials are ready for the 2nd quarter of 2013 for quality testing and eventually 3rd quarter training. This training course will fit into the same slot as mCR had in an already utilized large new hire curriculum. Julian Ericson, the owner’s grandson, is charged with developing the training materials. Eric Ericson, the owner’s son (and Julian’s father) is charged with evaluating the training materials using the Kirkpatrick Four Level Evaluation framework. The evaluation will provide metrics for future use in measuring the program’s effectiveness.Evaluation Goals and ScopeTo know the audience who will be completing the eCR training module, we can look at the Ericson Employer and Consumer Services Company’s hiring strategy. Traditionally, those hired to process mCR (eCR will use the same qualifications) have a minimum of the following:A high school diploma or equivalent.2 years’ experience in a call center environment.Able to type 50 words per minute (wpm).The new hires will undergo training in the curriculum prior to this module covering:Company basics.Customer service skills.Customer sensitivity.Core system overview.Each of the trainees were hired as full time employees. It is important to note that EECSC has a small staff of contracted temporary employees as well (that will not undergo the eCR training).GoalsThe goals of the evaluation of eCR is to prepare newly hired staff to operate at 75% proficiency once they “hit the floor.” This standard is compared to current floor figures and measured based on elements such as speed to resolution, average call time, quality, and customer satisfaction statistics. It is also expected for new hires to reach 95% proficiency (according to pre-determined standards) 6 months post training.PurposeThe purpose of this evaluation is to provide Ericson Employer and Consumer Services Company evidence that eCR is being trained at an effective level as measured using Kirkpatrick’s 4 Level Evaluation Model:Level 1 (Reaction).Level 2 (Learning).Level 3 (Behavior).Level 4 (Results).(Kirkpatrick & Kirkpatrick, 2006)Primary ObjectivesFor any organization, transitioning to a new platform can be a grueling process. Ensuring the training yields positive results is of utmost importance. The primary objectives of this evaluation are to:Measure the effectiveness of the training course to demonstrate the value, andInvestigate ways to improve it for future new hire classes.Evaluation ProcessesMy evaluation strategy utilizes Kirkpatrick’s 4 Level Evaluation Model. In it I measure the training program’s effectiveness at:Satisfying the customers (new hire trainees).Educating them (knowledge retention).Altering their behavior and performance (preparing them to perform the work).Impacting Ericson Employer and Consumer Services Company’s bottom line (results leading to gain). See Appendix A for the evaluation plan and also a timeline for completing the evaluation project.Evaluation InstrumentsLevel 1 Evaluation: Reaction. While planning, researching, and prioritizing, I determined that, while Level 1 results are a good indicator of customer (trainee) satisfaction, I placed my least amount of emphasis on this level of evaluation. I decided this after researching ASTD, which recommended spending more money on Level 3 and 4 (behavior and results) and less on Level 1 (reaction) (2009). Regardless of the lower emphasis, I still produced this level of evaluation to get a glimpse from inside the classroom, to ensure the environment and materials were conducive to learning. I understood the value of measuring the feedback from customers (trainees) and used it to ensure that, while efforts are made to incorporate the fundamentals, the course was also fun: fundamental + fun = fully functional. Truly capturing the interest of trainees can endorse a deeper level of learning that can hopefully transfer into the job place through results.See Appendix B for a copy of the Level 1 (reaction) evaluation survey. It was created using Metrics That Matter software.Level 2 Evaluation: Learning. Julian supplied me with the list of training objectives in order for me to write an adequate Level 2 evaluation:After completing this training, you will be able to pass an evaluation at 80% proving you are able to: Classify internal and external Electronic Customer Reporting (eCR) callers.Use the eCR Password Reset Standard Operating Procedure to process eCR password resets for internal callers.Demonstrate using the XYZ eCR Job Aid to:Determine which reports are available.Distinguish what access a user needs to view reports:Confidential.Non Confidential.Expanded.Standard.Provide instructions to an external caller about how to run a report. Demonstrate routing internal callers to the help plete a help desk email for an external caller with required information including his/her:Name.Phone number.Email address.Policy number.User ID.Full description of issue:Date report was run.Report with ments to explain issue.I created three Level 2 evaluations to be administered throughout the training to observe Gagne’s seventh event of instruction. I also created a final Level 2 evaluation. All evaluations were based on the training objectives. I used the evaluation instrument: Articulate Quizmaker 2009 and the company’s traditional LMS. I chose to use electronic assessment technology to harvest benefits such as accuracy, speed, excellent reporting capabilities, and to create some semblance of initial standard metrics. See Appendix C for copies of each evaluation.Level 3 Evaluation: Behavior. In order to assess the transfer of knowledge, and to ensure trainees were effectively performing tasks they learned in training, I interviewed the trainee’s supervisor to acquire information such as performance, quality, and customer satisfaction rating data 6 months after the end of the training program. I also sought to gain the supervisor’s impression of how well the training class had performed on average. The goal of this Level 3 evaluation was to see what they were still practicing on the job. Knowing that even if trainees excelled at demonstrating proficiency during training, that it doesn’t guarantee they would carry over that proficiency on the job, I sought this data to provide solid evidence that they had successfully learned eCR to the desired extent. See Appendix D for the interview questions I asked to their supervisor.Level 4 Evaluation: Results. I evaluated at Kirkpatrick’s 4th level one year after the completion of the eCR training. The criteria that I chose to evaluate the results impacted by the eCR training course were those that corresponded to the strategic goals determined by our pre-training stakeholder/strategy planning meeting. Our goal was that the training and utilization of electronic vs. manual customer reporting would:Increase quality results (higher percentage of work completed without critical errors) from 88% to 95%.Decrease the Turn around Time (TAT) of call resolution from 96 business hours (old mCR) to 48 business hours (new eCR).Increase average calls/hour by 50% (from roughly 6 to 9).Increase customer satisfaction figures from 64% to 85%.The rationale of evaluating these statistics in this way was because real, measurable data such as this would provide a clear direction and ability to make important business decisions regarding fully adopting eCR or reverting back to mCR until a better resolution could be established. Also, with success, these improvements would allow the company to eliminate the need for contracted temporary staff. This would result in a company savings of a projected $312,000/year over year (eliminating 10 positions @ an average of $31,200/yr. expenses [wages, insurance, additional training, other]). The statistics gathered from this first round of level 4 evaluation will be used to establish standards/goal figures that can be used as a benchmark for future eCR trainings.Analyze MethodsLevel 1: ReactionSince the Metrics that Matter software collects and produces reports, the process of analyzing the results was simple. I received reports via email for each trainee with their individual responses numerical data and comments. I manually compiled these figures into an average and forwarded the comments to Julian to pass along to the training team for encouragement and brainstorming. This method was economical and didn’t require a great deal of time.Level 2: LearningAscertaining level 2 data was equally as easy as level 1 data. The traditional LMS system automatically/electronically compiled individual data reports as well as class reports to provide an average. The reporting information was broken down into groups such as question, objective, and individual results. All reports were emailed to me and also accessible via our LMS direct website. Level 3: BehaviorRetrieving Level 3 evaluation data was relatively simple considering the electronic measurement systems we have in place. Supervisors have access to results from performance tracking software that tracks the number of calls processors are able to handle, a quality team that takes a random sampling of processors’ work and measures the actions against our pre-established standard operating procedures (a source of our original training objectives) and calculates a score based on their successful implementation of our rules/standards, and also a feedback phone survey function that some callers are asked to confidentially complete to report their level of satisfaction. All remaining to do was to sit down with supervisors and discuss and view the results, and record them. I placed the results into our evaluation database along with the summary of level 1 and 2 evaluations to make readily available for the stakeholders executive summary. Since the participants were all new hires, it is implied that there are no outside influences affecting their ability to perform, therefore no counter measures were performed to measure such influence. I also made it clear that if any additional training were to be performed, that I must be notified so I can consider these possible influences when evaluating.Level 4: ResultsOne year following training, I viewed the overall quality, calls/hour, TAT, and customer satisfaction statistics. I accomplished this by utilizing the trainees’ supervisor once more to get updated, current data. Again, he had access to electronic performance software, I was able to acquire quality statistics from the quality team and view the electronic compilation of customer satisfaction data produced by our phone survey software. Combining this data into averages and measuring them against our standards yielded comparable results. Stakeholder InvolvementKey stakeholders were involved in the evaluation plan first by being invited to a pre-program planning committee meeting. During this meeting, stakeholders were made aware of program and evaluation goals. They were also asked for their input and what they were willing to take ownership of to ensure the success and smoothness of the eCR transition. Stakeholders were empowered to be a part of the process and to collect data. Stakeholders were asked to ensure that the evaluation-specific data collection was accurate and ongoing throughout the year. Lastly, key stakeholders were invited to an executive summary meeting.Executive SummaryWhile results were encouraging, not all results met the goals. We were all aware we had set very high standards with high hopes. I decided to hold a meeting with the stakeholders, designers, and training staff to discuss the evaluation project’s findings. This would provide an excellent opportunity to explain in detail some of the elements that often get lost in charts and graphs in a memo. This also is advantageous as it allows an opportunity for questions and interjections, potentially even valid suggestions for further investigation or future exploration. View Appendix E for the executive summary meeting minutes.ReferencesKirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating Training Programs. San Francisco: Berrett-Koehler Publishers, Inc.The value of evaluation: Making training evaluations more effective. (2009). Alexandria, VA. ASTD.AppendicesAppendix AThe following is the evaluation plan and next, the evaluation project timeline.(Cont’d on next page)Appendix BThe following is the Level 1 evaluation survey I created for measuring the ECR training program trainees’ reactions.Appendix CThe following are all the Level 2 evaluations I created for measuring Learning.Level 2 Evaluation 1 – Knowledge Check Questions1. (Multiple Choice Question, 0 points, 1 attempt permitted)Correct! Incorrect2. (Multiple Response Question, 0 points, 1 attempt permitted)Feedback when correct: That's right! The correct answers are 1txiong and mnahag2.Neither 1txiong nor mnahag2 end with a u, so they are possible external user IDs.hupto4u and jsalv3u both end in u so you know they are internal callers.Feedback when incorrect: That is not correct.The correct answers are 1txiong and mnahag2.Neither 1txiong nor mnahag2 end with a u, so they are possible external user IDs.hupto4u and jsalv3u both end in u so you know they are internal callers.3. (Hotspot Question, 0 points, 1 attempt permitted)Feedback when correct: That's right! You selected the Change Password link. From here, you enter a user's last name.Feedback when incorrect: You did not select the correct link. You select the Change Password link. From there, you enter a user's last name.4. (Multiple Choice Question, 0 points, 1 attempt permitted)Correct! Incorrect5. (Hotspot Question, 0 points, 1 attempt permitted)Feedback when correct: That's right! You selected Continue to this website (not recommended).From here, you log in.Feedback when incorrect: You did not select the correct area.You need to select Continue to this website (not recommended).From here, you log in.Level 2 Evaluation 2 – Knowledge Check Questions1. Use the XYZ eCR Job Aid to answer the following question.Which of the following reports IS NOT available under eCR? (Multiple Choice Question, 0 points, 1 attempt permitted)CorrectChoice Key Generic Substitution Indicators by Month.XLarge Utilization Count. Membership by Market. Utilization by Diagnosis.Feedback when correct: That's right! You selected the correct response, Large Utilization Count. All other reports are listed in the job aid.Feedback when incorrect: You did not select the correct response. The correct response is Large Utilization Count. All other reports are listed in the job aid.2. (Multiple Response Question, 0 points, unlimited attempts permitted)Feedback when correct: That's right! The correct answer is Confidential and Expanded. Standard level users only see the reports with an ** after them. Non Confidential level users see all reports EXCEPT Detail Payment or Large Loss reports.Feedback when incorrect: Incorrect.The correct answer is Confidential and Expanded. Standard level users only see the reports with an ** after them. Non Confidential level users see all reports EXCEPT Detail Payment or Large Loss reports.3. Use the XYZ Job Aid to answer the following question.Which category is the Top Ranked by Net Paid report under? (Word Bank Question, 0 points, unlimited attempts permitted)CorrectChoice Financial Reports. Managed Pharmacy Reports.XMedical Reports. Membership Reports.Feedback when correct: That's right! The correct answer is Medical Reports. It is located toward the bottom of the Medical Reports list.Feedback when incorrect: Incorrect.The correct answer is Medical Reports. It is located toward the bottom of the Medical Reports list.4. (Matching Drag and Drop Question, 0 points, unlimited attempts permitted)Feedback when correct: That's right! The correct order is:1. Select a Policy Number or Group Segment Filter (GSF).2. Select a Date Type.3. Add More Details to the Report.4. Limit the Report's Results.Feedback when incorrect: Incorrect.The correct order is:1. Select a Policy Number or Group Segment Filter (GSF).2. Select a Date Type.3. Add More Details to the Report.4. Limit the Report's Results.Level 2 Evaluation 3 – Knowledge Check Questions1. (Multiple Choice Question, 0 points, 1 attempt permitted)Feedback when correct: That's right! You selected the correct response: Never.When an external caller is having technical issues with an eCR report, we write the email on their behalf, but do not provide the email address to them.When he/she is locked out, we reset the password via the Employer eServices portal.Feedback when incorrect: You did not select the correct response.The correct response is: Never.When external callers are having technical issues with an eCR report, we write the email on their behalf, but do not provide the email address to them.When they are locked out, we reset the password via the Employer eServices portal.2. (Multiple Response Question, 0 points, unlimited attempts permitted)Feedback when correct: That's right! The correct answers are:-Phone number.-User ID.-Report with parameters.Mailing address and user date of birth are not necessary.Feedback when incorrect: Incorrect. The correct answers are:-Phone number.-User ID.-Report with parameters.Mailing address and user date of birth are not necessary.3. (Fill in the Blank Question, 0 points, unlimited attempts permitted)Choicecustrept_help@custrept_help@.Feedback when correct: That's right! The correct answer is custrept_help@.Feedback when incorrect: Incorrect. The correct answer is custrept_help@.4. (Word Bank Question, 0 points, unlimited attempts permitted)Feedback when correct: That's right! The correct answer is external caller with eCR technical issues.Feedback when incorrect: Incorrect.The correct answer is external caller with eCR technical issues.Level 2 (Learning) Evaluation - Final Assessment Questions1. (Multiple Choice Question, 1 points, 1 attempt permitted)Feedback when correct: The correct answer is: Ends with a u.The quickest way to distinguish between an internal and an external caller is by his/her user ID.When a user ID ends with a u, this indicates an internal user.External callers' user IDs do not end with a u.Feedback when incorrect: The correct answer is: Ends with a u.The quickest way to distinguish between an internal and an external caller is by his/her user ID.When a user ID ends with a u, this indicates an internal user.External callers' user IDs do not end with a u.2. (Hotspot Question, 1 points, 1 attempt permitted)Feedback when correct: You selected the correct link: Users.After clicking this, a new link populates called: Search for User to Edit.Feedback when incorrect: You selected the incorrect link.The correct link is Users.After clicking this, a new link populates called: Search for User to Edit.3. From this screen, where do you click to advance to the eCR Admin Tool? (Hotspot Question, 1 points, 1 attempt permitted)Feedback when correct: You selected the correct location: Continue to this website (not recommended).From here, you log in.Feedback when incorrect: You selected the incorrect location.The correct location is: Continue to this website (not recommended).From there, you log in.4. Use the XYZ Job Aid to answer the following question.Which of the following reports are available under eCR? (Multiple Response Question, 1 points, 1 attempt permitted)CorrectChoiceXKey Generic Substitution Indicators by Month. Maximum Pharmacy Outputs.XMembership by Market.XUtilization by Diagnosis.Feedback when correct: The correct answers are: Key Generic Substitution Indicators by Month, Membership by Market, and Utilization by Diagnosis.Maximum Pharmacy Outputs is fictitious and is not a report available under eCR.Feedback when incorrect: The correct answers are: Key Generic Substitution Indicators by Month, Membership by Market, and Utilization by Diagnosis.Maximum Pharmacy Outputs is fictitious and is not a report available under eCR.5. (Multiple Choice Question, 1 points, 1 attempt permitted)Feedback when correct: The correct answer is: Non Confidential.Standard level users only see the reports with an ** after them.Confidential and Expanded level users see ALL reports.Feedback when incorrect: The correct answer is: Non Confidential.Standard level users only see the reports with an ** after them.Confidential and Expanded level users see ALL reports.6. Use the XYZ job aid to answer the following question.Which category is the Bill Count by Month report under? (Word Bank Question, 1 points, 1 attempt permitted)CorrectChoice Financial Reports. Managed Pharmacy Reports.XMedical Reports. Membership Reports.Feedback when correct: The correct answer is: Medical Reports. It is located near the middle of the Medical Reports list.Feedback when incorrect: The correct answer is: Medical Reports. It is located near the middle of the Medical Reports list.7. (Matching Drag and Drop Question, 1 points, 1 attempt permitted)Feedback when correct: The correct order is:1. Select a Policy Number or Group Segment Filter (GSF).2. Select a Date Type.3. Add More Details to the Report.4. Limit the Report's Results.Feedback when incorrect: The correct order is:1. Select a Policy Number or Group Segment Filter (GSF).2. Select a Date Type.3. Add More Details to the Report.4. Limit the Report's Results.8. (Multiple Choice Question, 1 points, 1 attempt permitted)Feedback when correct: The correct answer is: Never.When an external caller is having technical issues with an eCR report, we write the email on their behalf, but do not provide the email address to them.When an external user is locked out, we reset the password via the eCR website.Feedback when incorrect: The correct answer is: Never.When an external caller is having technical issues with an eCR report, we write the email on their behalf, but do not provide the email address to them.When an external user is locked out, we reset the password via the eCR website.9. (Multiple Response Question, 1 points, 1 attempt permitted)Feedback when correct: The correct answers are: Email address, name, and policy number.Direct supervisor name and fax number are not required or necessary.Feedback when incorrect: The correct answers are: Email address, name, and policy number.Direct supervisor name and fax number are not required or necessary.11. Which is the correct email address of the eCR Help Desk? (Multiple Choice Question, 1 points, 1 attempt permitted)CorrectChoice custrept_eCR@.Xcustrept_help@. eCRHelptDesk@. custhelp_rept@.Feedback when correct: The correct answer is: custrept_help@.This is the email address you give to internal callers when:-You are unable to reset his/her password.-He/she is having further issues with eCR.The other email addresses are fictitious.Feedback when incorrect: The correct answer is: custrept_help@.This is the email address you give to internal callers when:-You are unable to reset his/her password.-He/she is having further issues with eCR.The other email addresses are fictitious.11. (Word Bank Question, 1 points, 1 attempt permitted)Feedback when correct: The correct answer is: External caller with eCR technical issues.Reset an external caller's password via the eCR website.Reset an internal caller's password via the eCR Admin Site.Provide the eCR Help Desk email address to internal callers with eCR technical issues (they can write their own email).Feedback when incorrect: The correct answer is: External caller with eCR technical issues.Reset an external caller's password via the eCR website.Reset an internal caller's password via the eCR Admin Site.Provide the eCR Help Desk email address to internal callers with eCR technical issues (they can write their own email).Appendix DThe following is the Level 3 (Behavior) evaluation interview form I used with the supervisor of each trainee of the eCR course.Ericson Employer and Consumer Services CompanyLevel 3 Behavior Interview/Performance Skills AssessmentReview InformationYour Name: [Your Name]Date: [Date]Review Period: October 1, 2012 to April 1, 2013GuidelinesComplete this review, using the provided scales.The middle figures are our standards/goals for this evaluation.Repeat the review annually and compare your yearly scores.Quality(5) = 99%(4) = 97%(3) = 95%(2) = 93%(1) = Below 93%Processes eCR according to SOP?????Listens effectively to callers?????Communicates effectively (overall)?????Verbal communications?????Written communications such as reports?????Email etiquette?????Telephone etiquette?????Customer Satisfaction(5) = Exceptional(4) = Exceeds Requirements(3) = Meets Requirements(2) = Gets By(1) = Needs WorkStrong customer advocate?????Sets aside personal biases and wants?????Gives good, practical guidance?????Fosters loyalty in customers?????Productivity (compared to standard)(5) = Exceptional(4) = Exceeds Requirements(3) = Meets Requirements(2) = Gets By(1) = Needs WorkAverage call handle time?????Average calls/hour?????Project TAT completion rate?????Appendix EThe following are the meeting minutes of the executive summary meeting I held for stakeholders, design and training:Level 1 evaluation survey results indicated to us that trainees were pleased with the objectives of the course matching the content, the relevancy to their job, and the way the materials were written overall (engaging and grammatically correct). Trainees did however report lower scores for sufficient opportunities for practice. We will look into creating simulations and/or alternatives to enhance practice for this course if our budget will allow.Level 2 evaluation results were astounding. The average score of the class was a 99%! The knowledge gained due to training was a huge success. We think we attribute such success to multiple knowledge checks along the way and to an excellent design and training staff. Our hiring staff also deserves recognition due to carefully selecting excellent prospects who have delivered excellent results.Level 3 evaluation results yielded pleasing statistics providing evidence that trainees truly had performed at a level that was trained and applied the objectives well after 6 months. Their ability to follow the standard operating procedures and use eCR efficiently provided excellent evidence that the training was a success.Level 4 results yielded slightly below the desired results, but still an excellent improvement when compared to the previous mCR statistics. Quality had raised from 88% to 93%, 2% shy of the goal, but still a great improvement. Further planning and investigation will be held in order to continue improvements on this aspect. Customer satisfaction has greatly increased (64% to 88%) primarily (we think) because our callers are having to wait for problem resolution nearly half as long as they previously were with mCR (an average of 50.4 business hours vs. 96 business hours). This is a most pleasing result as it decreases strain, lowers FTE requirements, and keeps us a competitive option for our customers. Lastly our average calls per hour goal was met as our staff now handles an average of 9 calls/hour vs. 6 (mCR). Based on these results, it has been decided to retire mCR completely in the archives. ECR will be the primary/sole operating system from this point forward. We are able to eliminate the need for half of the contracted temporary staff (for now) which is still a savings of $156,000 year after year. We will carefully consider future decisions as we test the waters and see how we fare. As the new full time staff become seasoned, and our training program gets shaped and molded into a more successful version, we will consider further future business solutions. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download