Program Evaluation Rubrics



Program Evaluation RubricsKrista JacksonAET/570May 9, 2016Professor DlabachProgram Evaluation Rubrics for StarbucksLevel 1 Rubric: Reaction EvaluationExcellentModeratePoorDesign of the Training:Training Objectives The objectives were stated concisely and clearly for the training on how to make the coffee drinks. The objectives met every need to understand what the training was covering about how to make the drinks perfectly. The learners felt positive with the knowledge about making coffee that was attained through the training objectives.The objectives were communicated through the training, but they were not as clear as they could be about the objectives. The objectives helped to understand how to make the coffee, but they did not go in depth enough for the learners. It seemed effective about the knowledge of the objectives of making coffee, but the learners were still unsure about all of the objectives.The objectives were not stated clearly or at all during the training on learning how to make the coffee drinks. The missing objectives made the introduction into the training difficult and almost impossible. The learners felt troubled and unsure about the objectives going into training.Pace of TrainingThe pace of the training fits because it was learning at the learner's own pace training. The training had a specific date to be finished, but there was leniency because each barista needed to be ready when they stepped up to the bar to make drinks. The learners felt very comfortable with no stress on the pace, which brought incredible morale.The pace of the training was learning at the learners own pace, but there was too little or too much concern on the ending date. The pace helped with learning the drinks, but more time and concentration could have helped. The learners felt content on the pace of the training, but they could have learned more before stepping onto the bar.The pace of the training was learning at the learners own pace, but this rule was not followed often. The training had a specific date, but it was not followed, so it brought on more stress on everyone in the environment. The learners did not feel comfortable stepping foot into the bar area to make drinks.Difficulty Level During TrainingThe level of difficulty was challenging, but invigorating and motivating during the training. The information was a lot to take in, but learning at the learners own pace helped, and the information stuck better than most training. The learners were pleased with the difficulty level because it was a fun challenge.The level of difficulty was challenging, and it was hard to process some of the information. Learning all of the drinks was a lot of information, and it was harder to remember all of the information. The information was in the learner’s brain, but it was difficult to remember because of the difficulty level. The learners were sufficed with the difficulty level, but they needed something a bit simpler for beginning training.The level of difficulty was too high, and most of it was hard to understand how to make the drinks to their standards. Learning all of the drinks was almost impossible, and remembering the drinks was too difficult. The learners were discouraged and dissatisfied with the results in the end.Evaluation of the Instructor:Instructor’s PerformanceThe instructor was informative, and they let learners learn on their own. The instructor answered any question immediately with a correct and informative answer about the knowledge of the coffee drinks. The instructors would visually show the learners at the bar any doubts or questions the learners had about any coffee information. The learners felt completely comfortable and captivated with the instructor.The instructor was helpful, but they were not always around to assist the learners. The instructor answered the learners’ questions when the learners could find them, and the instructor had time for the learners training to be a barista. There was some helpful feedback when the instructor was not distracted, so the learners felt like they were left in the dark sometimes. The learners felt uneasy but content with the instructor.The instructor was not helpful during the learners training. The instructors were always missing when they had questions, so the learners had to resort to other employees who were taking care of customers. There was no feedback on how well the learner was doing with their coffee knowledge. The learners felt hopeless and scared of the future working at the store because of my instructor.Instructor’s KnowledgeThe instructor knew everything there was to know about how to make coffee to how to run the store. The instructor knew how to teach the subject of making each drink in depth. They could answer any questions without looking up the answer about the coffee drinks, but they could show the learner where the answer was located. The instructor knew easy ways of remembering the drinks, and these tips were not located in the training manuals. The instructor made the learners feel like they were learning information that they would use forever. The instructor’s knowledge helped the learners be the employees that they are, and the instructors will always be remembered for their knowledge and training skills.The instructor knew the information, but not to the specific level that the learner was supposed to be learning as a new barista. The instructor had to look up questions because they were forgetful or did not often train, which made them rusty with knowledge. The instructor did not want to get information wrong, so the instructor's research helped themselves, but they should have the knowledge pouring out of their brains. They pawned the learners off on other employees because the instructor was having trouble with a topic. The instructor made the learner feel self-conscious and unsure to begin working at the bar making drinks. The instructor's knowledge helped, but it was not enough to be considered to be a trainer.The instructor barely knew any information and was not knowledgeable at all. They could not answer a specific question, so they would tell the learner to look it up in the manual. The manual was large, but contained at least fifteen smaller booklets, and it was not easy to look up the answer. The instructor’s knowledge just made learning more difficult, or the learner learned the wrong information. The instructor made the learner feel like they did not learn anything, especially from the instructor. The instructor’s knowledge was extremely lacking and shouldn’t have been in a training position.Instructor’s Time ManagementThe instructor knew how to manage their time well by multitasking; the instructor would train the learners and do their own work in their spare time. The instructor was always working, but they were always available for questions. They knew how to make sure that they were not giving the learners enough attention, and they would check in on the learner when the learner was doing their individual work. The instructor even had time for a sit down to take the learner’s mind off of the coffee material for a moment. The instructor had their time management down perfectly, and the learner never felt like they were being ignored.The instructor knew how to manage their time, but they were not good at multitasking. It was difficult for the instructor to train and do their work at the same time. They were consistent with their work, but there were times when they were not doing anything. They did not check in on the learners during their spare time, or try to start a conversation with the learner. The instructor needed better time management skills, but they were still efficient enough to be helpful during the learner’s training as a barista.The instructor did not know how to manage their time well, and they did not like training and having their own work at the same time. They would be negative about the situation, or make the learner feel in the way when they were supposed to be training the learner. They did not check in on the learner, start a conversation, or offer any help unless the learner asked. When the learner did ask it would take forever for the instructor to get over to help, and someone else had already helped the learner by then. The instructor had no time management skills, and they did not help with the learners’ training at all.Instructor’s Feedback and Answers to QuestionsThe instructor offered feedback every chance they got because they knew it boosted morale and self-esteem in a new employee. The instructor gave useful feedback, and it was not all negative if the learner had things to work on, so the instructor used the sandwich method to make the learner feel comfortable. The instructor was always willing and around to answer the learners' questions and they were always very informative with their answers. No question went unanswered, and the instructor would add in extra information to help the learner. The instructor assisted the learner in any way that they needed during the training. The instructor was attentive and always offered answers and feedback, and this made the learner feel lively and passionate about learning about the coffee material.The instructor offered little feedback, and it was almost forced out of them, or there was not very much feedback. It did not help with motivation and enthusiasm for the knowledge. The offered answers to questions but they were not always right, or it took forever to get a question answered. They were not used to giving feedback, so their feedback was what I was doing wrong. Answering questions were overlooked or sometimes ignored because of what they were doing and then they would forget. The instructor did offer some feedback and informative answers that were correct sometimes, but they did not make the learner feel confident.The instructor did not offer any positive or negative feedback, and it made the learner feel like they were invisible while they were training. The instructor did not answer questions to help with learning the coffee material. The learner did not feel confident in the job, training or material because of the instructors’ actions.Evaluation of the Training Exercises:Understanding Concepts & Their ApplicationThe concepts were in the booklets to the learner’s manual, and it was very informative on the details of coffee and making coffee. The manuals explained and showed how to apply the concepts when the learner was behind the bar. The instructor would show the learner how to apply them in a hands-on environment for further knowledge on how to apply the concepts. The learner had a full understanding of the concepts, and how and when to apply them from the manuals and instructor.The booklets in the manual explained some of the concepts, at least enough information to get by behind the bar. It showed how to apply them in an unclear manner, but sometimes it was easy to understand. The instructor did not do much hands-on training to show how the concepts apply in person. The manual gave the learners a vague general understanding of the concepts and how to apply them.The booklets in the manual did not explain the concepts, so it was hard to understand anything. Since, the concepts were absent, then how to apply the concepts was not in the manual. The instructor did not do any hands-on training for learning how to apply the concepts. So, the concepts and how to apply them were not learned.Time to PracticeThere was always time to practice when the learner was at work training. If the learner wanted to practice, and there was a rush, then they would go back to their booklets to the manual about coffee. After a bit of training, the instructor would have time for the learner to practice on the bar with real customer's drinks. The slower times were the best times to practice with customer's drinks because there is less stress with the store not being busy. The whole idea of training was to practice, so there was never time lacking to exercise hands-on.There was time to practice when the learner was training at work, but it was only during certain times, and it was not always convenient timing during training. During a rush sometimes there was nowhere for the learner to work on their booklets, so time to practice in any way was hindered. The instructor did not let the learner on the bar make customer's drinks until the learner was almost done with their booklets. The learner did get chances to practice, but it depended on the day and trainer for the day.There was never any time to practice when the learner was training at work. There was not a proper area to practice the material, which affected the training and learning as a new learner to coffee and making coffee. During a rush, there was no time to practice, and the learner could not practice on customer's drinks. There was no hands-on experience during the training.Exercises for the ConceptsThe instructor for the day would give the learner their manual with their booklets in it. The booklets had information; guidance, practice questions, quizzes, a hands-on exercise to do behind the bar, and photos to go along with the information. The booklets were interactive, and the learner learned at their own pace with the booklets. The learners got to learn how to make every single drink on the menu, the cash register, the pastries, different coffees from different regions, and cleaning information. The learners had to have the instructor over each time they started something new, and then the learner was allowed to keep practicing on the bar. The instructor would check the learners' drinks to make sure they were correct; the instructor would ask the learners questions or give them easy tips on remembering the material. The booklets gave the learners experience in many ways because it was not all just reading. The quizzes and other answer places were all done in pencil, so the learners could go back on topics they had trouble with at that time. These different exercises for the learner's mind and practice making the coffee drinks helped exponentially.The instructor for the day would give the learner their manual with their booklets in it. The booklets had information about coffee, and exercises as quizzes to be graded, and they had to be done in pen ink. The exercises outside of the booklet were just the instructor showing the learner step-by-step how to do something. The learner would go into the training with no experience, and the instructors were not always watching, so the learners did not get as much out of the exercises as they could have because the instructor had their own work too. The exercises helped, but there was not a variety of ways to take in the material. It was not as in depth, so when the learner got to the bar to make drinks they did not feel confident.The instructor for the day would give the learner their manual with their booklets in it. The booklets were just filled with text with no photos, quizzes, or practice in them. The only exercise was reading the text, and then the learner went straight to the bar with no experience.How the Training is Relevant to Learners:Applying Knowledge at WorkAfter the learner’s training they were able to apply their knowledge at work. The learner was successful and knowledgeable from their training, so they had the ability to do anything in their work environment.After the learner’s training they were able to apply some knowledge at work. The learner was moderately trained, and their knowledge and ability were just enough to get by at work.After the learner’s training they were not able to apply any knowledge at work. The training was not relevant, and this made the learner unsuccessful at work.Applying Knowledge PersonallyThe knowledge that the learner acquired during training helped them at home when they were not working. It was relevant and helpful in their everyday life because some of the learners made coffee often. The knowledge helped the learners appreciate and learn even more about coffee.The knowledge the learner learned during the training helps them sometimes if they were home. It was helpful when the learner wanted to drink coffee for a change, but they did not learn anything more about coffee outside of training.The knowledge the learner learned during the training did not help them in their personal life. The learner did not drink coffee, so there was no need to make coffee, or apply any knowledge unless they were working.RecommendationsThe learner will recommend this type of training for all businesses and individuals. It had multiple learning styles, very informative, and the learner attained a large amount of knowledge from the training. The learner thinks everyone should be able to experience such a diverse way of training.The learner will probably recommend this training for others to try. The two different learning styles help learners in some ways, but it does not have a lot of diversity. The learner still thinks that people would benefit in some ways from the training.The learner would never recommend this training to anyone or any business. It only had one learning style that gives the learner no experience. The training was not knowledgeable enough, and it is a waste of time even to talk about it.Implementation and Logistics:Layout of the Training EnvironmentThe training was very laid back, and the instructor wanted everyone comfortable during training. The learner was able to sit anywhere in the lobby, at any chair, table, or comfy chair with a coffee drink, while they worked on their booklets in their manual. The layout of behind the bar was relatively roomy, and enough room to get practice in while still getting customer's drinks out. For both training environments, everyone has room, including the learner while they were training to make coffee.The training for the manuals is done on a specific table by the bathrooms unless the table is taken. If the table is taken, then there is another specified table. Behind the bar for practicing drinks is not roomy, so if someone is clumsy, then they may have trouble. The training environments are stricter for the learners training, and not as comfortable as they could be for training.The training in the booklets for the manual is done in the back room sitting on a fold out chair. The chair has to be away from everything the employees may need access to, and there is no table to set the booklets on for training. Behind the bar is crowded because of all of the rushes, and there is no room for practice, so then both environments are uncomfortable for the learners.Lighting and VisualsLighting and visuals depend on the individual, but the lighting was a little too bright in the store unless they put down the shades. The instructors always let the learners put the shades down, and everyone saw an improvement, and they started being down during the whole day. The visuals of the drinks, names, and prices were all easy to see for learners, so it was easy to refresh on the menu.Lighting was bright most of the time because they did not put down the blinds very often during the day. The learner had to find the right spot to work on their manual, so the sun was not shining on them. The visuals of the drinks, names, and prices were relatively easy to see, but the font needed to be bigger to read the menu.The lighting was always bright with the sun and florescent lights, and this is not a comfortable environment for customers or learners. The lights distracted from the booklets for the manual, and the sun or bright lights were always in the learners’ eyes while making coffee. The environment was uncomfortable for customers and the learners.BreaksBreaks during training were the same as if you were a regular worker on the floor. The breaks were on the schedule, organized, and printed out in a notebook. The breaks depended on how many hours the learner worked, and the company gave more breaks than Texas gives normally. Instructors always made sure that everyone got every break, even if there were a lot of rushes in a row. During training, the learner could clock out if they wanted to, and they could go on a walk to stretch, and then come back and clock back in because the instructors were concerned about the learners being comfortable. Breaks were always given and respected by everyone.Breaks during training were the same as a regular employee working on the floor. The breaks were shown on the schedule, but they did not always get to them. During training, the instructor forgot to explain breaks to the learner, so they had to ask after training for a few days. Breaks were not as paid attention to because employees were overlooked.There was no time for anyone to have breaks; and even in training and working on the manual, the learner was not able to take any breaks. The management did not find breaks important, and the instructors were more concerned with how high the labor was because they wanted to let people off early. Letting them off early and taking no breaks affected the learners training because of the instructor and not getting a break.Length of TrainingThe training was at the learners own pace, so the length of training was excellent. The learner still got their breaks each day they went into work for training to be a barista. The length of training was necessary for all of the information, so the learner was comfortable when they were behind the bar making drinks. The training length depended on how many days, hours, and how much training the learner got done each day. The training length made the learner confident when they walked behind the bar.The training is fast pace, and the learner did not get as much time to spend on the booklets and practicing as they needed. The length of training is set by standards of percentages of how quickly people did their training. The learner started having trouble reading, and they could not miss their deadline for the length of their training. The learner was unsure when they stepped behind the bar to make drinks for the first time.The training length is short, so the learner did not get the training they needed to be a great barista. The instructor did not pay attention to training because they wanted everyone working behind the bar waiting on customers and making drinks. The manuals were not paid attention to or used, and there was no formal training. The learner felt lost and confused for months every time they stepped behind the bar because there was not much training at all.Senses (seeing, tasting, smelling, touch, and hearing)All of the senses are used in a coffee shop. The learner could see the menu, bar, and everything they had to use. The learner could taste what they smelled, and they would do coffee tastings during training. The smell was wonderful every time they walked inside, and it was also useful during the coffee tastings. Touch during training is essential because the learner needed hands-on training for the machines. During the training with the instructor the learner could always hear them, or they were both in the back room, which is completely quiet and good for short discussions.All of the senses are use in a coffee shop, but not every instructor focuses on these for the environment. The learner did not have to have perfect vision because they did not care. The instructors did rare coffee tastings, so smell and taste were not used as much. Touch was used during the short training, and then after because the learner had to touch the cups and machines to do a drink. The hearing was not always focused on in training, so the learner missed out on some information, and the instructor did not want to repeat themselves. The senses are just not concentrated on completely.All of the senses are experienced in a coffee shop, but during training, none of them were used. The learner did not have to see the menu or anything, but they needed touch to use the machines. Tasting and smelling did not matter because the instructor did not do coffee tastings while the learner was training. The hearing did not matter because the instructor was not talking or training. So, none of the senses were necessary for the learners training.Level 2 Rubric: Learning EvaluationExcellentModeratePoorFive Steps for Learning Evaluation (Coscarelli & Shrock, 2008, p. 4)Analysis – What is being tested? The effectiveness of the training program at Starbucks six to nine years ago. Two choices of analysis:1. Performance Tests – Observable performance on a rating scale, and these tests were used at Starbucks.The instructor sets aside a slow time at work to test the learner. There are two to three people observing the learner's behavior, who are experienced instructors, and are honest with no bias. The instructors watch the learner's performance during the test and rate their performance fairly. The observers are all professional and nice, so the test is not as stressful. The learner shows them what they learned in training on how to make the coffee drinks and other tasks.The instructor set aside a time without paying attention to busy times at the store. The instructor only has one other experienced instructor with them sometimes. Honesty and bias can go either way with the instructors analyzing the learner's performance. They are professional but show no emotions, and it makes the test stressful. The learner tests their performance for them, and the score can be fair or unfair scoring. When there are fewer instructors during the testing, then there can be a gap between the results scoring the learner.The instructor pulls the learner aside during a rush to do their performance test. The instructor is the only one analyzing the learner’s performance, so honesty and bias are big issues because it becomes more of a test of if the instructor likes the learner or not. The instructor is not professional because no one is watching their performance, so the entire test is stressful and uncomfortable, and it makes it highly likely that the learner will mess up from their nerves.2. Top of the Skills Hierarchy – tests high-level skills on the hierarchyThe instructor tests the learners on higher skills from the training they went through to make coffee drinks. The test is not about memorizing, so it focuses on how the learner can apply their knowledge to the job, and other skills higher on the hierarchy. The test shows the instructor that the learner understands more than just how to make the drinks. The test is helpful when the learner wants to show their competence in the training and knowledge they attained. It shows the instructor that the learner does not just know how to perform, but they are knowledgeable and able to use it in many areas. A coffee tasting and pairing will show the instructor that the learner can apply all of the material learned during the test. The scoring is accurate because the instructor is testing for the right material.The instructor tests the learner on higher skills from the training the trainers went through to make coffee. The instructor tests on memorization and a couple of higher skills. The test has nothing to do with memorization, but the instructor is not handling the testing correctly. The learner's competence can be shown with both ways of testing, but that is not the focus of this type of test, so the results and scores are not accurate for the test.The instructor does not test the learner on any higher skills they learned from the training. The instructor goes straight to memorization testing, which is a completely different type of test. The test and scores do not apply because the test was given in the wrong manner.3. Validity – Does the test measure what it intends to measure?The instructor tests the learner on the training; to see if they learned the information, and to make sure the training is effective and valid for the job. Experts look over the test first to legally claim that the test is valid. The test is for how much was learned, understood, and had knowledge of the material for making drinks and more.The instructor tests the learner on parts of the training, but missing some important parts. The instructor is trying to see if they learned the information, and to make sure the training the instructor uses is effective and valid. Experts are supposed to look over the test first to make it legally valid, but this test had not been made valid yet. The test did not do a great job showing how much was learned, understood, and had knowledge of the material. It was not a completely valid test, so it should not have been used, technically.The instructor tested the learner on random details or facts about what to do when they were at work. The instructor did not try to find out how much had been learned, so they could not tell if it was effective or valid. Experts are supposed to look over the test first to make it legally valid, but this test was never looked at as a test, let alone a valid test. The test did not show how much was learned, understood, and had knowledge of the material that the learner attained during training. The test was not valid, and it should not have been used, or even considered any type of testing. It was more of a conversation with random topics, which did not relate to the leaner’s training.4. Construction – How to write the test elements and details?Checklists – “breaks down the process or product into a series of binary choices, for example, “present” or “absent,” (Coscarelli & Shrock, 2008, p. 6). They are not scored like rubrics, which use words and number scales.The training has a checklist of points to cover in performance tests. The instructor follows the checklist to make sure everything is covered, and no information is missed on the checklist. The uses of binary choices are available instead of a number scale for the performance tests.The training has a checklist of a few points to use in the performance test, but the checklist does not cover very much information. The instructor may end up using a numerical scale to rate evaluations instead of the binary rating for performance tests. The change will make the performance test scores inconsistent, and not reliable for proof of performance.The training has a checklist with no information or a couple of points to cover for a performance test. It is not enough information for a performance test. The instructor does not use the binary rating for the performance test. The checklist is not even usable in this situation.“Multiple choice items above the memory level” (Coscarelli & Shrock, 2008, p. 6)The test is a multiple choice which the instructor will make higher cognitive level test. The test is more in depth than a memorization multiple choice test. The test has questions about the material that was presented, but the question will be asked differently. The learner has to use their higher cognitive thinking skills to connect the question with an example they have not experienced or covered in the training. The test does not have questions that are memory based because the questions are supposed to make the learners think in depth about each topic.The test is a multiple choice that is supposed to tap into the learner’s higher cognitive level. The test has mistakes where some of the questions are memory based questions. The questions do not use the right examples to connect to the information from the training. The learner does not have to think as in depth as they were lead to believe.The test is a multiple choice that does not go over the memory level of thinking. So, this makes the test easy for the learners, and they do not have to use cognitive skills to figure out examples for themselves. The test does not help the learners at work because if a problem occurs that they have not experienced, then they are not prepared to think cognitively at a higher level. The test is dumb downed because of the individuals’ lack of knowledge and understanding of the training.5. Standard Setting – “Establish a legally defensible cut-off or mastery score.” Three approaches to a score that is standard: Informed JudgmentThe instructor did their research to make an informed decision on what the standard score should be for mastery after training. The judgment can be backed up with references and individuals who helped the instructor make the informed judgment on the standard score. The methods they use give learners a more complete and accurate standard score.The instructor did some research, but is still unsure of what to make the standard score for mastery after training. They end up using their research and personal opinion for their decision. Their judgment can only be backed up with references they found, that created the standard score. The method they used does not give a completely accurate standard score for learners.The instructor uses themselves or nothing as an informed judgment on what they personally think the standard score should be for mastery after training. The judgment cannot be backed up on paper in any way. The method of choosing the score themselves gives the learners an inaccurate standard score for learners.ConjectureAt least two SME's look over the training and calculate the probability of an individual with basic competence can pass each part of the training for mastery level. By using a numeric scale, the SME's can judge what the average mastery score will be by debate and agreement. How in depth the mastery score level is attained makes it effective and legally valid.They had one to two SME’s looking over the training to calculate the probability of an individual with basic competence can pass each part of the training for mastery level. The bar is set lower than it should because there are not enough opinions. The numerical scale is used, but they are not as experience, so the master score is just a decision or agreement. The mastery score level can still make it effective but not as legally valid.There was only one SME in training looking over the training to calculate the probability of an individual with basic competence passing each part of the training for mastery level. There is set way too low for competence because there are only one person’s opinions. They do not pay attention to the numeric grading because of their lack of experience, so the mastery level score is just a decision by the one SME. The mastery level scoring is not effective, and it is definitely not legally valid.Contrasting GroupsThe instructor and SME's gather information together about similar and different tests and test scores. They contrast each of the groups to find the standard median score. The different examples and standard scores give all of them a better idea on how to change the mastery score for the training. The contrasting groups approach helps learners because it is fairer because it shows where their actual level should be after training.The instructor gathers a few examples of contrasting group standard scores, but does not do a lot of research for multiple groups to contrast. They need to find the standard median score, but with little information to contrast group scores, it is harder to come up with a more accurate conclusion. The few examples they attained will at least help to give them a different perspective and idea about changing the mastery score for training. The contrasting groups approach helps instructors and learners, but there is not enough information to make a concrete standard score after training.The instructor does not gather contrasting group standard scores because they think that they can choose what they want. There is no way to find the median score because of the lack of contrasting groups as examples and comparisons. The absence of examples makes it impossible to find the standard score of mastery level after training. The missing group examples to contrast makes the quoted standard score by the instructor is inapplicable.Level 3 Rubric: Evaluation/Behavior & Level 4 Rubric: ResultsNote for the beginning:First:- Discover to what extent training is being used or not being used.- Assuming there is some application comes the second.Second:- Then, dig into the reasons that training is being used or not used.- Sometimes, although not often, these reasons will lead one back to deficiencies in the training design or delivery.- In these cases, feedback to the L&D functions is provided, enabling leaders there to make the changes needed to resolve the problems.- More often, though, the reasons that training is being used (or not used) will lie in performance factors under the control of managers and others outside of the L&D function.Third:- Level 3 refers to gathering information on how (and how well) the skills, knowledge, and attitudes that were acquired in a training program are being used in workplace performance.- Evaluations = Behaviors = Application Fourth:- The design and delivery can be root causes in learners not using their training.ExcellentModeratePoorThe Method of Level 3 Evaluation: Observation (Direct and Covert)- Observation of learners working, after training, can tell the instructor a lot about the effectiveness because they can see if they are applying their new knowledge in their work.- The observation was the method that they used for the training because it was the fastest and easiest for management in a fast paced environment.Direct ObservationThe trainer would start out on the register, so they could watch the learners make the drinks, and see how well the learner was doing after the training. The instructor would put the learner in other positions on the job to test the whole effectiveness of the training. The learner would excel in all of the positions, and all of the knowledge attained during training would be visible to the instructor.The instructor would start out in any position, and have the learner in a random position that was convenient for the instructor. The instructor would test the learner in some positions to get an idea of the effectiveness of the training. The instructor is not looking focused completely on the results because they are distracted. The instructor does see some results during the observations.The instructor puts the learner anywhere and does not observe the learner. The instructor is too busy and finds the direct observation as a waste of time. The results they recognize are feedback from other employees, so the results are not valid for the effectiveness of the training.Covert ObservationThe trainer would have the shift leader during that shift observe the learner’s behavior because they can observe the learner without them knowing they are still being evaluated. The shift leader can report back on how the learner is doing. Having each shift leader do observations will give the instructor valid and unbiased evaluations of the training.The instructor would have the shift leader during that shift observe the learner’s behavior. The shift leader is supposed to report back on how the learner is doing, but they are not as attentive because they have their own work to do. Each shift leader does the observation of the learner, but the inconsistencies in how dedicated the shift leaders were during the observation make the evaluations harder to evaluate.The instructor would have the shift leader during that shift observe the learner’s behavior. The shift leader is supposed to report back on how the learner is doing, but they do not report back to them or claim they couldn’t do it. The shift leader does not observe the learner, and they do not give evaluations, so the information is not valid and usually biased.The Method of Level 3 & 4 Evaluation: SCM – Success Case Evaluation Method- A short survey of how and how much the learners are using the objectives during training.- Interviews a diverse number of learners that goes more in depth on:Level 3 – Behavior and behavioral changes that resulted from the training. All of the behavioral changes can be from design, delivery, instructor, and/or the learner.If the training is being used by the learnersThe training is being used by the learners because they have to use it to be effective workers. The learner’s behavior changes because the learners have to use the training to be able to work.The training is usually being used by the learners because they need it to be a good enough worker. The learner’s behavior changes slightly because the learners have to use the training to be able to work.The training is not being used by the learners because they are not interested in the work. The learner’s behavior may change, but it is not a positive change to assist them with their ability to perform work duties.How often the training is being used by the learnersThe learner is using all of the training during their whole shift. The learner cannot be an effective worker if they are not using all of the training. A new job makes the learner attentive to always using their training to make sure they are doing everything correctly. The only objective the learner needs is feedback.The learner is using the training often, but they skip many parts to the training during their shift. The learner does not take all of the training seriously and there can be many contributors. The new job makes the learner attentive to a certain extent on doing everything correctly. Feedback is needed when the learner is not using the training often.The learner is not using the training during their shift. The learner is not an effective worker because they are not concentrated on the training while working. The new job does not give the learner more motivation to make sure how often they are using training. Feedback needs to be given immediately to fix this issue.Exactly how the training is being usedTraining is being used exactly by the learner in everything that they do while they are working. The learner does each part of their work exactly like they were trained. Each position in the work environment is done exactly as the training instructed the learner to be successful with their job.The training is being used by the learner in their work, but they are not following everything exactly to the training. The learner is doing all of the parts of their work, but they alter some parts because of the issues that can present. Each position in the work environment is done almost like the training instructed the learner to be good with their job.The training is not being used by the learner in their own work. The learner is not following any of the training exactly because of factors that arise. Each position in the work environment is not done as the training instructed the learner to do.Difference the training madeThere is an exponential difference in the learner after the training is over. The learner can do each position in the work environment, and they are confident and motivated by the training they received. The training covered a lot of specific information, and the difference in the employee's work shows because they wanted to learn.There is a slight difference in the learner after the training is over. The learner can do each position in the work environment, but they are not as confident and motivated as they could be by training. The training covered specific information to make a difference, but the employee’s work does not show a big difference because of the lacking parts.There is no difference in the learner after the training is over. The learner cannot do any of the positions at work because they did not learn what they needed to during the training. The learners are not motivated or confident in their work, which leads to low morale and can be contagious to other workers.Adds value to the organization with proof of the changes from trainingThe learner adds value to the organization after the successful training. The learner’s ability to do everything correctly and their knowledge help the customers and the organization. If the learner knows how to do everything, then there is proof that they are helping the organization because it is better to have highly trained employees.The learner adds some value to the organization after the training. The learner’s ability to do some of the things correctly and their partial knowledge help the customers and the organization. If the learner knows how to do basics, then there is proof that they are helping the organization in some ways because they are trained enough for working on the floor.The learner does not add value to the organization after the training. The learner’s lack of ability to do everything correctly and lack of knowledge does not help the customers or the organization. If the learner does not know how to do the basics, then there is no proof that they are helping the organization because they need to be retrained.Justify the training by showing behavior resultsThe learner's behavior justifies the training because the learner has the knowledge, skills, and ability to do all of the concepts presented in the training for the work environment. The behavior does justify the training and shows proof by the learner, who has changed their aspect on training and learning.The learner’s behavior justifies the training because the learner has some knowledge, skill, and ability to do some of the concepts presented in the training for the work environment. The behavior does not justify the training as well as it could.The learner’s behavior does not justify the training because the learner does not have the knowledge, skills, or ability to do the concepts presented in the training for the work environment. The behavior presented does not justify the training at all.Level 4 – Results EvaluationExcellentModeratePoorLevel 4: Results Evaluation – shows the results that were attained for the business from the training.Operations:Revenue per EmployeeThe revenue per employee depends on how long the employee has been with the company, their performance, and their title at work. Learner start as baristas and their revenue would most likely be minimum wage. They could start off with a little more if they have an impressive work history or higher title status at a previous job. The higher pay helps employees, but it can have an effect on the organization's revenue.The revenue per employee depends on their performance, which is not always paid attention to, or their title at their work. All learners start as baristas, and their revenue is always minimum wage unless they ask for more and give a good reason. The wages helps the employees and the organization because the revenue for employee varies too much.The revenue per employee depends on how much the manager wants to pay the workers. All learners start as baristas, and their revenue is always minimum wage. The wages helps the organization's revenue but not the employee's revenue.Time to DeliveryThe learners can positively affect the time to delivery because they are still new to the training and knowledge. Successful training will help the store have a wonderful time to delivery to the customers because the learners are confident and fast. The learners know what they are doing, so time is never an issue to deal with because of the training.The learners can affect the time to delivery because they are still new to the training and knowledge. Subpar training will help the store have an average time to delivery to the customers because the learners are not as confident and motivated as they could be after training. The learners know what they are doing, but they are hesitant and the time can vary because of the training.The learners can affect the time to delivery negatively because they are still new to the training and knowledge. Unsuccessful training will not help the store have a good time to delivery to the customers. The learners are not confident or motivated in the work or timing. The learner does not know what they are doing, so timing is always an issue to deal with because of the lack of training.ReturnsThe products that were not drinks were always bought and never returned. The customer was knowledgeable after the learner fully explained the product. The customer knew exactly what they were buying, and how it was what they were looking for to buy.The products that were not drinks were bought and sometimes returned. The customer has imparted some knowledge that the learner had of the product. The customer had an idea of what they were buying, and how it might be what they were looking for to buy.The products that were not drinks were bought and always returned. The customer did not receive any knowledge from the learner. The learner did not have knowledge of the product, so the customer was confused on what they were buying when they left the store. The product was not what the customer was looking for, and they end up returning the product.Error and Error RatesThe learners go through the training, and if they are successful in developing the knowledge, skills, and abilities for work then the error rates are minimal. They are less likely to make errors because of their training.The learner go through the training, and they do an average level with developing knowledge, skills, and abilities for work, then the error rates will be more frequent. The learners are more likely to make errors when they did not get the entire training.The learners go through the training, and they do not do well with developing their knowledge, skill, and abilities for work. The error rates are constantly high because they did not retain the training.Productivity:Number of Correct DrinksThe learners make all of the drinks correctly at about 98% because of their training. The successful training helps the flow of correct drinks and does not mess up the flow because they have to go back to fix drinks.The learners make the drinks correctly most of the time at about 70% because of their training. The average training helps the flow of drinks until every drink they do not make correctly leaves the customer waiting longer. The incorrect drinks mess up the flow because they have to go back and fix all of the drinks.The learners make the drink incorrectly most of the time at about 20% because of their training.Response Time to CustomersLearners should have learned about greeting each customer as they walk in the door, so the learners use this for every customer. The learners should immediately be able to assist a customer, and multitask if they are in the middle of something. When the response time is immediate, then the customers get an amazing feeling walking into a positive environment.Learners should have learned about greeting each customer as they walk in the door, but the learners cannot always achieve this for every customer. The learners get to the customer in a relatively timely manner, but they do not get to the customers immediately. The quicker the learner gets to responding to the customer; then the more satisfied the customer will be towards the organization.Learners should have learned about greeting each customer as they walk in the door, but the learners cannot get to the customers in time or do not care. The learners get to the customers in an untimely manner, and they never get to a customer immediately. They do not seem to care about their reputation or the organizations.Timing on How Long the Learner Takes to Make the DrinkThe learner is confident, motivated, experienced from practice, and fast at making drinks after all of the training.The learner is feeling nervous, but they have the ability from the practice to make drinks. The learner is too careful, and the drinks do not get out to customers very fast.The learner feels helpless, lacking motivation, and lacking experience from the training. They are not fast on their timing to make a drink.Adds Value to the Organization with Proof of the Changes from TrainingThe learner adds value to the organization after the successful training. The learner’s ability to do everything correctly and their knowledge help the customers and the organization. If the learner knows how to do everything, then there is proof that they are helping the organization because it is better to have highly trained employees.The learner adds some value to the organization after the training. The learner’s ability to do some of the things correctly and their partial knowledge help the customers and the organization. If the learner knows how to do basics, then there is proof that they are helping the organization in some ways because they are trained enough for working on the floor.The learner does not add value to the organization after the training. The learner’s lack of ability to do everything correctly and lack of knowledge does not help the customers or the organization. If the learner does not know how to do the basics, then there is no proof that they are helping the organization because they need to be retrained.Justify the Training by Showing Behavior Results The learner's behavior justifies the training because the learner has the knowledge, skills, and ability to do all of the concepts presented in the training for the work environment. The behavior does justify the training and shows proof by the learner, who has changed their aspect on training and learning.The learner’s behavior justifies the training because the learner has some knowledge, skill, and ability to do some of the concepts presented in the training for the work environment. The behavior does not justify the training as well as it could.The learner’s behavior does not justify the training because the learner does not have the knowledge, skills, or ability to do the concepts presented in the training for the work environment. The behavior presented does not justify the training at all.Management:Employee Retention and SatisfactionThe learners stay longer because they are happy, and the management is kind, respectful, and helpful to the learners with the training. Employee’s retention is high because they are satisfied and have fun with the job. A positive environment, polite management, and being knowledgeable keeps the employees around longer.The learners are likely to stay after the training because they like the job and management. Employee’s retention is average because the employees start to lack motivation when they are not challenged. Employees get bored with the job especially when the management does not create an inviting environment.The learners are not likely to stay after the training because they do not like the job and management. Employee’s retention is low because employees are not happy, and only the management can fix this problem.Budget GoalsThe learner should know about the budget goals that management has during their training. The organization will have lower labor expense when people are paying attention to the number of people working; compared to how much sales are coming in during that period of the day. The learners will understand when they have to be let off work early, or if they have to help out during rushes. The management keeps the labor low for their budget.The learner should know about the budget goals that management has during their training. The organization wants the lower labor expense, but people do not always pay attention to the numbers. The workers do not care when they have slower times, and learners may be let off of work early. The management wants to keep the labor low for their budget, but this is not always attainable when they are not at the store all of the time.The learner should know about the budget goals that management has during their training, but they left out the budget goal and how labor is a big factor. The learners and workers do not pay attention or address high labor. The management does not address high labor either, so it does not help their budget goal.Absenteeism, Tardiness, and TurnoverThe learners are dedicated to their work, so they do not call in, and they are never late. The management pays attention to these numbers and people so that they can give positive feedback to the learners. When learners and employees want to go to work on time and do not want to quit, then it has a lot to contribute to the management and their positive environment.The learners are interested in their work, but sometimes they are late or call into work. The turnover can be a problem when learners are not motivated. The management does not offer as much feedback or positive environment.The learners show no interest in their work, so they are late and call in to work consistently. The turnover rate is bad because the management does not take care of their employees or look at them as assets to their organization. Management does not offer feedback, and they are usually negative at work.Financial:Operating Income GrowthThe gross income is growing, and the operating expenses are declining. The operating income growth helps the company expand, come out with new products, or update something in their stores. The employees can have a lot to do within the gross income growing, and that is why companies keep loyal, motivated, and dedicated employees.The gross income is growing, or the operating expenses are declining, but it is not both situations at once. The operating income growth can help with changes, but this can be inconsistent with how employees are doing throughout the company. The organization has to find ways to keep their employees happy to keep up operating income.The gross income may be growing, or the operating expenses are declining. The operating income growth does not help the company a lot because the growth is due to new employees. The employees who have been with the company for a while are not motivated or dedicated because of many factors. The operating growth will fluctuate because of employees.Budget VarianceThe budget variance can be positive because the budget has extra money than the actual calculated amount. The management and employees can help by going under the budget that their store has that year.The budget variance can be about even because the budgeted money and the actual calculated amount are about even. The management and employees can positively or negatively affect the budget variance. It all depends on how well they do with their budget for the year at their store.The budget variance can be negative because the budget has less money than the actual calculated amount. The management and employees can affect the budget variance because if they go over their budget, it will affect the variance negatively.Inventory TurnoverThe store can have a high inventory turnover because their sales were higher than usual. The store needs to stock up on inventory when they have a high inventory turnover. Employees have a lot to do with increasing sales, and management will keep up with inventory to support the sales.The store can have an even inventory turnover because their sales are not higher or lower than usual. The store does not need to stock up on inventory because they have a turnover it will be minimal. Employees can increase or decrease the inventory turnover with their sales.The store can have a low inventory turnover because their sales were lower than usual. The store does not need to stock up on inventory because they have enough if sales are down. Employees have a lot to do with decreasing sales, and management will have to watch the inventory carefully.Customer: Customer AcquisitionThe employees have proficient customer service skills because they are trying to bring in a new regular customer. If the customer looks unfamiliar, then the employee should cater the individual, so the customer leaves with a drink or product that they love. The customers’ repeat visits will show that the customer service provided by the employee acquired or gained a new customer.The employees have limited customer service skills, but they are still trying to bring in new regular customers. The employee will help the customer with what they ask for, but they do not go above and beyond to make sure to keep the customer coming back to their store.The employees have no customer service skills, and they are not trying to bring in new regular customers. The employee will treat the new customer like any other person with bad customer service skills, which ensure the customer will not want to repeat their service.Customer ProfitabilityThe customer is helpful with the revenue of the store because each customer spends about the same amount of money each time they come in the store. Excellent customer service makes a normal customer, who normally buys one drink, but they start buying bags of coffee beans or pastries. Many customers will increase their drink order because they are picking up coffee for others as well. The customer service helps with customer profitability.The customer is sometimes helpful with the revenue of the store because each customer spends around the same amount. Regular customer service will be helpful sometimes, where a regular customer buys something extra every once in a while. The customer service helps a little with the customer profitability.The customer is not helpful with the revenue of the store because each customer spends different amounts and does not always come in regularly. Bad customer service will make a regular customer into a customer that does not come in as often because of fear of that employee and experience. Bad customer service does not help with customer profitability.Customer RetentionCustomers like to have a routine, especially if they are regular coffee drinkers, so customer service is important in retaining customers. Employees practicing good customer service and becoming friends, in a way, helps retain customers.Customers like to have a routine, especially if they are regular coffee drinkers, so customer service is important in trying to retain some customers. Employees try with their customer service, but it does not always retain customers because they get curious about other coffee shops. When customer service is exceptional, the customers do not wander away from being a customer.Customers like to have a routine, especially if they are regular coffee drinkers, so customer service is why the store cannot retain customers. Employees practice bad customer service, so customers wander around coffee shops until they find a better place to be a regular customer. The store does not retain regular customers because of their negative customer service skills.Customer SatisfactionEmployees make sure that the customer is satisfied before they leave the store. If they are not happy with their drink, then the employee makes sure it is perfect for them or offers a free drink. The employees always practice impressive customer service, and no customer leaves without a smile.Employees try to make sure the customer is content when they leave the store. If the customer is unhappy with a drink, then the employee will remake the drink, but they are not as apologetic about the situation because their customer service skills are lacking. The customer leaves bland because of the customer service skills.Employees do not try to make sure the customer is happy before they leave the store. The employees are not concerned if an employee complains about a drink. They ignore the situation, argue, or just give out a free drink coupon. The bad customer service skills do not make the customer satisfied at all.Employees:Problem ResolutionThe learners are taught how to resolve problems in the work environment or between each other during training. The employees fix the issues quickly without negativity, and everyone goes home happy.The learners are taught how to resolve problems in the work environment, but they do not always follow the training. People are ruled by their emotions, and they do not know how to fix issues in a calm, positive manner sometimes. The issues are resolved, but they are not always resolved in the best manner.The learners are taught how to resolve problems, but they do not follow the training on resolving problems. Everyone in the work environment always thinks they are right, so nothing ever gets resolved, and this brings a negative environment until a resolution, or it is forgotten.Employee SatisfactionEmployees are satisfied with their job and the people that they work with each day. The learners are satisfied with the training, coworkers, and management.Employees are content with their job and the people that they work with each day. The learners are content with the training and management, but they are not satisfied with everything.Employees are not satisfied with their job or the people that they work with each day. The learners are not satisfied with the training, coworkers, and management.OvertimeThe learners and employees will have the overtime if they make a request to the management. The management always tries to accommodate all requests. The employees may have to work overtime during a week, but the manager lets the employee know, and usually throws in an incentive (tangible or intangible). The employees go into overtime with a positive attitude.The learners and employees may have overtime if they make a request to the management. The management will sometimes try to help out with overtime, but sometimes they are too busy, or it is not in the budget. The employee may be scheduled overtime, but the manager does not always remember to warn employees, and make sure that they are comfortable with the overtime and offer no incentive.The learners and employees may not have overtime if they make a request to the management. The management does not allow overtime unless it is an emergency because of labor and budget goals. The employee may be scheduled overtime, but the manager does not discuss the change to the employees.Efficiency and ProductivityThe learners and employees are efficient and productive together because the learners are trying to be as good as the older employees. After the training, the employees will help out with motivation. Contests on quickness, correctness, and looks are practiced to increase efficiency and productivity.The learners and employees are efficient and productive together to a certain extent. After training the employees to help learners if they have a question, but they do not help as much with motivation.The learners and employees are not efficient and productive together because they lack concern on these topics. Work is work to them, so they do not work together to be more efficient and productive.Marketing and Sales:Revenue per EmployeeAll of the revenue each employee brings into the organization with marketing and sales helps. The employees are excited and want to help with sales, and they help learners with more knowledge about how to sell the products. The employee’s dedication to each bring in revenue with sales helps the budget and store.The revenue each employee brings into the organization with marketing and sales is minimal, but it helps. The employees try a little to gain revenue for the organization, but they are not as interested in babysitting learners. The budget and the store revenue do not affect the employees as much, so they do not always see the point.The revenue each employee brings into the organization with marketing and sales is null. The employees do not try marketing and sales because they do not get anything out of it. The lack of motivation and encouragement does not help the employees in this area.Market ShareThe market share is improved because of marketing and sales in the whole organization. The learners and employees increase the market share by getting creative with new marketing ideas. The employees increase the sales by offering something as small as paying for an extra shot in the customer’s coffee. The employee’s knowledge of different coffee beans and products available will help sales. The market share is increased when all of the employees are working together in marketing and sales.The market share is improved a little because of marketing and sales in the whole organization. The learners and employees sometimes come up with marketing ideas, but it is not a goal unless there is an incentive. The employees try to increase sales, but some employees are just not good at selling more than what the customer came in to buy. The sales can affect the market share of a company.The market share is not improved because of marketing and sales in the whole organization. The learners and employees do not involve themselves with marketing and extra sales. The employees need an incentive to try to bring up the company’s market share. The negative attitude will affect the company’s market share in a dramatically unfavorable way.Improved DeliveryThe learners and employees have improved delivery with marketing and sales. The management, training, work environment, practice, and contests help them work quicker together. Everyone takes a turn with learning or teaching each other because everyone has something to learn.The learners and employees have improved delivery with marketing and sales minimally. The employees and learners are not always interesting in marketing and sales, so improving delivery does not always happen.The learners and employees have not improved delivery with marketing and sales. The employees and learners do not see the point in making marketing and sales quicker, so the delivery is never improved.ReferencesCoscarelli, W., & Shrock, S. (2008). Chapter 29: Level 2: Learning-Five Essential Steps for Creating Your Tests and Two Cautionary Tales ASTD handbook for workplace learning professionals. Alexandria, VA: ASTD Press. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download