Running Head: CLASSWORKS SOFTWARE EVALUATION



Running Head: CLASSWORKS SOFTWARE EVALUATION

Evaluation of Classworks Software

Kathy Cobb

University of West Georgia

March 29, 2011

Author Note

Kathy Cobb, Department of Media and Instructional Technology

Prepared for Mona Haynes, Director of Instructional Technology

Correspondence concerning this evaluation should be addressed to Kathy Cobb,

E-mail: kcobb@commerce-city.k12.ga.us

Executive Summary

Background Information

During the 2005-2006 and 2006-2007 school years, Commerce Middle School did not achieve the criterion level on the Criterion-Referenced Competency Test (CRCT) and thus did not make Adequate Yearly Progress. The middle school was placed on the state’s “Needs Improvement” list. Commerce Middle was well below the state’s score on the test particularly in the math area. Also, the high special education population played a major role in low test scores. Faculty and administrators were well aware that some type of innovation had to be implemented to counteract the low test scores. To address this need for improvement, administration and faculty considered several new innovations to assist in improving student’s academic skills. In order to raise the standardized test scores several computer software programs to enhance student academic achievement were reviewed. In order to address these needs Classworks software program was the innovation chosen by the director of instructional technology, principals, board of education members, faculty, and parents.

Purpose

The purpose of this evaluation was too determined if the software is being used to its fullest capabilities and if it is providing the students with the appropriate individualized instruction needed to raise standardized test scores. The purpose was also to determine whether or not the teachers feel that they have received adequate training on the program and to receive the teachers input on how the program can be improved. The evaluation was both formative and summative.

Evaluation Questions

• Is the implementation of the software program showing an improvement in classroom averages and standardized test scores?

• How often is the software being incorporated into your curriculum?

• Is the teacher using the software to assist with student’s individualized instruction?

• How much training have you had in using the software and what suggestions do you have for improvement?

Methods

The participants for this evaluation were the academic teachers in Commerce Middle School who are required to incorporate Classworks into their curriculum. The participant’s ages range from 25 to 55. They were all certified teachers, who are teaching in their specific area. Educational background for the participants includes Bachelor’s to Specialist’s degrees. The students were not directly involved in the evaluation, but data regarding their Criterion-Referenced Competency Test (CRCT) results was used to complete this evaluation.

The evaluation began with a Likert Scale survey administered anonymously to the fifteen academic participating teachers. The data received through the surveys was qualitative. Also, two teachers were interviewed by the evaluator concerning their use of Classworks Software. The data received through the interviews was also qualitative. This data was analyzed for trends and commonalities among the participating teachers. The evaluator was looking for information regarding teacher’s perceptions about the use of the software in raising student’s test scores and their thoughts on if there has been adequate training on the software. This data was analyzed for implications, commonalities and trends.

Key Findings

The Commerce Middle School academic instructors who implement the Classworks Software into their curriculum generally reflect a neutral opinion about the program. Teachers do perceive an increased amount of time spent on planning and implementing the program. While the findings in this evaluation are encouraging and do show an increase in student CRCT scores, teachers question whether the increase is related to the implementation of Classworks software. Thirty-three percent of the teachers surveyed did not feel that Classworks has been instrumental in improving standardized test scores of the students. Forty percent of the teachers surveyed did not feel that Classworks has been instrumental in improving the class averages of the students. One of the teacher’s concerns about the program is that not all of the lessons are aligned with the Georgia Performance Standards. The majority of the teachers reflected that they had received adequate training on the software.

Recommendations and Conclusions

The research indicates that while there are some positive aspects of Classworks software, no assertions can be made with complete certainty without more information on the implementation of the program. It is recommended that more data needs to be collected in order to make a correlation between Classworks and increased test scores. Perceptions are generally neutral, but this data could be used as a baseline for future studies. However, it is important from the CRCT results that the school system continues to support the program.

Evaluation of Classworks Software

Introduction

Commerce City Middle School is nestled in northeast Georgia, approximately 20 miles from the University of Georgia, and one hour from the state capital. Commerce Middle School provides students with the advantages of a small town as well as access to educational and cultural opportunities in Athens and beyond in Atlanta. The school is part of the Commerce City School System and serves grades fifth through eighth. Commerce is a small town with a population of only 6,574 people. The estimated median household income in 2009 for Commerce was $36,200.00 (City-, 2009). The population of the student body at Commerce Middle School is 452. The student’s population is 76% Caucasian, 14% black, 7% Hispanic, and 3% other. Students with disabilities make up 15% of the school’s population. Forty eight percent of the student body is eligible for free lunch (Commerce Middle School, 2010).

During the 1990’s the state began an initiative to increase the academic level of all elementary level students. “Intended to improve academic achievement, Georgia began, during the 1999-2000 academic year to initiate educational reforms designed to increase the amount of academic content that elementary children must know and to assess this new baseline of student achievement with a statewide criterion based test called the Criterion Referenced Competency Test (CRCT),” (Livingston & Livingston, 2003). The CRCT was initiated and put into place to measure the success of the individual student’s progress in obtaining skills and knowledge described in the Georgia Performance Standards. During the 2005-2006 and 2006-2007 school years, Commerce Middle School did not achieve the criterion level on the Criterion-Referenced Competency Test (CRCT) and thus did not make Adequate Yearly Progress. The middle school was placed on the state’s “Needs Improvement” list. Commerce Middle was well below the state’s score on the test particularly in the math area. Also, the high special education population played a major role in low test scores. Faculty and administrators were well aware that some type of innovation had to be implemented to counteract the low test scores. To address this need for improvement, administration and faculty considered several new innovations to assist in improving student’s academic skills. In order to raise the standardized test scores several computer software programs to enhance student academic achievement were reviewed. Faculty and staff realized that web-based programs play an important part in enriching student’s achievements. “Web-base programs, such as games promote higher order learning outcomes such as increased meaningful dialogue among students and the identification of student misconceptions, both of which contributed to deeper student understanding (McDonald & Hannafin, p. 459). It was decided that a web-based program, which included critical thinking activities and was centered on individualized instruction was to be implemented into the curriculum at Commerce Middle School. “Students who perform web based activities are more motivated, have increased retention of information, and demonstrate improved reasoning skills and a greater level of higher-order thinking” (McDonald & Hannafin, p. 461). Commerce City Middle began the search for an innovative software program that would be easy to implement into the teacher’s curriculum.

In order to address these needs Classworks software program was the innovation chosen by the director of instructional technology, principals, board of education members, faculty, and parents. According to its 2009 website Classworks by Curriculum Advantage, Inc. is comprehensive, instructional software that gives students the edge to succeed. Dynamic, interactive lessons engage students and offer new ways to address difficult concepts (Classworks by curriculum, 2011). Since the state of Georgia launched the Georgia Performance Standards, schools have been working to enhance their instruction to help students achieve the rigorous expectations for student work. The Classworks instructional improvement system is built upon a rigorous, standards-aligned supplemental curriculum that includes both skill-level instruction and project-based extension activities The administration at Commerce Middle School was looking for a software program that would provide individualized instruction to the student and at the same time be aligned to the Georgia Performance Standards. Helping teachers find an easy ways to individualize instruction is critical in today’s classroom. Classworks uses the philosophy that not every student learns the same way or at the same pace. With the advanced technology of the Classworks solution, teachers are free to do what they do best – teach (Curriculum advantage inc…,2009). However, since the implementation of Classworks during the 2006-2007 school year there have been some difficulties with the teacher’s use of the program. During the first year of implementation many problems were encountered and had to be resolved. During 2006 Commerce Middle School struggled with the new instruction software and was very disappointed when the CRCT results came back with no improvements. During the following years Commerce Middle began seeing results that the software was raising student’s test scores, but the Director of Technology, Mona Haynes has not been sure that the teachers are actually utilizing the software to its full potential. The administrators of the school feel that teachers are not providing the student with the individualized instruction in which the Classworks software allows. There has been no follow up from the administration since the initial staff development of the Classworks software program over three years ago. I will be evaluating the integration of the Classworks software program for my client, the Director of Instructional Technology at Commerce City Schools. This evaluation will determine if teachers are using Classworks effectively in their instruction to increase the student’s standardized test scores and classroom averages. The active stakeholders of this evaluation include students, teachers, administrators, parents, and ultimately the community, city, state, and nation.

Purpose

As requested by the contractor, Mona Haynes, I will be utilizing formative evaluation to assess the teachers at Commerce Middle School who use Classworks. I will gather their thoughts and perceived effectiveness of the software program. The formative evaluation will be used to determine the benefits and effectiveness of the Classworks Software on student achievement. It will be determined if the software is being used to its fullest capabilities and if it is providing the students with the appropriate individualized instruction needed to raise standardized test scores. It should be determined through this evaluation as to whether or not the teachers feel that they have received adequate training on the program and to receive the teachers input on how the program can be improved.

Evaluation Questions

The goal of the project is to assist teachers in their use of the Classworks Software in order to meet the needs of the individualized student and to assist them in meeting standardized test scores and higher classroom averages. The evaluation plan described here focuses on formative measures of implementation to gather results of four major project objectives that lead to this goal. The objectives, framed as research questions are:

• Is the implementation of the software program showing an improvement in classroom averages and standardized test scores?

• How often is the software being incorporated into your curriculum?

• Is the teacher using the software to assist with student’s individualized instruction?

• How much training have you had in using the software and what suggestions do you have for improvement?

Methods

Participants

The participants were all the academic teachers in the middle school who are required to incorporate Classworks into their curriculum. There are fifteen academic teachers at Commerce Middle School from which data were gathered. All fifteen of the participants returned the questionnaire. However, only two participants participated in the interview process. The participant’s ages range from 25 to 55. They are all certified teachers, who are teaching in their specific area. Educational background for the participants includes Bachelor’s to Specialist’s degrees.

Design and Procedures

The evaluation consists of the following activities designed to address each of the four objectives:

• Strategies to evaluate objective 1: Formative evaluation will consist of a likert-type survey given to all the teachers who use the Classworks software, an interview with open-ended questions was performed with two teachers and statistical data on the CRCT scores from the last four years was gathered from the Georgia Department of Education’s website.

• Strategies to evaluate objective 2: Formative evaluation for this objective will use a likert-type survey ranging from “Never to Daily” to assess the perceived use of Classworks Software.

• Strategies to evaluate objective 3: Formative evaluation for this objective will use a likert-type survey for all teachers using the software and an open-ended interview of two teachers who use the software.

• Strategies to evaluate objective 4: Formative evaluation for this objective will use a likert-type survey ranging from “Strongly disagree to Strongly agree” to assess the amount of training and assistance the teachers have received in their use of Classworks. Also, selected teachers were asked open-ended questions during an interview concerning the challenges and amount of training received in using the software.

Instruments:

For this evaluation, a survey and interviews were used to determine the effectiveness of Classworks software. Each teacher who uses the software completed a ten questions survey about his/her Classworks Software use in the classroom. The teacher also answered questions concerning whether or not they believe that implementing Classworks is beneficial to the student. An optional section of the survey included a place for teachers to make additional comments about the software. Two teachers were interviewed and asked a series of open-ended questions concerning the effectiveness of the software and whether or not they believe that adequate training of the software was provided. The Department of Education’s statistical data was used to show an increase or decrease on student standardized test scores throughout the last four years.

Table 1. Evaluation Instruments in the Classworks Software Project

Project Interview Questions Survey Questions Data

Objectives Open-ended Likert-Type Results

1. Test Improvement X X X

2. Use X

3. Effectiveness X X X

4. Challenges &

Improvement X X

Summary of Key Findings

Fifteen teachers completed the questionnaire on Classworks Software. Teaching experience of the instructors is as follows:

0-3 years – 1; 4-10 years – 8; 11-15 years – 2; 16-20 years – 2; 21+years – 2.

Numbers of years at Commerce Middle School were as follows: 0-3 years – 3; 4-10 years – 10; 11-15 years – 1; 16-20 years – 1. All teachers surveyed either agreed or strongly agreed that they could navigate through the basic functions of a computer and operate basic software.

Evaluation Question One

Is the implementation of the software program showing an improvement in classroom averages and standardized test scores?

Key Findings for Question One

Two of the Likert Scale survey questions were analyzed for commonalities among the fifteen classroom teachers that responded to the survey. The two questions focused on the teacher’s perception on whether or not the implementation of Classworks Software has caused an increase in classroom averages and standardized test scores.

When analyzing the data of the following two questions it shows that there is a discrepancy in how teachers perceive the effects of the software in showing an increase in test scores and class averages. The majority of the teachers were neutral on the two questions addressing “evaluation one”.

The Classworks Software has been instrumental in the improvement of student’s standardized test scores at Commerce Middle School over the last three year (Strongly Disagree-0; Disagree-5; Neutral-6; Agree-4; Strongly Agree-0).

[pic]

Classworks software has been instrumental in improving the class average of my students (Strongly Disagree-0; Disagree-6; Neutral-7; Agree- 1; Strongly Agree-1).

Further analysis of this evaluation question through the interviews of the two teachers also gives us insight into the perceptions about the effects of increase in test scores and classroom averages through the use of Classworks Software. During the interview the following question was asked, “In your opinion, has Classworks software played an instrumental part in improving student achievement by assisting in the increase of CRCT scores at Commerce Middle School?” The first response received was, “yes, Classworks has proven valuable at identifying academic strengths and weaknesses that enable me to better customize my instruction. The second response received was, “no.” Also, this was the only question which received comments from the questionnaire survey. The comments were as follows:

“I think the Classworks software is a good resource for students who need remediation and reinforcement of skills already taught. However, with the fewer number of school days and so much material to cover, Classworks is not always the best use of my instructional time”.

“Takes too much time to find the activities and go through them. Some problems are too hard for my students and do not relate to standards. Can’t find all the standards either.”

“The classworks for Science is not the best, because it is not Georgia specific. There are a lot of units and lessons that cover information that we do not teach.”

“Classworks is as good as the user-namely achievement of the program relies namely on the motivation of the user. Likewise, although Classworks is supposed to be aligned to Georgia Performance Standards (GPS), there are often key terms that are presented contrary to GPS or in a manner students are not used to seeing. This often makes lower level students confused and frustrated when navigating through the program.”

“I find that many of the units are not based on the GPS. Many students find Classworks boring so they do not try to do well.”

“Classworks is a great resource when working on specific SKILLS (not content) with students. It is best for low students because it reviews the skills with them. You can set up individual lessons for the students to work on their specific weak areas or if you are unfamiliar with the students weak areas you can design snapshot skills assessments to find their weak areas and place them in the units they need to work on.

Evaluation Question Two

How often is the software being incorporated into your curriculum?

Key Findings for Question Two

This question was also addressed in the Likert Scale questions. An average of 67% of the teachers incorporate Classworks software into their curriculum one to five times per month with the rest of the instructors using it more frequently. It is apparent from this question that teachers are using the software quite frequently in their instruction.

The Classworks software is being incorporated into my curriculum (Never-0; 1-5 times per month-10; 1-5 times every two weeks-4; 1-5 times per week-1).

Evaluation Question Three

Is the teacher using the software to assist with student’s individualized instruction?

Key Findings for Question Three

This question which was addressed in the Likert Scale reveled that 59% of the teachers were neutral in their opinion of student individualized instruction. Twenty-seven percent of the teachers agreed that Classworks did provide a resource for individualized instruction.

Classworks software is an outstanding resource for providing student individualized instruction.

[pic]

Further analysis of this evaluation question occurred during the interview with the two teachers. When asked “Do you feel that the students are motivated and prosper individually from using a web-base program such as Classworks?” the following responses were received.

“Yes and no. Some students enjoy working on the web and playing video games, which is great for Classworks. However, students who do not enjoy the web or video games do not like it and put forth little effort”. Another response was, “the higher students seem to be more motivated in the programs. The weaker students are not. They struggle and need more one-on-one. One comment made by a teacher on the questionnaire survey addressed this evaluation question. “I love Classworks, because it allows me to see student mastery and student struggles. Classworks helps me see what areas need re-teaching and which ones do not. Our current Classworks is far superior to the first version. I love the ability to customize student’s activities with classroom lessons.”

Evaluation Question Four

How much training have you had in using the software and what suggestions do you have for improvement?

Key Findings for Question Four

Several of the Likert Scale survey questions were analyzed for commonalities among the fifteen academic teachers that responded to the survey. The questions focused on the teacher’s perceptions of whether or not they have received adequate training of Classworks and their opinions for improvement of the software. All of the teachers either agreed or strongly agreed that they had the capabilities of navigating through the basic functions of a computer and software. The majority of the teachers felt that they have received adequate training and technical assistance in using the Classworks software.

I have received adequate training on how to use Classworks software effectively in my classroom (Strongly disagree-0; Disagree-2; Neutral-1; Agree-7; Strongly Agree-5).

[pic]

I am proficient in using all of the components of the Classworks software program (Strongly Disagree-0; Disagree-1; Neutral-4; Agree-8; Strongly Agree-2).

[pic]

Technical assistance is always available if I need help with the Classworks program (Strongly Disagree-0; Disagree-2; Neutral-3; Agree-8; Strongly Agree-2).

[pic]

Further analysis of the evaluation question four gives more insight into the training and suggested improvement of Classworks. During the interview process both teachers stated that were very proficient in the use of the software and that they had been trained adequately. They both stated that if there were problems there was an individual at the school to assist them. When asked about suggestions for improvements of the software the following responses were received.

“The software needs to be more aligned with state standards. The games are not always user friendly. Some of the programs will not run. I have to go through the games to see which ones the students need (because it’s not totally aligned).

“None. After a few years, we finally “have it down.” It enables me to customize lessons which works out great! The only problems I have encountered is some repetitiveness of lessons and band width issues that cause disruptions in the program since it is web based. I would suggest that Classworks needs to make it available for all subject areas. If we had a mobile computer lab and be able to use it in our classroom, that would be a great instructional tool. Be able to create Benchmark Exams on Classworks, much like OAS.

When asked the interview question, “Would you recommend the district to purchase an alternative program to assist in raising student test scores?” the following responses were received.

“Yes, I would recommend everyone use TestPrepUSA in addition to Classworks. It gives students a greater range of questions and keeps them from being “bored” with the same program.”

It really depends on the subject. I would recommend Read Naturally for Reading and maybe IXL for math. For science, it would be nice to have more lab items for more hands on work.”

Additionally in evaluating the benefits of using Classworks Software to raise Standardized test the evaluator gathered statistical data on the CRCT scores from the last four years from the Georgia Department of Education’s website. The following charts will reflect some of the information gathered from the website.

Upon an initial look at the data provided with this report that the CRCT scores of the students at Commerce Middle School have sufficiently increased over the last four years as reflected in the above graphs.

Recommendations and Conclusions

This report in its entirety will be given to the Director of Instructional Technology to disseminate according to the needs of the stakeholders at Commerce Middle School. It is the recommendation that more research be done over a longer period of time on the Classworks Software Program with stakeholders who are fully involved in the implementation of the program. While the findings in this evaluation are encouraging and do show an increase in student CRCT scores, it is questionable whether the increase is related to the implementation of Classworks software. Several of the teachers who completed the survey questionnaire had numerous problems with the software. Thirty-three percent of the teachers surveyed did not feel that Classworks has been instrumental in improving standardized test scores of the students. Forty percent of the teachers surveyed did not feel that Classworks has been instrumental in improving the class averages of the students. One of the main comments implied by the teachers was that Classworks is not aligned with all of the Georgia Performance Standards, which causes difficulty when implementing it. Only two teachers felt that they had not been trained sufficiently on the program and that technical support is not always available. It is recommended that all teachers new to the district should receive basic technology training and a sequential course which introduces every component of Classworks over multiple days. Establish a system for teachers to receive quicker assistance with technical support at the district level for using the program, Classworks, when needed. Teachers are incorporating Classworks frequently into their curriculum and are mostly neutral in their overall opinion of the program. However, it is important from the CRCT results that the school system continues to support the program. The research indicates that while there are some positive aspects of Classworks software, no assertions can be made with complete certainty without more information on the implementation of the program. School administrators can now assess the troubled areas of the program and begin working to correct the areas of need to make Classworks Software Program a more effective program for all stakeholders involved.

References

City data. (2009). Retrieved from

Classworks by curriculum advantage. (2011). Retrieved from

Curriculum advantage inc.;Georgia districts implementing classworks instructional software report state test score gains. (2009) Education Letter, 16.

Jones, t. (2011). Commerce city schools. Retrieved from

Livingston, D.R., & Livingston, S. (2003). Unbraiding the rhetoric about student achievement and teacher quality in georgia. Association of Teacher Educators. Retrieved from eric. on January 20, 2011.

McDonald, K.K., & Hannafin, R. D. (2003). Using web-based computer games to meet the demands of today's high-stakes testing: a mixed method inquiry. Journal of Research on Technology in Education, 35(4), 459-472. 

Appendix A

Data Collection Organization Matrix

|Questions |Related Results |Level |Required |Data Source |Data Collection |Date |Primary |

| | |(Mega, |Data/Measurable | |Tools/Procedures |Collected |Responsibility |

| | |Macro, |Indicators | | | | |

| | |Micro) | | | | | |

|1-2 |Work |Macro |Teacher feedback on |Teachers |Questionnaire Survey |3/1/11 |Evaluator |

| |Experience | |survey | | | | |

|3-6 |Technology |Macro |Teacher feedback on |Teachers |Questionnaire Survey |3/1/11 |Evaluator |

| |Experience | |survey | | | | |

|7-10 |Student |Micro |Teacher feedback on |Teachers |Questionnaire |3/1/11 |Evaluator |

| |Performance | |survey | |Survey | | |

|1-3 |Student |Micro |Interview |Teachers |Questions on Interview |3/8/11 |Evaluator |

| |Performance | |feedback | | | | |

|4-10 |Technology |Macro |Interview |Teachers |Questions on interview |3/8/11 |Evaluator |

| |Experience | |feedback | | | | |

Appendix B

Classworks Software Questionnaire

1. Teaching Experience

0-3 years 4-10 years 11-15 years 16-20 years 21+ years

2. Number of years at Commerce Middle School

0-3 years 4-10 years 11-15 years 16-20 years 21+ years

3. I can navigate through the basic functions of a computer and operate basic software.

Strongly disagree Disagree Neutral Agree Strongly Agree

4. I have received adequate training on how to use Classworks software effectively in my classroom.

Strongly disagree Disagree Neutral Agree Strongly Agree

5. I am proficient in using all of the components of the Classworks software program.

Strongly disagree Disagree Neutral Agree Strongly Agree

6. Technical assistance is always available if I need help with the Classworks program.

Strongly disagree Disagree Neutral Agree Strongly Agree

7. The Classworks software has been instrumental in the improvement of student’s standardized test scores at Commerce Middle School over the last three years.

8. Classworks software has been instrumental in improving the class average of my students.

Strongly disagree Disagree Neutral Agree Strongly Agree

9. The Classworks software is being incorporated into my curriculum

Never 1-5 times per month 1-5 times every two weeks 1-5 times per week Daily

10. Classworks software is an outstanding resource for providing student individualized instruction.

Strongly disagree Disagree Neutral Agree Strongly Agree

Appendix C

Interview Questions

1. In your opinion, has Classworks software played an instrumental part in improving student achievement by assisting in the increase of CRCT scores at Commerce Middle School?

2. Would you recommend the district to purchase an alternative program to assist in raising student test scores? If yes, what program?

3. Do you feel that the student’s are motivated and prosper individually from using a web-base program, such as Classworks? Why or why not?

4. In your opinion, how proficient are you with using Classworks Software?

5. If you experience difficulty using Classworks Software, are you confident there is an individual at your school or at the district level to assist you?

6. What additional steps, if any have you taken on your own to promote your understanding of the program?

7. What additional steps would you recommend Commerce Middle School offer to support you while implementing Classworks into your curriculum?

8. What challenges have you encountered while using the program?

9. What suggestions do you have on the improvement of Classworks Software?

10. Do you have any other comments that you would like to provide about the use of Classworks Software in your instruction?

-----------------------

[pic]

[pic]

[pic]

[pic]

2010 CRCT ELA

Meets and Exceeds % Comparisons

[pic]

2010 CRCT Math

Meets and Exceeds % Comparisons

[pic]

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download