Evaluation Tool - Office of Educational Technology

Evaluation Tool

Examining Progress Towards Future Ready Professional Learning

U.S. DEPARTMENT OF EDUCATION

1

This report was prepared for the U.S. Department of Education under Contract No. ED-PEP-10-C-0059 with American Institutes for Research. The mention of trade names, commercial products, or organizations in this report does not imply endorsements by the U.S. government. This publication also contains URLs for information created and maintained by private organizations. This information is provided for the reader's convenience. The U.S. Department of Education is not responsible for controlling or guaranteeing the accuracy, relevance, timeliness, or completeness of this information. Further, the inclusion of information or URLs does not reflect the importance of the organization nor is it intended to endorse any views expressed or products or services offered.

U.S. Department of Education Arne Duncan Secretary

Office of Educational Technology Richard Culatta Director

November 2014

This report is in the public domain. Authorization to reproduce this report in whole or in part is granted.

This report is available on the Department's website at .

Requests for alternate format documents such as Braille or large print should be submitted to the Alternate Format Center by calling 202-260-0852 or by contacting the 504 coordinator via e-mail at om_eeos@.

Why Evaluate?

By this point in your use of the Future Ready Schools: Empowering Educators Through Professional Learning toolkit, your leadership team composed of district, school, and teacher leaders will have used your selfassessments to plan and begin to implement professional learning and collaboration activities intentionally aligned with your goals and with best practices for professional and connected learning. The next step is to collect, analyze, and share evidence of the actual value these activities are creating for your educators and students. In other words, it is time to evaluate the effectiveness of your work. The types of value that are being created include the immediate satisfaction participants experience, the knowledge they gain, the changes in practice they make through applying that knowledge, the impact of those changes on their effectiveness in improving student learning outcomes, and the changes made to beliefs and policies that are justified by such results. By identifying strategies using the continuum tools, you have already begun to consider evidence you might collect. This evaluation tool provides a framework for systematically working with such evidence. There are two primary reasons to evaluate the efficacy of professional and connected learning:

? To make evidence-based decisions about how to improve the activities you are implementing: What is working that you should do more of? What can be deemphasized in order to focus effort on more promising practices?

? To share effective practices and comparison of results: What difference has your investment in implementing new strategies made? Why have certain activities proven effective, and how could funders and other districts learn from and build on your successes?

Evaluating professional learning and collaboration activities that integrate learning through online communities and networks can be challenging in two ways:

1. Research shows that the types of value that these activities create are diverse and not always predictable (Office of Educational Technology, 2011a). We need to capture the different kinds of value that are being created and examine how they compare with what we intended. To what extent are we realizing our goals? What other kinds of value are being created that we did not anticipate but upon which we might build?

2. Many conventional educational research designs are not efficient enough to provide actionable information in a timely fashion at a sustainable cost. We need to put into place an evaluation process that is empirically grounded but also appropriate to the district's needs and assets.

This evaluation tool presents a process of self-evaluation that captures diverse types of value fairly rapidly, as part of the process of implementing professional learning and collaboration programming to increase student learning. The tool can provide a useful complement to more traditional evaluation approaches, which you might also wish to consider. This tool is part of the Future Ready Schools: Empowering Educators through Professional Learning toolkit and can be found at tech.FutureReady/Professional-Learning.

1

Instructions for Use

1. First, review this entire tool so that you become familiar with the possibilities for use of the evaluation framework and the following: ? Five types of value created through participation in professional learning activities ? Three kinds of evidence--indicators, stories, and artifacts--that can document the value of the professional learning ? Many ways that you can collect, measure, analyze, and share evidence of the value of the professional learning

2. Use the Indicators Worksheet on page 8 to choose your indicators and determine how you will gather the necessary data to track them.

3. Consider how you will make use of value creation stories, pages 9?10, and artifacts, page 11, in relation to the indicators.

4. Use the Portfolio Planning Worksheet on page 14 to help you determine what you want your portfolio to communicate and which audiences you want it to address.

5. Consider how you can make ongoing use of the toolkit to: ? Make evidence-based decisions about how to improve the professional learning activities you are implementing ? Share effective professional learning strategies and comparison of results

2

Evaluation Framework

This tool introduces you to a simplified version of the evaluation framework developed by Wenger, Trayner, and de Laat (2011), building upon Kirkpatrick (1976, 1994). These internationally recognized experts on professional learning and collaboration developed their value creation framework to address the dual challenges of attending to the diversity of types of value and providing systematic analysis that is timely and efficient. Using this framework as a guide, you will determine the evidence you will collect over time, how you will make sense of the evidence, and how you will share your results with others.

The value creation framework defines five types of value that are created through participation in professional learning and collaboration activities. (The authors refer to these types as cycles of value to emphasize that they are interconnected.) The types of value vary from that which participants experience right away as they participate in an activity (e.g., a sense of excitement) to the ultimate results that the goals to which activities are aligned aim to improve (e.g., graduation rates). The five types of value are defined in Table 1.

To document the value created by your professional learning and collaboration activities, you will collect three kinds of evidence: indicators, stories, and artifacts.

Table 1. Types of Value

Type

Definition

Example

Immediate: Value educators experience Activities and immediately through their interactions participation

A teacher feels less isolated because he was provided the opportunity to connect, via webinar, with other teachers who are working to overcome similar challenges.

Potential: Knowledge and social capital

Value educators receive that could prove useful in the future; this might be new knowledge, skills, resources, or relationships

During an online workshop, a principal learned about a new classroom observation process that she may employ in future teacher evaluations and was introduced to two experienced school leaders whom she can access for advice.

Applied: Changes in practice

Value generated when educators apply what they have learned or developed in the professional practice

A teacher uses new ideas learned, via a Twitter chat, to adapt two lesson plans in her classes.

Realized: Performance improvement

Value that results from the application of what has been learned or generated

Digital literacy assessment scores improve across the district after implementation of a codeveloped, statewide bring-your-owndevice acceptable use policy.

Reframing: Changes in what is valued

Value created when educators redefine what success means as a result of their collaboration

A school-based professional learning community, focused on literacy, connected with other professional learning communities through an online network, which resulted in the integration of literacy instruction across all subjects, not just language arts.

3

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download