Evaluation of Educational Change
CCT693 Spring 2000
Version 29 January 2000, subject to revision

Instructor: Peter Taylor, Critical & Creative Thinking Program
Email: peter.taylor@umb.edu; Phone: 617-287-7636
Office: Wheatley 2nd flr 143.09 (near Counseling & School Psychology)
Class Time: M 4-6.30; Office/phone conference hours: M 2-3.30, or by arrangement.
Course Website: http://www.faculty.umb.edu/peter_taylor/693-00.html
General email: Emails sent to PT with "for CCT693" in the subject line will be forwarded to all students in the course.

Course description
Techniques for and critical thinking about the evaluation of changes in educational practices and policies in schools, organizations, and informal contexts. Topics include quantitative and qualitative methods for design and analysis, participatory design of practices and policies, institutional learning, the wider reception or discounting of evaluations, and selected case studies, including those arising from semester-long student projects.

Overview
Sections to follow in syllabus
Overview
Assessment and Requirements
Texts and Materials
Schedule of classes
Additional sections in course packet
Notes on Teaching/Learning Interactions
Learning through dialogue around written work
Rationale for the Assessment system
Before, during and after class--Critical thinking about course readings and discussions
Communication before, during, and after class
Basic course protocols/expectations
Guide to Classes, Assignments and Readings
Evaluation and On-going Development of this Course


Overview
Reflective practice--the goal of the CCT Program--implies that we evaluate the effects of any changes we make--in curricula, other educational practices, institutional arrangements, and so on--and learn from that evaluation. We can also use the results of systematic evaluations to try to influence others to adopt our innovations, recognizing that people do not always take notice of positive evaluations--or negative ones for that matter. One way to increase the likelihood that others will adopt changes and sustain them is to involve them in designing the innovations in the first place and in implementing them.
In light of these considerations, there are a number of strands to this course:
1. Throughout the semester, but particularly in the first 5 weeks, the case studies, class activities, assignments, reading & precis of the text, and the mini-project are designed so you can learn concepts and techniques for systematic evaluation and for facilitating group participation.
2. In the second half of the semester you design an evaluation of an educational change you are interested in--here education can be construed very broadly. You might conduct a pilot version of this evaluation and revise the design in light of the experience. You will also have to spend time planning the change in practice, policy, etc. unless the change you are concerned about has already been instituted. Perhaps you will decide to facilitiate participation of those potentially affected in planning the change and/or its evaluation. In any case, from the start of the course, you should be thinking about what educational change you want to focus on for the design project.
3. Starting in week 4, you will as a group undertake an "umbrella" project of working out how best to support each other to get competent and comfortable in evaluating and facilitating educational change. This includes planning the remaining classes (subject to some constraints).
4. At various points we also step aside from the educational changes you are interested in and think critically about evaluation, especially, to consider the politics of evaluation and educational change more generally--the wider array of influences shaping the evaluation (or lack thereof) of changes, or shaping the impact (or lack thereof) of evaluations.

Assessment and Requirements
Your grade will be based on the components to follow, which are described in more detail in the course packet. For each of the two parts of the grade, satisfactory completion of 80% of assignments or items gives you an automatic B+. For those who meet the B+ requirements, a higher grade may be assigned following the rubric below. (See course packet for rationale and some additional details.)

Written assignments and presentations, 2/3
A. Assignments introducing you to evaluation (5).
B. Evaluation Design Project: Design an evaluation of a change of or intervention in a specific classroom or workplace practice, educational policy, educational institution, or social policy. A sequence of 5 assignments is required--initial description, notes on work-in-progress, work-in-progress presentation, complete draft report, and final 1500-2500 words report.

Participation and contribution to the Class Process, 1/3.
C. Prepared participation and attendance at class meetings (12)
D. Journal & Clippings, collected for perusal mid-semester & end
Journal = weekly responses/notes on homework tasks, readings, projects, class discussions, and clippings
Clippings = at least 10 items from current magazines and newspapers, with your annotations
E. Minimum of two in-office or phone conferences on your assignments and projects.
F. Peer commentary on another student's draft report
G. End-of-semester Process Review on the development of your work

Rubric
B+
80% of assignments satisfactory, i.e., submitted and revised until OK/RNR (8 of 10)
80% of participation items fulfilled (14 of 18)
In addition to the requirements for B+, if your work shows half of the following qualities, you will get an A-, and almost all of them, an A:
Assignments on time, often revised thoroughly and with new thinking in response to comments
Project innovative; work undertaken with considerable initiative
Final report detailed and professionally presented
Final report reflects on what you have learned and indicates future directions to develop
Active participation in class
Process Review submitted, with cover note & annotations that show deep reflection on your process of development

If you do not reach the 80% level, the grade for Written assignments & presentations is pro-rated from B+ down to C =50% of assignments OK/RNR. Similarly the Participation & Process grade goes down to C = 50% of participation items.
Texts and Materials
Required:
Weiss, C. H. (1998). Evaluation. Upper Saddle River, NJ: Prentice Hall.
Calhoun, E. F. (1994). How to Use Action Research in the Self-Renewing School. Alexandria, VA: ASCD.
Course packet and Supplementary Readings, available from PT.
A journal/workbook to carry with you at all times and an organized system to store handouts and loose research materials (e.g., a 3 ring workbinder with dividers and pockets, an accordion file, or file folders).
Recommended:
- as a guide to writing and revising: Elbow, P. (1981). Writing with Power. New York: Oxford Univ. Press (also on reserve).
- as a guide on technical matters of writing scholarly papers: Turabian, K. L. (1996). A Manual For Writers of Term papers, Theses, and Disertations. Chicago: Univ. of Chicago Press (in library's reference section).
On reserve in Healey library:
Patton, M. Q. (1982). Practical Evaluation. Newbury Park, CA: Sage. (See also the compilation of chapter precis from 1999.)
Pietro, D. S. (Ed.) (1983). Evaluation Sourcebook. New York: American Council of Voluntary Agencies for Foreign Service.
Spencer, L. J. (1989). Winning Through Participation. Dubuque, Iowa: Kendall/Hunt
Stanfield, B. (Ed.) (1997). The Art of Focused Conversation. Toronto: Canadian Institute of Cultural Affairs.
Stark, J. S. and A. Thomas (Eds.) (1994). Assessment and Program Evaluation. Needham Heights, MA: Simon & Schuster.
Mini-Project reports from Spring 1999
Evaluation Design Project reports from Spring 1999
Folders of additional readings

Schedule Of Classes

Class 1 (1/31)
Introduction to Issues in Evaluation & Educational Change
Initial look at Issues in Evaluation (Gallery Walk activity)
Overview of Course and its strands
Overview of Evaluation (through two mini-case studies):
Core issue = Test before change->change->Test after change
Surrounding Issue = How to get people to change
Additional Steps in the Evaluation Process
Critical Incident Questionnaire

Post-class reading: Read the syllabus and Notes on Teaching/Learning Interactions in the course packet. Review the structure of remainder of course packet.
Assignment 1 (due week 2): Use evaluation clock to analyze an educational study as reported in a clipping from set 1 (in the course packet). For more instructions on this and future assignments see course packet. Related reading (in course packet): Pietro, D. S. (Ed.) (1983). Evaluation Sourcebook. New York: American Council of Voluntary Agencies for Foreign Service, p. 22-23 (on eval. clock) and p. 12-17, & 21 (to provide context and introduce you to the rest of the Sourcebook).

Class 2 (2/7)
Using the Evaluation Clock to analyze an Evaluation
*A* Asmt 1 due
Feedback on Critical Incident Questionnaire
Questions on Teaching/Learning Interactions, described in the Course packet
Reports in small groups and then to whole class on analysis using Evaluation Clock (Asmt 1)

Assignment 2 (due class 3): Precis of one chapter from Evaluation (hard copy & emailed version)

Class 3 (2/14)
Overview of Evaluation
*A* Asmt 2 due
Digest chapters of Evaluation (in pairs)
Digest PT's comments on Asmt 1 (in pairs)
Focused Conversation on analysis using Evaluation Clock
Provisional selection of topics for mini-project

Post-class reading (in course packet): Stanfield, B. (Ed.) (1997). The Art of Focused Conversation. Toronto: Canadian Institute of Cultural Affairs, 6-29.

2/21 No class (Presidents' Day)
PT e/mails comments about chapter precis (by 2/18).
Task by 2/24: Email to PT about topic chosen for mini-project (from sets 1 or 2), leads found & guidance needed (or by mail to 41 Cornell St., Arlington, MA 02474)
Work on assignment 3 (due class 4): Evaluation clock for mini-project topic

Class 4 (2/28)
Supporting each other in evaluating and facilitating educational change (generating a group project through Strategic Participatory Planning (SPP)), part I
*A* Revised chapter precis (asmt 2) due, to be collated, copied, and distributed
*A* Asmt 3 due

Post-class reading:
Review SPP notes from ICA Facilitators Manual (in course packet). See also Spencer, L. J. (1989). chaps. 5 & 7 of Winning Through Participation. Dubuque, Iowa: Kendall/Hunt (on reserve).
Read Evaluation using the Collection of Precis as a guide & shortcut.
Assignment 4 (due 3/14): Questions & comments on Evaluation
Schedule first conference before 3/9

Class 5 (3/6)
Supporting each other in evaluating and facilitating educational change (generating a group project through Strategic Participatory Planning), part II

Post-class reading:
Review SPP notes and chaps. 5 & 7 of Spencer (continued)

3/13 No class (Spring break)
*A* Asmt 4 due 3/14 by email (or by mail to 41 Cornell St., Arlington, MA 02474)
Work on assignment 5 (due class 6): Mini-project report
Read How to Use Action Research (by 3/27)
Class 6 (3/20)
Activities to be determined by SPP in classes 4 and 5 (some possibilities listed at end of syllabus)
*A* Asmt 5 due
*A* Journal & clippings collected for perusal

Class 7 (3/27)
Activities to be determined by SPP
*A* Asmt due: Initial Project Description

Class 8 (4/3)
Activities to be determined by SPP
Schedule second conference before 4/6

Class 9 (4/10)
Work-in-progress Presentations on Student Projects I
*A* Asmt due: Work-in-progress notes on Design Project

4/17 No class (Patriots' Day)

Class 10 (4/24)
Work-in-progress Presentations on Student Projects II

Class 11 (5/1)
Activities to be determined by SPP
*A* Asmt due: Complete Draft of Design Project (2 copies)
Make comments on another student's draft, and send them to the person by 5/8.

Class 12 (5/8)
Taking stock of course & of change: Where have we come & where do we go from here?
Historical Scan
Designing different kinds of course evaluations
*A* Journal & Clippings collected for perusal II, with Process Review
*A* Comments due on another student's draft

*A* Asmt due 5/15: Final version of Design Project

Some possible topics for unspecified weeks (based on previous year's class; see descriptions of topics, activities, readings in course packet)
Tools Of Evaluation
Making sense of quantitative analyses
Complete a small statistical study to address "arithmophobia"
Establishing a statistical support network
Facilitating Change In Groups
Focused Conversations
Constructivist Listening
Action Research
School Change
Participatory Action Research
Politics Of Evaluation
Case studies of Headstart, The Coleman Report, Mass Teachers Test
Theories Of Evaluation And Educational Research
Action Research as an alternative to Positivist and Interpretivist approaches
From action research to Heterogeneous Re/constructions of Social Situations