University of Massachusetts at Boston
Graduate College of Education
Critical & Creative Thinking Program

Seminar on
Evaluation of Educational Change
CrCrTh693
Spring 2001

Syllabus (Version 28 Jan 01)

Instructor: Peter Taylor, Critical & Creative Thinking Program
Email: peter.taylor@umb.edu
Phone: 617-287-7636
Office: Wheatley 2nd flr 143.09 (near Counseling & School Psychology)
Class: M 4-6.30, Wheatley 2-209
Office/phone call hours: M 2-3.30, Tu 4.30-6, or by arrangement (Tuesdays preferred)
Course Website: http://omega.cc.umb.edu/~ptaylor/693-01.html
General email: Emails sent to PT with "for CCT693" in the subject line will be forwarded to all students in the course.

CATALOG DESCRIPTION
This course covers techniques for and critical thinking about the evaluation of changes in educational practices and policies in schools, organizations, and informal contexts. Topics include quantitative and qualitative methods for design and analysis, participatory design of practices and policies, institutional learning, the wider reception or discounting of evaluations, and selected case studies, including those arising from semester-long student projects.

COURSE DESCRIPTION for Spring 2001: Evaluation and Educational Change
Techniques for and critical thinking about two kinds of evaluation:
evaluation of changes in educational practices and policies in schools, organizations, and informal contexts; and
evaluations of needs in such settings so as to design those changes.
Topics include participatory design and evaluation of practices and policies, reflective practice, the wider political reception or discounting of evaluations, and selected case studies, including those arising from student projects.

PREREQUISITES: CrCrTh601 and 602, or permission of instructor
Sections to follow in syllabus
Texts and Materials
Assessment and Requirements
Course Objectives/Overview
Schedule of classes
Key Teaching/Learning Tools
Bibliography
TEXTS:
Required:
Calhoun, E. F. (1994). How to Use Action Research in the Self-Renewing School. Alexandria, VA: ASCD.
Supplementary photocopied readings, distributed during semester. Expected cost for copying and copyright permissions, $25, payable to instructor
A journal/workbook to carry with you at all times and an organized system to store handouts and loose research materials (e.g., a 3 ring workbinder with dividers and pockets, an accordion file, or file folders).
Recommended:
- as a guide to writing and revising: Elbow, P. (1981). Writing with Power. New York: Oxford Univ. Press (also on reserve).
- as a guide on technical matters of writing scholarly papers: Turabian, K. L. (1996). A Manual For Writers of Term papers, Theses, and Disertations. Chicago: Univ. of Chicago Press (in library's reference section).
On reserve in Healey Library:
From Spring 1999 and 2000: Evaluation Design Project reports; Mini-Project reports; and Precis of texts
Binder of clippings from previous years
Books and additional readings marked in the bibliography

ASSESSMENT & REQUIREMENTS:
(Detail on these requirements is provided later in the syllabus and in handouts.)

Written assignments and presentations, 2/3 of grade
A. Four introductory Assignments (phases 1 and 2).
B. Project: Design EITHER an evaluation of a change or intervention in a specific classroom, workplace or personal practice, educational policy, educational institution, or social policy OR your facilitation of a collaborative process to shape such a change or intervention. A sequence of 5 assignments is required--initial description, notes on research and planning, work-in-progress presentation, complete draft report, and final (1200-2000 words) report.
Participation and contribution to the class process, 1/3 of grade.
C. Prepared participation and attendance at class meetings (13)
D. Journal & Clippings, collected for perusal mid-semester & end (= 2 items)
Journal = weekly responses/notes on homework tasks, readings, projects, class discussions, and clippings
Clippings = at least 10 items from current magazines and newspapers, with your annotations, and/or your commentary on clippings included in the course packet
E. Minimum of two in-office or phone conferences on your assignments and projects (before 3/8 & 4/6) (=2 items)
F. Peer commentary on another student's draft report and/or Buddy support on another's project
G. End-of-semester Process Review on the development of your work

Rubric
B+ is earned automatically for 80% (7 of 9) of Written items OK/RNR (=OK/ Revision-reflection-resubmission Not Requested) and 80% (15 of 19) of Participation items fulfilled.
The qualities below will determine whether a higher grade is earned. If you show half of the qualities to follow, you earn an A-. If you show almost all of these, you earn an A:
A sequence of assignments paced more or less as in syllabus, often revised thoroughly and with new thinking in response to comments
Project well planned and innovative, carried out with considerable initiative
Final report clear, well structured, detailed, reflecting on what you have learned and indicating future directions to develop
Final report professionally presented, with supporting references
Active participation in all classes
Process Review that shows deep reflection on your process of development through the semester

If you do not reach the B+ level, the grade for Written assignments & presentations is pro-rated from B+ down to C for 50% of assignments OK/RNR. Similarly, the Participation & process grade goes down to C for 50% of participation items.

ACCOMMODATIONS: Sections 504 and the Americans with Disabilities Act of 1990 offer guidelines for curriculum modifications and adaptations for students with documented disabilities. If applicable, students may obtain adaptation recommendations from the Ross Center (287-7430). The student must present these recommendations to each professor within a reasonable period, prefeably by the end of the Drop/Add period.

Students are advised to retain a copy of this syllabus in personal files for use when applying for certification, licensure, or transfer credit.

This syllabus is subject to change.
COURSE OBJECTIVES

Overview
A long version of the course title might be "Evaluation related to Engagement in Educational Change," where
"Education" is construed broadly to include not only school curricula, but also educational policies and institutional arrangements, and training, coaching, and the conduct of workshops in any setting;
"Evaluation" stands for the systematic study of i) what has been happening before; and ii) of the effects of any changes you implement, presumably changes designed in response to evaluating what had been happening before; and
"Engagement" denotes that the course is not only about evaluating past situations and any future changes, but also about collaborating with other people in several ways:
to design and bring about constructive change;
to undertake the evaluations;
to dialogue and reflect on the implications of any results; and
to insure that the results of the evaluation have an influence on the relevant people and groups, i.e., the potential "stakeholders."
"Engagement" also reminds you that, in order to contribute effectively to change, you need to be engaged yourself--to have your head and heart together. The course, therefore, provides tools for personal reflection on your practice.

Evaluation related to Engagement in Educational Change can be summarized in an "Action Research" cycle (or spiral):
Systematic study of what has been happening ->
Reflection & dialogue ->
Design an action/change/engagement ->
Implement this action ->
Systematic study of effects of the action ->
Reflection & dialogue ->
Revise the action to improve it and/or Promote its wider adoption ->
etc.

As the course unfolds you should come to appreciate the following flow of thought:
0. Suppose you are concerned about some educational practice or policy or institutional arrangement, or the equivalent in some other setting.

1. In order to influence/change what is going on, it is important to study systematically what:
a. has been happening; or
b. is about to happen (e.g., under a new mandate); or
c. could happen given a change you or someone else is designing. (Reflective practice--the goal of the CCT Program--implies that we evaluate the effects of any changes we make and learn from that evaluation. )

2. "Study systematically" means to evaluate the effects of changes in practices/ policies/ institutional arrangements, either by comparing before vs. after the change, or the changed situation vs. a unchanged control.

3. Evaluation of changes help you to
a. Promote their wider adoption, or
b. Revise the changes, or advance new courses of action

4. Whether anyone pays attentionto the evaluation depends its political use/fulness for mobilizing support and addressing (potential) opposition. The politics of evaluation and educational change more generally could be a course in itself, but for now note that:
a. If you build evaluation into your proposals for change, it shows your preparedness to learn from the effects of the change, and this might increase support for making changes (3b) or promoting their wider adoption (3a); and
b. If you identify the different stakeholders and look ahead to what the research results could allow them to do, this can enter into the process of designing the change and its evaluation.

5a. Action Research typically starts with 1a, focuses first on studying what has been happening, but inevitably gets drawn into issues of stakeholder buy-in.
b. The Evaluation Clock helps you keep an eye on the buy-in of sponsors or stakeholders in deciding what and how to evaluate, how to analyze results, etc.
c. Participatory Action Research (PAR) achieves buy-in through participation of stakeholders in the designing changes, implementing them, and evaluating them.

6. Participation is enhanced by facilitation, other group processes, and reflective practices that bring out and acknowledge the different participants' voices.

7. In this course you develop your ability to go from concern about some educational (or related) practice/ policy/ institutional arrangement to influencing what is going on. To this end you:
a. experience, learn, and practice various ways to promote participation and reflective practice (including your own participation);
b. examine critically the evaluations of others (or the lack of the appropriate evaluations); and
c. undertake a project in an area of your particular concern in which you design and perhaps carry out a pilot version of either i) an evaluation of a change and/or ii) facilitating participation in change or facilitating reflective practice.

8. The course consists of four phases:
PHASE 1: Get introduced to issues of facilitating and evaluating change through experience. I throw you right into Action Learning teams working on a common case: Extending CCT's impact beyond its formal programs of study, as begun by an outreach organization, "Thinking for Change--a resource center for critical and creative thinking and reflective practice."

PHASE 2: Examine the text and other readings in light of your experience in phase 1.

PHASE 3: Build on phases 1 & 2 by undertaking individual projects and supporting each other to complete them.

PHASE 4. Reflect on the knowledge and experience gained during the course, with a view to future research and engagement.

See also the Description of Key Teaching/Learning Tools
SCHEDULE OF CLASSES

See Overview for rationale of the four phases of the course
See Key teaching/learning tools for brief rationale of activities and assignments
**More details on preparation for classes and on assignments (with examples of previous students' assignments) will be distributed through handouts and emails**

PHASE 1 INTRODUCING ISSUES THROUGH EXPERIENCE BY THROWING YOU RIGHT INTO ACTION LEARNING TEAMS WORKING ON A COMMON CASE.
Class 1 (1/29)
Introductions
--to each other, our experiences and concerns, including Gallery walk
--to the overall schema of the course
--to the case for the Action Learning sessions: Extending CCT's impact beyond its formal programs of study, as begun by an outreach organization, "Thinking for Change--a resource center for critical and creative thinking and reflective practice." (Freewriting and brainstorming on what might be involved in this endeavor)
Critical Incident Questionnaire

Post-class reading: Read the syllabus, especially the overview. Begin browsing the binder of clippings from previous years.
Other tasks: Begin your journal and collection of annotated clippings. Email your definition of the problem for distribution to the other members of your action learning team (as convened in class 1 or by the next day).
Schedule first conference before 3/8

Class 2 (2/5)
Action Learning Session 1
"Action learning is an approach by which a small team (generally 5-7 persons) contract together to achieve a learning objective through a combination of shared reflection and action" (Jenkins 2000).
Feedback on Critical Incident Questionnaire
Questions on Course Overview and Teaching/Learning Tools
Action learning team work, facilitated by guests--moving from check-in, to reporting, to action planning, to closing agreements (incl. separate work assignments).
Reading: Jenkins, "Action learning: Taking the time it takes."

Class 3 (2/12)
Action Learning Session 2
Teams facilitated by group member, with floating assistance from PT
Focused Conversation on Action Learning.
Preliminary verbal reports to the whole class.
Pre- or post-class reading on Focused Conversations (in course packet): Stanfield, 6-29.

2/19 No class (Presidents' Day)
*A* Asmt. 1: Email PT by 2/20 a 1-2 page report on your work assignments and how that contributes to the progress your Action Learning team has made.

Class 4 (2/26)
Action Learning Session 3
Teams facilitated by group member, with floating assistance by guest facilitator
In class writing: brief written reports on progress and plans for remaining week.

Class 5 (3/5)
Presentations by Action Learning teams and Closing-out Sessions
*A* Asmt. 2: Draft report from each team. Either a) use initials to indicate the different contributions within the team, or b) combine separate contributions under a cover sheet that explains their combined message.
Post-class reading: To reflect on your experience, start early on reading for class 6.

PHASE 2: EXAMINING THE TEXT AND OTHER READINGS IN LIGHT OF YOUR ACTION LEARNING EXPERIENCE.
Class 6 (3/12)
Comparing your Experience as Novice Action Researchers with the Considered Formulations in the Texts
Small group discussions and reports back to the class
Panel of guest school change action researchers
Reading: Selection from Calhoun, How to Use Action Research, Hitchcock & Hughes, Chap. 3, "Access, ethics, and objectivity," Greenwood & Levin, Chaps. 8 & 11, "Action research cases," & "Action science and organizational learning"
Schedule second conference before 4/6

3/19 No class (Spring break)
*A* Asmt. 3 due 3/20 by email (or by mail to 41 Cornell St., Arlington, MA 02474). Reflection paper (2-4 pages) relating your Action Learning experience to points made by at least one of the readings for class 6.
Reading (between class 6 and 8): Review Mini-Project reports from Spring 1999 and 2000, and Evaluation Design Project reports from Spring 1999 and 2000 (on reserve).

PHASE 3: UNDERTAKING INDIVIDUAL PROJECTS AND SUPPORTING EACH OTHER TO COMPLETE THEM & Phase 2 continued
Class 7 (3/26)
Formulating informative comparisons as a basis for evaluations (phases 2 & 3)
Comparison steps (2-4) in the evaluation clock, used to analyze a clipping
Introduction to Statistical formulations of comparisons
Establishing a statistical support network
Critical Incident Questionnaire
*A* Journal & clippings collected for perusal
Pre- or post-class reading: PT's precis of Pietro, Evaluation Sourcebook, p. 22-23 (on evaluation clock) and p. 12-17, & 21 (to provide context)

Class 8 (4/2)
Strategic Participatory Planning, applied to personal course and life projects
Feedback on Critical Incident Questionnaire
Strategic personal planning workshop
In-class drafting of initial project description
Reading Spencer, chaps. 5 & 7.
Post-class reading: Materials on Strategic Participatory Planning from ICA Facilitators Manual (in folder on reserve)
Weissglass, "Constructivist listening"
*A* Asmt. 4 Review the process of your Action Learning project using evaluation clock.

Class 9 (4/9)
Depending on student need, either
More on Evaluation Clock & Strategic Participatory Planning
or
Politics and Theories Of Evaluation And Educational Research (phase 2 cont. & segue to phase 4?)
Selection to be established from:
Case studies of Headstart (Woodhead), The Coleman Report (Hunt), San Jose United School District (Rokovich et al.)

4/16 No class (Patriots' Day)
*A* Asmt due by email or mail: Revised Project Description
Reading: Hitchcock & Hughes, Chaper 5, Designing, planning and evaluating research.

Class 10 (4/23)
Work-in-progress Presentations on Student Projects I
*A* Asmt due: Notes on Research and Planning for Individual Student Project

Class 11 (4/30)
Work-in-progress Presentations on Student Projects II

Class 12 (5/7)
Politics and Theories Of Evaluation And Educational Research (phase 2 cont. & segue to phase 4?)
Selection to be established from:
a. Case studies of Headstart (Woodhead), The Coleman Report (Hunt), San Jose United School District (Rokovich et al.)
or
b. Action Research as an alternative to Positivist and Interpretivist approaches
Participatory Action Research -- video on the Highlander center
From action research to Heterogeneous Re/constructions of Social Situations
Reading: Carr & Kemmis or Greenwood (selections TBA), Taylor 1999a,b

*A* Asmt due: Complete Draft of Design Project (2 copies)
*A* Make comments on another student's draft, and send them to the person by 5/12.

PHASE 4. REFLECTING ON THE KNOWLEDGE AND EXPERIENCES OF THE COURSE, WITH A VIEW TO FUTURE RESEARCH AND ENGAGEMENT.
Class 13 (5/14)
Taking stock of course & of change: Where have we come & where do we go from here?
Historical Scan
Designing different kinds of course evaluations
*A* Journal & Clippings collected for perusal II
*A* Final Project report and Process Review
KEY TEACHING/LEARNING TOOLS

Activities for "self-affirming" learning
Students already know a lot. If this knowledge is elicited and affirmed (e.g., through the gallery walk in class 1), they are more able to learn from others. Activities such as freewriting bring to the surface students' insight that they were not able, at first, to acknowledge. Over the course of the semester, students are encouraged to recognize that there is insight in every response and share their not-yet-stable aspects. The trust required takes time to establish.

Taking stock during semester ("formative evaluation")
Through activities, such as the Critical Incident Questionnaire, I encourage students to approach this course as a work-in-progress. Instead of harboring criticisms to submit after the fact, we can find opportunities to affirm what is working well and suggest directions for further development. (See also Taking stock at end of semester).

Journalling and thought pieces
The Journal should include weekly responses and notes on readings, class discussions, clippings, websites, and weekly questions. Through writing in your journal, you will be better able to weave the course material into your own thinking, and to bring your own thinking into class activities. In preparation for class, you might write in your journal a commentary on readings, or, after class, review the readings and the class activities. In either case explore, when appropriate, the relationship between, on one hand, your interests and possible projects and, on the other hand, the readings and activities. I encourage those of you who find it hard to make space for journalling to stay 10 minutes after class and write while your thoughts are fresh.
Journal excerpts are to be submitted 5 times during the semester, and then revised and resubmitted in response to my comments on these extracts. Journals as a whole will be collected for perusal twice during the semester. Bind together pages with post-its or otherwise indicate which bits you do not want me to look at. I want to get an overall impression of your developing process of critical thinking about course readings and discussions.

Clipping collections
To keep up with current developments--and get you into the habit of this for your lifelong learning-- compile a packet of clippings and xeroxes of articles from newspapers, magazines, journals, and websites. Write the full citation on each article, unless it is already included. Use large post-its to add your own reflections on specific points in the articles you choose. Submit the packets twice at the same time as the journal is reviewed.

Guided freewriting
In a freewriting exercise, you should not take your pen off the paper. Keep writing even if you find yourself stating over and over again, "I don't know what to say." What you write won't be seen by anyone else, so don't go back to tidy up sentences, grammar, spelling. You will probably diverge from the topic, at least for a time while you acknowledge other preoccupations. That's OK--it's one of the purposes of the exercise to get things off one's chest. However, if you keep writing for ten minutes, you should expose some thoughts about the topic that had been below the surface of your attention--that's another of the aims of the exercise. Reference: Elbow, chapter 2.

Think-pair-share
After preparing your thoughts, you pair up with another person, and, through sharing ideas, you refine your own and prepare to share a key part of them with the whole class (time permitting).

Dialogue around written work
I try to create a dialogue with each student around written work, that is, around your writing, my responses, and your responses in turn. I am still learning how to engage students in this, given your various backgrounds and dispositions, and my own. Central to this teaching/learning interaction are requests to "Revise and Resubmit." The idea is not that you make changes to please the teacher or to meet some standard, but that as a writer you use the eye of others to develop your own thinking and make it work better on readers. I continue to request revision when I judge that the interaction can still yield significant learning; the request does not mean your (re)submission was "bad." Even when the first submissions of written assignments are excellent, angles for learning through dialogue are always opened up.

Stages of development for course project
The course project should not be seen as a final term paper, but as a process of development that involves dialogue with the instructor and other students and revision--re-seeing--in light of that dialogue. To facilitate that process, a sequence of five assignments and peer commentary is required. Brief descriptions below will be supplemented in later handouts.
Initial description
Building on your in-class draft and comments back from me, compose an initial overview of your project--one or two paragraphs that may, several revisions later, end up setting the scence in the introduction of your report.
Notes on research and planning
Pull together notes on your reading and your thinking and present it in a form that will elicit useful comments from me.
Work-in-progress presentation
Preparing presentations, hearing yourself deliver them, and getting feedback usually leads to self-clarification of the overall direction of your project and of your priorities for further work. In this spirit, I schedule presentations early. They are necessarily on work-in-progress and I encourage you to indicate where additional investigation is needed and where you think it might lead you.
Complete draft report
The report of your design project should include material that conveys your process of development and future plans, and it should Grab readers' attention, Orient them, and move through Steps so that they appreciate what you, the writer, have accomplished. The draft must get to the end to count, even if some sections along the way are only sketches.
Final report (1200-2000 words, plus bibliography of references cited)
For the final project report to be counted as final, you must have revised in response to comments on previously submitted drafts. Allow time for the additional investigation and thinking that may be entailed.

Peer commentary
After the draft is completed I require you to pair up and comment on another student's draft. Keep Elbow's chapters 3 & 13 in mind when you decide what approaches to commenting you ask for as a writer and use as a commentator. In the past I made lots of specific suggestions for clarification and change in the margins, but in my experience, such suggestions led only a minority of students beyond touching up into re-thinking and revising their ideas and writing. On the other hand, I believe that all writers value comments that reassure them that they have been listened to and their voice, however uncertain, has been heard.

Taking stock at end of semester involves multiple angles on course evaluation (including written evaluations and Process review portfolios--see below):
a) to feed into your future learning (and other work), you take stock of your process(es) over the semester;
b) to feed into my future teaching (and future learning about how students learn), I take stock of how you, the students, have learned.

Process review portfolios
These should contain 4-6 examples that capture the process of development of your work and thinking about environmental education and critical thinking, not simply the best products. Journal entries, free writing, drafts, etc. may be included. Explain your choices in a cover page and through annotations (large post-its are a good way to do this).

BIBLIOGRAPHY
(* indicates on reserve in Healey)
Calhoun, E. F. (1994). How to Use Action Research in the Self-Renewing School. Alexandria, VA: ASCD.
Carr, W. and S. Kemmis (1986). Becoming Critical: Education, Knowledge and Action Research. Geelong: Deakin University Press., chapters 6 & 7 (up to p. 200)
Elbow, P. (1981). Writing with Power. New York: Oxford Univ. Press*
Greenwood, D. J. and M. Levin (1998). Introduction To Action Research: Social Research For Social Change. Thousand Oaks, CA: Sage.
Hitchcock, G. and D. Hughes (1995). Research and the Teacher: A Qualitative Introduction to School-based Research. New York: Routledge.*
Hunt, M. (1985). "The dilemma in the classroom: A cross-sectional survey measures the effects of segregated schooling," in Profiles of Social Research: The Scientific Study of Human Interactions. New York: Russell Sage, 51-97.
Institute of Cultural Affairs, n.d., Facilitators Manual (excerpts on Strategic Participatory Planning). Toronto: Canadian Institute of Cultural Affairs.
Jenkins, M. (2000). "Action learning: Taking the time it takes." Paper presented to the International Association of Facilitators, Toronto, April 27 2000.
Patton, M. Q. (1982). Practical Evaluation. Newbury Park, CA: Sage.*#
Pietro, D. S. (Ed.) (1983). Evaluation Sourcebook. New York: American Council of Voluntary Agencies for Foreign Service.*
Rokovich, M. A., M. Stevens and J. Stallman (2000). "Implementing change at SJUSD: An unfinished case study." Presented to the International Association of Facilitators, Toronto, April 27 2000.
Senge, P., N. Cambron-McCabe, T. Lucas, B. Smith, J. Dutton and A. Kleiner (2000). Schools That Learn. New York: Currency.
Spencer, L. J. (1989). Winning Through Participation. Dubuque, Iowa: Kendall/Hunt *
Stanfield, B. (Ed.) (1997). The Art of Focused Conversation. Toronto: Canadian Institute of Cultural Affairs.*
Stanfield, B. (2000). The Courage To Lead: Transform Self, Transform Society. Gabriola Island BC: New Society Publishers.*
Stark, J. S. and A. Thomas (Eds.) (1994). Assessment and Program Evaluation. Needham Heights, MA: Simon & Schuster. (in Healey stacks)
Taylor, P. (1999a). "Bridging the divide between subjects and outsider-scientists: Participatory action research in a Kenyan agroforestry project" ms.
Taylor, P. (1999b) "Constructing Heterogeneous Webs in Socio-Environmental Research." ms.
Turabian, K. L. (1996). A Manual For Writers of Term papers, Theses, and Disertations. Chicago: Univ. of Chicago Press
Weiss, C. H. (1998). Evaluation. Upper Saddle River, NJ: Prentice Hall.*#
Weissglass, J. (1990). "Constructivist listening for empowerment and change." The Educational Forum 54(4): 351-370.
Woodhead, M. (1988). "When psychology informs public policy." American Psychologist 43(6): 443-454.
# See also the compilation of chapter precis of these books on reserve.*