Evaluation of Educational Change
CCT693 Spring 2000
"We did make a terrible lot of mistakes...
So we had a little self-criticism, and we said, what we know, the solutions
we have, are for the problems that people don't have.
And we're trying to solve their problems by saying they have the problems
that we have the solutions for.
That's academia, so it won't work.
So what we've got to do is to unlearn much of what we've learned, and then
try to learn how to learn from the people."
Myles Horton, founder of the Highlander Center, now located in New Market,
Tennessee,
in an interview with Bill Moyers, June 5, 1981
Transcript published in Horton, M. and B. Moyers (1983). "The adventures of
a radical hillbilly: An interview with Myles Horton." Appalachian Journal
9(4): 248-285. Quote from p. 259.
See also Southern Exposure, 6(1) for a history of the Highlander
Center.
Course Packet
Seminar on Evaluation of Educational Change
CCT693 Spring 2000
Web version in development. Apologies for formatting glitches
Instructor: Peter Taylor, Critical & Creative Thinking Program
See syllabus at http://www.faculty.umb.edu/peter_taylor/693-00.html
Table of Contents
Notes on Teaching/Learning Interactions
** please read & bring questions you have to class in week 2**
Learning through dialogue around written work (incl. "revise
& resubmit")
Rationale for the Assessment system
Before, during and after class--Critical thinking about course readings and discussions
(incl. journal, clippings, process review, course evaluation)
Communication before, during, and after class
Basic course protocols and expectations
Guide to Classes, Assignments and Readings
including
Assignment 1 on Evaluation Clock
Assignments 2 and 4 Precis of Evaluation
Assignments 3 and 5 Mini-project
Strategic Participatory Planning
Evaluation Design Projects
Evaluation and On-going Development of this Course
Notes on Teaching/Learning Interactions
Learning through dialogue around written work
I try to create a dialogue with each student around written work, that is,
around your writing, my responses, and your responses in turn. I am still
learning how to engage students in this, given your various backgrounds and
dispositions, and my own. (My efforts to make this system work better for
students are reviewed at http:// www.faculty.umb.edu/peter_taylor/citreport.html.)
Central to this teaching/learning interaction are requests to "Revise and
Resubmit." The idea is not that you make changes to please the teacher or to
meet some standard, but that as a writer you use the eye of others to develop
your own thinking and make it work better on readers. I continue to request
revision when I judge that the interaction can still yield significant
learning; the request does not mean your (re)submission was "bad." Even when
the first submissions of written assignments are excellent, angles for learning
through dialogue are always opened up.
In my comments I try to capture where the writer was taking me and make
suggestions for how to clarify and extend the impact on readers of what was
written. After letting my comments sink in, you may conclude that I have
missed the point. In this case, my misreading should stimulate you to revise
so as to help readers avoid mistaking the intended point. If you do not
understand the directions I saw in your work or those I suggest for the
revision, a face-to-face or phone conversation is the obvious next
step--written comments have definite limitations when writers and readers want
to appreciate and learn from what each other is saying and thinking. Please
talk to me immediately if you do not see how you are benefitting from the
"Revise and resubmit" process.
I keep carbon copies of my comments, but when you submit revisions, please
resubmit the previous version(s) with my marginal notes. Please revise and
resubmit promptly. The yield for your learning is lower if you are no longer
thinking about what you were at the time you wrote. I continue to request
revision, not until a certain standard is reached, but as long as the
interaction can still yield significant learning.
A minimum of two in-office or phone conferences are required. I want to reduce
the chance that you avoid dialogue around comments on written work. Through
such dialogue profound issues are sometimes opened up about one's relationship
to audience and influencing others.
Read chapters 3 and 13 of Peter Elbow's Writing With Power (Oxford U.
P., 1981) for a wealth of insight about the processes of sharing written work
and revising with feedback. My website also has links to some Notes on writing
and revising, including Freewriting suggestions (.../698-99p.html).
I encourage you to make use of class meetings and the list of others students'
phone numbers to arrange pair peer sharing and commenting according to whatever
terms you pre-arrange. This will enable you to expand the kinds of readers to
whom you are responding and to avoid a common trap, namely, writing as if the
reader is the professor who knows enough about your topic and thinking to fill
in what isn't explicit or clearly stated.
In parallel with dialogue around comments on your written work, making notes on
readings can be thought of as an active dialogue with others who are not
physically present. Such dialogue helps you to think deeply about ways that
the information you are reading, listening to, or writing about connects with
and perhaps alters your course project and your work more generally.
Rationale for the Assessment system
The rationale for not grading the different assignments and granting an
automatic B+ for 80% satisfactory completion* is to keep the focus on your
developing through the semester. It allows more space for students and
instructor to appreciate and learn from what each other is saying and thinking
(see Learning from dialogue above). My goal is to work with everyone to
achieve the 80% satisfactory completion level. Students who progress steadily
towards that goal during the semester usually end up producing work that meets
the criteria for a higher grade than a B+ (see rubric in syllabus).
* Satisfactory completion for written assignments means you must submit
the assignment, revise in response to comments, and resubmit it promptly until
marked OK/RNR = "OK, Resubmission Not Requested." For the final report to be
OK/RNR, you must have revised in response to comments on the draft. I
sometimes request revise and resubmit on final reports. If not enough time is
left for revisions, I submit an incomplete grade or, if you specifically ask me
to do this, calculate and submit a final grade without an OK/RNR for the
report.
Before, during, and after class--Critical thinking about course readings
& discussions
CCT courses aims to help students become reflective practitioners (or
"practicing reflectors"). The most important goal of this course, therefore,
is that you actively ask questions about evaluating changes in educational and
social practices and policies--not just during class time, but all the time.
In this spirit, the class meetings are designed assuming that you will have
already done quite a bit of thinking, formulated questions, and connected the
week's topics to previous week's topics and to your own interests and projects.
Furthermore, after class you are expected to reflect on the class and integrate
new perspectives into your notes, preparation for subsequent classes, and your
developing projects. Various components of the course are intended to
contribute to this reflection/critical thinking:
1. A Journal with weekly responses and notes on exercises, projects,
readings, class discussions, and clippings. Through writing in your journal,
you will be better able to weave the course material into your own thinking,
and to bring your own thinking into class activities. In preparation for
class, you might write in your journal a commentary on readings, or, after
class, review the readings and the class activities. In either case explore,
when appropriate, the relationship between your project/ interests and the
readings/activities.
Journals will be collected for perusal twice during the semester. Bind
together pages with post-its or otherwise indicate which bits you do not want
me to look at. I want to get an overall impression of your developing process
of critical thinking about course readings and discussions.
2. Clippings. To keep up with evaluations of changes in educational
and social practices and policies, compile a packet of clippings and xeroxes of
articles from newspapers, magazines, journals, and websites. These should be
affixed onto 3 hole punch paper, so they can be collected in a binder, which
will be kept in the entrance hall to the Department of Counselling and School
Psychology so everyone can have access to. Write the full citation on each
article, unless it is already included. Use post-its to indicate where more
critical thinking is needed and to add your own reflections on specific points.
Submit the clippings twice at the same time as the journal is reviewed.
3. End-of-semester Process Review. At the end of your journal include
a) a reflection (1-3 paragraphs) on the process of development of your thinking
and work this semester and b) 4-6 exhibits--not necessarily your best
products--that demonstrate this process. Exhibits may be journal entries, free
writing, drafts, etc. Explain your choices through annotations (post-its are a
good way to do this) or in the reflection. One xample of a Process Review
(a.k.a. portfolio) is included at the end of the Gudie to Classes, etc. Ask me
if you want to see other examples.
4. End of semester Evaluations. I devote the whole of the last class
to "taking stock":
a) to feed into your future learning (and other work), you take stock of your
process(es) over the semester;
b) to feed into my future teaching (and future learning about how students
learn), I take stock of how you, the students, have learned.
Standard evaluation forms are not very conducive to taking stock, so we will
design another evaluation form for you to complete.
Communication before, during, and after class
The limited class meeting time means that we have to a) use the time
efficiently, and b) keep lines of communication open out of class. The
following practices should help:
Email or call me during the week if you see a problem in the readings (e.g.,
missing pages), the instructions need clarification, etc.--especially when
others might share your concern.
Arrange to have time on campus when you can do library research for your
projects or consult with me during office hours. For people who have arranged
a back-to-back class schedule, this will probably mean visiting campus on
another day as well as the day of classes.
We'll start class on time. Late-comers should quietly but firmly join
us--don't take a seat at the back or off to the side.
Build relations with your classmates--a lot of learning and opportunities for
clarification can happen when you talk and share work with peers. This will
also allow you to find out what happened and to get the handouts given out if
you miss a class, and so you'll be able to prepare and participate actively in
subsequent classes. The break mid-class, for which we take turns providing
light refreshments (see sign-up sheet), is a good opportunity for connecting
with others.
Remember to drop off and collect written work on your own from my in/out trays
before you leave class. This gives me more time to set up the class and talk
with you before and after class.
If you are not ready to submit an assignment or revision on the due date,
submit a note about when you plan to do so. I am flexible about extensions,
but I need to know that you are keeping track of your work, not simply falling
and feeling behind.
Complete the Critical Incident Questionnaires (CIQs). These enable me to
identify areas that need to be addressed explicitly or revisited via email or
in class the following week, or taught differently in the future.
Give yourself a chance to digest comments on your assignments, and don't try to
squeeze in a discussion on them when we're in a rush or otherwise distracted.
Instead, use office hours, phone calls, and email.
Later in the semester, when you're concentrating on your own projects, you
might establish a daily check-in with a live or phone buddy to ensure that
you're doing what is essential and not simply doing what has accumulated on
your list of things to do. And to help you balance the divergent and
convergent aspects of the research and writing process.
Office hours and phone conferences
Office hours for in-person (W 2.143.09) or phone conferences
(617-287-7636): M 2.-3.30, or by arrangement
Priority is given to those who have signed up for an appointment.
If these times don't fit, please leave a voicemail with times I can call you.
Email: I respond daily to email. If you have a problem that other
students may share or a general comment send the message to me with "for
CCT693" in the subject line and it will be forwarded to everyone in the course.
I can handle email attachments, but prefer to comment on printed copies of
assignments.
To setup a UMass email account, get your student ID and then go to UL in Healey
to sign up. If you want to use UMass as your link into the internet (a.k.a.
Internet Service Provider), go to the Help Desk in Computer Services in the
Science building and ask to set up a PPP account. You'll need to know the
operating system of your home computer, the speed of your modem, and your UMass
email account (user name) and password. Make sure you get the PPP software and
persist in asking for help until you have the PPP software working. Note: a
PPP link is no longer needed to use the library's databases. Instead you can
use the proxy server at the bottom of the library's web page
(www.lib.umb.edu).
If you don't have a home computer or don't have a modem to hook up to the
internet, you can still access your email via the WWW at your local library,
cybercafes, etc. One way is to get a free web based email account (e.g., at
www.hotmail.com) and the other is to access your UMass account using a web
based interface (e.g., webinbox.com). Mail can be automatically forwarded from
your UMass account to another account. Ask the Help Desk in Computer Services
to help you set up forwarding.
Email protocols:
Confirm receipt of emails so I know they've got through.
Download and read emails carefully--don't respond quickly while you're logged
in and paying for each minute.
Don't send a message with emotional impact until you've slept on it.
Don't send a message when it's a way to avoid talking or if it would be better
to talk.
"For CCT693" emails:
-be nice to each other--no flaming, no sharking, lots of praise and
constructive suggestions;
-stick to business = the course, educational and social change, and its
evaluation; and
-never quote anything from email outside the class without permission from the
submitter.
Basic course protocols/expectations
1. Make time, at least 6-7 hours/week, to work on the course outside class,
especially when you have to collaborate with others on projects. Preferably,
set aside clear block(s) of time to do this.
2. Be responsible about class activities and collaboration (pre-reading,
attendance, arrival on time, discussion, contact about non-attendance and late
work)--don't wait for me or others to check in with you. If you miss a class,
arrange to find out what happened and get the handouts given out so you can be
prepared to participate actively in subsequent classes.
3. Read guidelines and rationales given in this course packet and in other
handouts. The class meeting times are often too short to explain everything
(see section above on "Communication before, during, and after class").
4. Use the 80% requirement in the assessment system (see above) to drop some
assignments and miss some classes when you need to accommodate to competing
demands from work and life in general.
5. Do assignments on a wordprocessor so you can revise them readily. Resubmit
assignments when requested, responding to comments from me and other students.
Submit assignments and revisions on due dates, or submit a note about when you
plan to do so.
6. Bring journal/workbooks to every class, to draw from or write in during
in-class activities.
7. Arrange WWW access and get an email address, either through UMB or a place,
e.g., your local library, where you can use a web browser and access email
during the week between classes.
8. Make suggestions about changes and additions to the course activities and
materials. Support me as I experiment in developing this course (see section
Evaluation and On-Going Devel. of this Course).
9. Arranging an Incomplete and making it up: If you anticipate not finishing
the assignments you wish to finish (allowing for my comments and your response
to them), you must prepare a schedule of when you plan to complete them. I
will discuss this with you and arrive at a contract, signed by both of us at
the last class meeting. Although I want everyone to complete the course, my
experience is that indefinite incompletes are a drag for all concerned. Little
learning takes place if and when the unfinished work is completed.
If either component on your grade is below C at the end of the semester, the
incomplete contract will require you to attend the course the next time it is
offered--there is no substitute for the development of teaching/learning
interactions that happens in a class over the course of the semester.
Guide to Classes, Assignments and Readings
Questions for Gallery Walk -- CCT693, Class 1
1. What changes (big or small) would you like to see pursued in teaching,
schools, and educational policy?
2. What changes (big or small) have you personally been involved in trying
to make?
3. In what ways did anyone take stock of the effects of the changes in
2?
4. What's helped you make the changes successfully, and what's hindered
you?
Omitted from website: pointers about gallery walk
activity
Omitted from website: clipping on effect of smoking ban in
restaurants
Omitted from website: brochure of "important information" for phonografix
approach to teaching reading
EVALUATION CLOCK
Case (educational practice/policy/intervention evaluated) =
points from clipping added Qs to consider
12. Why is evaluation needed?
________________________________________________________________________<>
1. Who wants/sponsors eval? For what decisions? [What could they DO
with the results?]
________________________________________________________________________<>
2. Key issues of evaluation?
________________________________________________________________________<>
3. Specific Qs to answer? [in the survey/questionnaire/pre- vs.
post-test]
Info to be sought & where?
________________________________________________________________________<>
4. Methods to gather info
for the evaluation?
________________________________________________________________________<>
5. Who participates?
In what situations is info gathered?
________________________________________________________________________<>
6. How is info analyzed to
produce evidence? [e.g., which statistical tests]
________________________________________________________________________<>
7. Who needs analysis? In
what form? Who interprets
evidence? [from evidence -> meaning]
________________________________________________________________________<>
8. What happened (re: the
thing evaluated)?
How to report that?
________________________________________________________________________<>
9. Why did that happen?
________________________________________________________________________<>
10. Lessons learned by
sponsors of evaluation?
________________________________________________________________________<>
11. What do they do differently?
Critical Incident Questionnaire*
Please take about 5 minutes to respond anonymously to each of the questions
below about tonight's class. Keep one copy for yourself and put the other by
the door as you leave. I'll digest the responses, report back to you next week
about them, and try to make changes to respond to your responses.
1. What incident/comment/reaction/quote stands out from tonight's
class?
2. At what moment did you feel most:
a. engaged with what was happening?
b. distanced from what was happening?
3. What action that anyone (teacher or student) took did you find:
a. most helpful or affirming?
b. most puzzling or confusing?"
4 (Optional). Other comments?
*Adapted from Brookfield, S. D. (1995). Becoming a Critically Reflective
Teacher. San Francisco: Jossey-Bass Publishers, 115 (on reserve).
Critical Incident Questionnaire, version 2*
Please take about 5 minutes to respond anonymously to each of the questions
below about tonight's class. Keep one copy for yourself and put the other by
the door as you leave. I'll digest the responses, report back to you next week
about them, and try to make changes to respond to your responses.
1. What concrete incidents/comments/reactions caught your attention?
2a. What excited you?
b. What frustrated you?
3a. What trends do you see emerging?
b. What are the implications of this for yourself?
4a. What are your next steps?
b. What support do you want in taking those steps?
* For use later in the semester when trends are emerging. Adapted from
previous CIQ in light of "Sense-making" theory (see p. 37 of course packet)
Assignment 1 Evaluation Clock
In preparation, first read and make notes on the Evaluation Clock from the
Evaluation Sourcebook (see next section of Course packet). Then, choose
from the set of clippings that follow one that interests you. (Some come from
areas other than education.) Fill in as much of the Evaluation Clock as you
can using information from the clipping. Use loose Evaluation clock sheet for
the assignment and continue on the back if you run out of space on any item.
Notes:
1. For the Evaluation Clock, the case is the evaluation, not the
educational or social change at stake. If the clipping doesn't involve an
evaluation, but suggests a need for one, then invent what you think should be
the answers to the 12 questions of the clock. When answers to steps in the
Evaluation Clock are not specified in the clipping, invent them (either design
them or imagine an outcome).
2. In the right hand column add questions you have about the evaluation
both:
as a critical thinker (Why didn't they do X instead?); and
as a learner (I would like to know more about that fact, technique,
organization, etc.).
You might also want to note, but in a separate place, the questions you have
about the educational issue itself.
3. This is a clock, a circle, not a strict chain. If your answers to one
question require/ allow you to revise/ refine an earlier one, that's to be
expected.
4. What is the specific evaluation (or evaluations) the clipping is based
on?
If more than one, choose one to focus on. Note that different kinds of
evaluation are routinely conflated by teachers, administrators, and policy
makers, namely, evaluations of:
students' knowledge;
the new curricular frameworks as a means to improve students' knowledge;
the performance of teachers;
the performance of schools; and
the performance of school districts.
For this assignment, separate the diferent kinds of evaluation in the clipping,
choose one or more of them to analyze, and complete a separate clock for each
one you're analyzing.
Omitted from web version: Excerpts from Pietro, D. S. (Ed.) (1983).
Evaluation Sourcebook. New York: American Council of Voluntary Agencies
for Foreign Service, p. 22-23 (on eval. clock) and p. 12-17, & 21 (to
provide context and introduce you to the rest of the
Sourcebook).
Omitted from web version: Example of completed Evaluation
Clock
Omitted from web version: Clippings set 1
Assignment 2 Precis of chapter from Evaluation
Think of your precis as summaries that give other students a quick start if
they were to address the topic of your chapter. This assignment allows the
course to cover more topics than is possible in classtime and allows you to
practice helping others address the explosion of information.
The precis will be more useful if readers do not have to go and read the
chapter, so, for example, rather than saying "there are 6 ways of flushing out
goals," list the 6 ways.
The first version of precis should be submitted by email as well as on paper so
I can send comments by email. You should revise in response before we
distribute copies to everyone in the class. See an example of a precis from
last year (below) or the full compilation on reserve.
"Briefings" on various topics related to research and engagement, prepared in
the same spirit of giving other students a quick start on the issues, can be
viewed at http://www.faculty.umb.edu/peter_taylor/briefings-TOC.html.
Assignment 4 Questions & comments after reading Evaluation
Once you have digested one chapter thoroughly, and discussed
another with a fellow student, you should be in a position to read the rest of
the book "actively." That is, to decide what chapters to read word for word,
and what to read for overall ideas (e.g., via intro sections, headings, and
conclusions). More importantly, you should approach your reading and
note-taking as an active dialogue with someone (the author) who is not
physically present, thinking deeply about ways that the information you are
reading connects with and perhaps alters your intended course project and your
work more generally.
In this spirit, assignment 4 is to submit 5 statements or questions from your
reading of Evaluation (excluding the chapter for which you made a
precis). These should be expressed so that other students can appreciate the
context of your statements and questions--Imagine that you were going to kick
off a class discussion with each statement or questions.
An Overview of Practical Evaluation --
a briefing on chapter 1 of Patton, M. Q. (1982). Practical Evaluation.
Newbury Park, CA: Sage
Sections
I. Evaluation definition
II. Mandate: Evaluation must be practical
IIA. Alternatives to practicality
IIB. Components of practicality
I. Evaluation definition
1. systematic collection of info
2. about a topic incl. activities, characteristics, outcomes or personnel,
programs, policies, products
3. for use by specific people
4. for some purpose incl. reduce uncertainties, improve effectiveness, make
decisions, etc.
II. Mandate: Evaluation must be practical
IIA. Alternatives to practicality
prior to about 1975 methodological rigor and accuracy
emphasized
but the resulting evaluations tend to be impractical in that the process:
* takes too much time, $$, administration
* requires artificial controls (limiting program change, selection of subjects,
etc.)
* uses questions that are difficult to answer
* is politically unacceptable
IIB. Components of practicality
1. Utility (of product) concern for how specific decision makers
will use results
2. Feasibility (of process)
* politically anticipation of various interest groups, obtaining their
co-operation & forestalling potential obstructions or misuse of results
* practically minimum disruption to participants' lives
* $$ sufficient value to justify resources expended
3. Propriety (of process) fair & ethical
and then
4. Accuracy (of product)
Combining all four requires
1. situational responsiveness
(vs. alternative: standard procedures and use of deeply embedded implicit
rules/ heuristics)
This requires awareness of these procedures/rules and
willingness to risk trying something different
& depends on grounding & refreshing in fundamentals =
writing proposals, working with evaluation task force, clarifying goals,
design alternatives, questionnaire construction, interviewing, data analysis,
making recommendations, etc. (p. 24)
2. methodological flexibility
3. multiple roles for evaluators e.g. training sponsors in evaluation and use
of info
4. political sophistication incl. self-consciousness about style
5. creativity to operate under multiple constraints
Peter Taylor (peter.taylor@umb.edu)
Omitted from web version: Stanfield, B. (Ed.) (1997). The Art of Focused
Conversation. Toronto: Canadian Institute of Cultural Affairs, 6-29.
Assignments 3 & 5 Mini-project
Goal: Investigate the the evaluation of a topical educational
"intervention" (a change in curriculum, organization, policy, etc.) or propose
a (better) design for evaluating the intervention.
This mini-project builds on the evaluation clock exercise, but takes it further
in that you will need to find out more about the case, either through
locating and talking to relevant people or through library research. Even
if you don't complete everything you think is needed for the topic you
investigate, you should get a feel for what is in store when you work on your
own evaluation projects.
Steps (see syllabus for deadlines)
i) Read through possible cases (from sets 1 and 2) and choose one, or
add one of your own.
ii) Use the evaluation clock framework (see Assignment 1) to fill in as much
information from the clipping/description as you can (= assignment 3);
iii) Investigate the case, either through locating and talking to relevant
people or through library research (ask PT for leads & guidance); and
iv) Prepare a report on the case (= assignment 5). These reports should be
summaries that give other students a quick start if they were to address that
issue or topic. When composing the report think about what you would have
liked to have known when you started investigating the case. See sample from
last year in this course packet and the collection on reserve for possible
models.
Note: The mini-project issues have many dimensions and connections to
other issues. The newspaper clippings are intended as an entry point into each
case, so you know its significance in the world and be interested in looking at
the evaluation that set of the debate. The focus of the mini-project, however,
should be on thinking about the evaluation(s) of educational change that
are involved in the project (or that need to be involved). If your issue
concerns student assessment or testing, that is not the evaluation of
educational change that is involved; instead it's the evaluation of the test.
One caveat: In the Evaluation Clock handout I ask you to add questions you
have both as a critical thinker (Why didn't they do X instead?) and as a
learner (I would like to know more about that fact, technique, organization,
etc.). You will have more of these questions a) the more carefully you examine
the specific evaluation and b) the more you learn about the larger range of
positions and proponents around the case. So do not ignore b), but don't let
it distract you from a).
Case studies, Set 2
(Clippings and articles mentioned below are available from PT to copy and
return)
Possible cases and initial leads
Note: To allow you to choose a case that suits your interests, these
cases range across different levels--from specific classroom or workplace
setting, to educational policy testing, educational institutional change, and
other kinds of social policy/politics.
1 Barry Kaufman's and his wife's work with their autistic son and other
hard to reach children. Start with clipping from Utne reader. Borrow (or
order via ILL), Son-Rise and Son-Rise: The Miracle Continues.
View website www.option.org. Call 413-229-2100, fax 229-8931, or email
raun@option.org to request references that systematically evaluate their
efforts. This may be a sensitive issue for them, because standard evaluations
may not be sympathetic to their method. Locate other sources on their work,
perhaps critical. You may have to design what they think would be a
sympathetic but fair evaluation of the Kaufmans' interventions.
2 Third International Mathematics and Science Study (TIMSS) relased in spring
1998 by US Dept. of Education National Center for Educ. Statistics. To
appreciate the importance of this, use Lexis-Nexis to get a representative
sample of responses in the media. Locate a get a copy of a 1999 article in
Science that defends the study against the criticisms in Rotberg (1998) (
Science 280:1030-1031 -- I have a copy of this).
3 C. H. Rossell and K. Baker (1996) "The effectiveness of bilingual
education," Research in the Teaching of English, 30, 376-385 has been used by
opponents of bilinguial education. Cummins, J. (1998). "Rossell and Baker:
Their case for the effectiveness of bilingual education." Currents in
Literacy 1(2 (Fall)): 10, 11, 31 (see xerox) critique their study. Use
Lexis-Nexis to get a representative sample of responses to Rossell and Baker in
the media. Locate other critiques of this work.
4. Gerald Coles, Reading Lessons: The Debate Over Literacy (Hill &
Wang, 1998) reviews the "reading wars," i.e., highly politicized debates over
methods for teaching reading. Borrow and extract key references to research
evaluating different approaches (see also clipping from Education Week 2 Dec
98).
5 Tenure is granted to tenure-track faculty granted after an extensive review
of their work, supposedly not as a reward for that work, but because that work
indicates that they will continue to contribute in research, writing, teaching,
and other ways of serving the institution and society. How good an
indicator/predictor/guarantee is the tenure review process of continued
contributions?
6 The CCT Program joined the Grad. College of Ed. a few years ago and became
part of the Department of School Organization Curriculum and Instruction
(SOCI). SOCI has standard end-of-semester course evaluations forms that
students fill in. In what ways and how well do these forms contribute to
faculty improving their courses?
7 President Clinton announced in his 1999 State of the Union address that he
wanted schools to end "social promotion." The implication is that it is better
for failing students to repeat the grade (see clipping).
8 The Graduate School at UMass Boston consists of many graduate degree
granting programs. Many of these programs are located in the Grad. College of
Ed. (GCOE). Some years back the Dean of the GCOE divided GCOE into three
departments--School Organization Curriculum and Instruction, Counselling and
School Psychology, and the Doctoral Programs (a.k.a. Leadership in Education).
GCOE Programs continue but, with one exception, are now part of one of these
departments. What effects was this change in the organization of an
educational institution (namely, GCOE) intended to achieve? How successful has
it been in achieving those effects?
9 Critique of remedial reading programs by Diane McGuiness in Why Our
Children Can't Read, and What We Can Do About It. New York: The Free Press,
1997.
10 The program criticized the least by McGuiness (see #9) is the Wilson
Reading System. Nevertheless, the Phono-Graphix approach promoted by
McGuiness' son and daughter-in-law appears to have several advantages over the
Wilson system--success rate, speed, simplicity (see articles on both). What
kind of evaluation might lead Wilson teachers to appreciate and incorporate the
insights of Phono-graphix?
11 In Excellence and Equality, David Fetterman recommends "the
reauthorization of a national center for the gifted to revitalize gifted and
talented education and in the process serve as a beacon of educational
excellence for the entire educational system" (p.xv). In Playing
Favorites, Mara Sapon-Shevin argues that the construction of the category
of giftedness not only secures these students unequal resources, but disrupts
the sense of community and diversity that she considers essential for
educational progress.
12 The Educational Testing Service (ETS) has developed an electronic rater
(E-rater) of essay questions and will use this in the Graduate Management
Admission Test. Kaplan has already prepared tips to help test-takers to
impress the E-raters (see clipping).
13 The Massachusetts Comprehensive Assessment System (MCAS) is supposed "to
improve the quality of teaching and learning... by identifying students and
schools that may need help meeting rigorous education goals" (S. Gorrie, MTA
Today, 30 Nov. 98). What evaluations, evidence, and theory support the idea
that student tests lead to effective changes in teaching and learning?
14 Seymour Papert developed LOGO in the 1970s and in his 1980 book,
Mindstorms, proposed that computer technology could lead to fundamental
reforms in education. What were the outcomes of using LOGO in elemntary
schools?
15 In October 1998 Harold Wenglinsky of the Educational Testing Service
released his study on the relationship between educational technology and
student achievement in mathematics (see Education Week, 1 Oct. 98). The
results were mixed, and were cited in the media to support a range of positions
(see Lexis-Nexis). What can be said about the effectiveness of computer use in
mathematics classrooms?
16 Jane M. Healy's Failure to Connect: How computers affect our children's
minds is the work of a "techno-pusher turned critic." She warns against
technology co-ordinators making decisions, while leaving more experienced
educators out of the loop. What evaluations and evidence support this
recommendation?
17 Problem-based learning has been described as "one of the major success
stories in education since the 1970s" (D. R. Woods, Problem-Based
Learning, p. ix). What do evaluations and evidence tell us about how best
to get teachers to shift to PBL? (Borrow Woods' book from PT.)
18 Howard Gardner suggests that people are capable of developing strengths in
eight different approaches to learning and interacting with the world. What do
evaluations and evidence tell us about how teachers can best mobilize or
develop these multiple intelligences? (see info package from Robert Gates).
19 The effect of Headstart type pre-school programs on later indicators (e.g.,
high school graduation, employment, lessened criminal convictions, etc.) has
been brought to the attention of Congress and used to promote additional
federal funding. More detailed studies, however, have indicated that positive
effects depend on many other factors, such as, parental involvment in
schooling. What do evaluations and other evidence tell us about the design of
pre-school programs for the poor? (See clippings and Woodhead, M. (1988).
"When psychology informs public policy." Amer. Psych. 43(6): 443-454.)
20 Bob Gaudet of the McCormack Institute at UMass Boston analysed MCAS looking
at how much "value was added" by different communities' school systems, based
on socio-economic predictions of how the children should have done. (Contact
the McCormack Institute to get a copy of his report.)
21. "A promising new drug for depression, "MK-869," failed to clear efficacy
tests last year when it turned out that patients who recived a dummy pill had
done almost as well as those using MK-869. Science 284:238-240, 1999.
22. Of the six goals President Bush set for education for the year 2000, none
has been achieved. What does it mean for education reformers to set goals that
can't be achieved? (NY Times 22 Dec. 1999).
23. "Graduate student unions do not interfere with the ability of faculty to
advise and educate graduate students, according to a recent survey... conducted
by researchers at Tufts University."
24. The influence of SAT tests in College Admissions is declining according to
a news item in the Dec. 99 issue of NEA Higher Education Advocate. The
Princeton Review claims that the SAT "fails to measure student preparedness for
college." A new approach is taking into account such things as a student's
socioeconomic status, race & ethnicity, and the quality of the student's
high school (Oct. 99 issue).
25. In providing more than student aid, TRIO & GEAR-UP programs encourage
economically disadvantaged students to attend college.
26. "Reform after reform is being introduced [in education] with little
supporting research. An economist [Alan Krueger] argues that education should
be a science, not an art." NY Times Education Life Supplement, Nov. 7, 1999.
See also NY Times, April 20, 1999, p. C1 & C10.
27. Dissenting voices about the value of computers in every classroom. NY
Times, 17 Mar. 1999
28. "The debate about [the health and legal effects of] heroin prescription
[in European countries] should promote continuing assessment of the scientific
evidence underpinning current treatment, law enforcement, and prevention
policies..." Science 284: 1277-1278, 1999.
29. Diet and Blood Pressure: "The emphasis on sodium as the single dietary
culprit is counterproductive to our significantly reducing cardiovascular risk
for most of us and diverts attention from the issues we need to address."
Science 281: 933-934, 1998.
30. "Changes in public health campaigns, including those against smoking, and
medical advances have substantially decreased deaths" [from cardiovascular
diseases]. NY Times, 6 August 1999.
31. What is known about the effects of welfare reform on education and
children? MTA Today, June 7, 1999, p. 14.
32. The Edison Project, a for-profit school system, says students gain.
Others say the project is most significant as a marketing phenomenon. NY
Times, 7 April 1999.
33. "Learning alongside students of different racial and ethnic backgrounds
significantly enhances the educational experiences of law students, according
to a recent survey released by The Civil Rights Project at Harvard University."
Jan. 00 issue of NEA Higher Education Advocate.
34. "Teach Next Year boosts teaching and learning at Dorchester High School
and UMass Boston" University Reporter [U. Mass. Boston] Dec. 99.
35. What is the gap between teacher's and education faculty's views about
educational theory and practice? Dona Kagan (SUNY Press, 1993) observed both
groups and got them to respond to each other's accounts.
36. IMP (Interactive Mathematics Program) holds that "mathematics is far more
interesting to students when they get to do some real thinking." See Alper at
al. in Educational Leadership, May 1996.
37. Identify an evaluation that interests you from the following: "The Math
Education pages at the Math Forum are an information center for teaching
issues. Topics include adult numeracy, assessment, block scheduling, and other
reform topics." http://forum.swarthmore.edu
38. Identify an evaluation that interests you from the following: "A series of
discussions inspired by recent work on how students learn mathematics -- ideas
that form the basis of the NCTM Standards. This series is intended to be an
informative and sometimes provocative overview of the current research and
thinking of some key researchers in mathematics education and educational
psychology. Much of the work in this area indicates that the "traditional"
classroom needs to be changed if more effective learning (i.e., learning that
is more conceptual and less formulaic) is to take place."
http://forum.swarthmore.edu/~sarah/Discussion.Sessions/Discussion.sessions.html<>
Omitted from web version: A sample of a Mini-Project report (more on reserve in
Healey Library)
Strategic Participatory Planning
There are two goals for using the 4 stage strategic participatory planning
(SPP) process in classes 4 and 5:
1. To introduce you to techniques and have you experience them so that you
become interested in learning, experiencing, and using them more in
facilitating participation in envisioning and planning some
organizational/community/social/educational change.
2. For us, as a group, to plan the "umbrella" project of working out how best
to support each other to get competent and comfortable in evaluating and
facilitating educational change (subject to some constraints). This means
planning the remaining classes and work between classes, addressing issues such
as:
the sequence of classes;
which, if any, of the possible activities and readings to use (see descriptions
on the next pages);
the rhythm of each class;
how to support each other taking initiative;
monitoring when we revert back to conventional student-professor relations;
etc.
The parameters within which this planning takes place are that:
* we meet on the dates listed in the syllabus, with two classes of
Work-in-progress Presentations on Student Projects (presently scheduled for
weeks 9 & 10), a final "Taking Stock" class;
* each of you completes an Evaluation Design project, with stages and deadlines
as listed in the syllabus; and
* we operate with the assessment system (including the revise-and-resubmit
process) given in the syllabus.
The pages and worksheets to follow (from the ICA Facilitators manual) will be
used to lead you through the 4 stage SPP. As an example of an outcome of the
first stage of the SPP, the following are the headings given to clusters of
post-its during a "card-storming" workshop on the Practical vision for the
spring 1999 course.
"For this course to work well, it will be or will involve:
Co-operative project Heterogeneous
Diversity of views collectivity Reflection-in-collective
Reflection on ideas facilitates change Mobilizing
Change that liberates us Deep reflection resources
Going into issues in from self,
depth
Resources Practical class, and
Specific models models Learning from practitioners
Identifying of change Practitioners
agents
A vignette of strategic participatory planning
Date: Tue, 23 Mar 1999
From: jnelson@icacan.ca (Jo Nelson) [an experienced facilitator from ICA
Canada]
Subject: A Story
>Rob Edelmann wrote:
>Now that that's out of the way, what's happening in the world of
Facilitation? Anyone had an interesting
>discussion lately, exciting meeting, or innovative use of technology for
group discussion?
How about this one?
I had a wonderful trip to Iqaluit, on Baffin Island, five weeks ago. I was
working with 22 Inuit school leaders from across the new territory of Nunavut.
I arrived on the day of the first elections (the territory comes into being on
April 1 this year). The next day we focused on schools - actually we threw out
the word school because it has very Southern white associations. First they
did small group conversations (so they could talk in Inuktitut among
themselves) on successful learning -- looked at their own learning, and
extracted the keys to learning in an Inuit cultural context from that. In the
afternoon we did a workshop on "What would a "place of learning" look like that
supported Inuit culturally appropriate learning. For the first time they began
to pull together their images of Inuit cultural learning and academic learning,
and think about how you might teach both kinds of knowledge in a culturally
appropriate way. For example, teaching math by demonstrating a number of
times, then the students copying the adult. (This was one of their keys to
successful learning.) Or going out on the land to study astronomy.
The second day they did small group conversations on successful leadership --
traditional leadership on the land, in the communities, in schools -- and drew
out the keys to Inuit culturally appropriate leadership. Then we did a vision
workshop on what they want leadership to look like in a place of learning that
supports Inuit culturally appropriate learning, then obstacles and strategies
on the third day, ending with personal commitments.
I got comments back afterward like "I know now my dreams can come true" and "I
have courage to be a leader again". It was an utter privilege to be a part of
a society that is standing at a new beginning point to forge out what it means
for them to be truly human in the 21st Century!
The task now for the project is to sustain the hope and momentum as people are
spread out in the isolated communitites again. I hope to be a part of it
again, maybe to train people in the methods.
Jo
from the (mobile) desk of Jo Nelson
"Never doubt that a small group of thoughtful, committed citizens can change
the world. Indeed, it's the only thing that ever has." Margaret Mead
Try our web site <www.icacan.ca>
-------
Date: Fri, 28 Jan 2000
Postscript: The new Nunavut Department of Education asked us last fall to train their staff in facilitation methods so that they can involve communities in their decisions. This was a direct spin-off of the first program, as the Minister of Education was aware of and participated in part of the February event.
Jo
Some possible topics, activities, readings that might be woven into the
Strategic Plan developed for the course
Note: There is no expectation that the Plan will incorporate these topics and
activities.
TOOLS OF EVALUATION
Activity: Discussion with Brian White, UMB Biology Department, of White,
B. et al. (1999) "The effect of students' note sheets on exam performance."
Read the manuscript and formulate questions about design of educational
"experiments,' their statistical analysis, and finding people from whom to get
technical assistance re: design and analysis. Brian will briefly introduce his
project and respond to your questions. The goal of reading is not to
understand everything nor to say "I don't know enough to follow it," but to
formulate questions about the evaluation process at whatever level is
appropriate to you.
Activity: Presentation by PT on the basic logic of quantitative analysis and
getting help in undertaking such analysis. In preparation, write down any
technical questions you have from your reading or investigation.
Activity: Complete a small statistical study together to address
"arithmophobia"
Activity: Establishing a statistical support network
FACILITATING CHANGE IN GROUPS
Focused Conversations
Activity: Respond to Video on students complaining to teacher via a
Focused Conversation (and other means). In a focused conversation a neutral
facilitator leading participants through four phases--objective, reflective,
interpretive, decisional--with the same philosophy of participation as SPP.
Reading: Stanfield, B. (Ed.) (1997). The Art of Focused Conversation.
Toronto: Canadian Institute of Cultural Affairs, 6-29 (in course packet) and
30-48 (additional handout).
Constructivist Listening
Activity: Supportive listening (in pairs). Supportive or
"Constructivist listening," as described by mathematics educator, Julian
Weissglass, is practiced at the start of professional development workshops or
other meetings with the goal of helping participants think more clearly, make
better sense of experience, and listen well to others. This practice grows out
of "Re-evaluation Counseling," a form of peer counselling in which two people
take turns listening appreciatively to each other "re-evaluate" current and
past experiences, especially when they were young, of being hurt and not
helped. When such experiences come to interfere less with one's current
thinking and relationships, people more readily people take more initiative in
educational and social change and in their day-to-day relationships with
others.
Reading: Weissglass, J. (1990). "Constructivist listening for empowerment and
change." The Educational Forum 54(4): 351-370.
ACTION RESEARCH
School Change
Activity: Attend the session of the Ed. Admin. action research class in
which students report on their school change projects-in-progress (Thurs 6.45).
(Action research speaks to the principle of evaluating in process, rather than
all at once afterwards.) "One impression [from last year] was that real world
considerations led the students to shift their focus or topic quite a lot
before settling in on an educational practice to try to change and to evaluate.
Their questions weren't so well defined either. We shouldn't feel bad that we
(teacher & students) haven't got right to the formula for how to do an
effective evaluation."
Activity: Presentations by Panel of collaborators in a school change project
(former M. Ed. students), followed by questions and appreciations
Participatory Action Research
In Participatory Action Research (PAR) social scientists strive to
empower the people most adversely affected by some issue and allow their
inquiries to be shaped through on-going engagement with them. PAR lessens the
pressure to evaluate and get people to take the evaluation into account.
Activity: Discussion of readings
Readings: Taylor, P. (1999). "'Whose trees are these?' Participatory Action
Research in a Kenyan agroforestry project." ms.
Excerpts from Greenwood, D. J. (1998). Introduction To Action Research:
Social Research For Social Change. Thousand Oaks, CA: Sage.
POLITICS OF EVALUATION
(Why this intervention and not others? Why evaluate this intervention,
but not others? Selective interpretation of evaluations. Who discounts this
evaluation, but highlights others?)
Activity: Discussion of Woodhead, M. (1988). "When psychology informs public
policy." American Psychologist 43(6): 443-454. From the abstract: "The
potential of early educational intervention to make along-term impact on the
life chances of socially disadvantaged children has now been clearly
demonstrated... The evidence has been... frequently cited in support of public
policies on Head Start and early childhood programs in general... [L]ess
attention has been paid to the sense in which long-term effects are embedded in
a wider context of family, community, and school processes that affect children
not only at the time of intervention but also throughout their childhood
years..."
Activity: Discussion of Hunt, M. (1985). "The dilemma in the classroom: A
cross-sectional survey measures the effects of segregated schooling," in
Profiles of Social Research: The Scientific Study of Human Interactions.
New York: Russell Sage, 51-97. This case, in which the educational
practice being evaluated was segregated schooling, should serve as an
introduction to the politics of evaluation. While reading note the different
ways political pressures shape the origin, conduct and interpretation of the
evaluation, and responses to its findings. (The list in parentheses above is
just a beginning.) And also note the ways that educational changes in
educational practices can have unintended consequences because of on-going
social change and politics.
Activity: Visit by education scholar-activist who has evaluated the new Mass
teacher test and commented on the State's lack of evaluation of the tests.
THEORIES OF EVALUATION AND EDUCATIONAL RESEARCH
Action Research as an alternative to Positivist and Interpretivist
approaches
Activity: Discussion of reading
Reading: Carr, W. and S. Kemmis (1986). Becoming Critical: Education,
Knowledge and Action Research. Geelong: Deakin University Press., chapters
6 & 7 (up to p. 200)
From action research to Heterogeneous Re/constructions of Social
Situations
Activity: Make an intersecting strands diagram (that teases out
multiple, diverse resources) for implementing some change in an educational
setting. You could begin from your own position now and fill in your
background and projected future steps. Or you could begin from a general
teacher's position and add items specific to you on top of that.
The strands might be:
in the classroom with students
prep & planning
admin & funding
tech. & econ. devts.
critical thinking & social contextualization of the change
Reading: Taylor, P. J. (ms.) "Constructing Heterogeneous Webs in
Socio-Environmental Research."
Omitted from web version: Excerpts from ICA Facilitators Manual
Evaluation Design projects
Design an evaluation of a change/intervention in a specific classroom or
workplace practice, an educational policy, an educational institution, or a
social policy. A sequence of 5 assignments (initial description, notes on
work-in-progress, work-in-progress presentation, complete draft, and final
version; 2000-3000 words) is required.
If the educational change is not one that already happening somewhere, you will
have to spend some time researching and designing that change as well. If you
only have a general idea about the change you seek, it's difficult to design an
evaluation of it. A risk to watch out for is getting immersed in the issues
and neglecting the evaluation. Remembering that the evaluation is the
centerpiece of the project may help you see what aspects of the issue to focus
on, and what to put aside for future exploration.
If you are ambitious, you can go beyond designing the evaluation to carrying it
out, or carrying out a pilot version. Another risk to watch out for is jumping
in and carrying it out before you've designed it carefully. Even if you don't
carry out any evaluation, you should design it as if you were seriously
intending to. This means you'll have to imagine the situation in which the
educational change would be implemented, tease out what would be involved in
evaluating it, and think especially about what needs to happen in your
evaluation to get someone to use the results (ref: Patton and the Evaluation
Clock).
If you are really ambitious, you might pull together a group of stakeholders
and practice some of the facilitation of reflection and planning introduced in
this course. At the very least, I hope I interest some of you in doing more
training in the facilitation methods I myself are just learning/practicing with
you.
Stages in developing your project
Towards initial description:
- Sign-up for office hours or a phone conference to discuss your project
ideas.
- Compose an initial statement about your project (one or two paragraphs that
may, several revisions later, find their way into the introduction of your
report). The point is not to have your project defined straight away, but to
begin and then to continue the process of defining and refining it. In
preparation for writing this statement, I suggest making notes for yourself
describing:
your area of interest;
the specific case(s) you plan to consider;
the more general statement of the problem or issue beyond the specific case;
how you became concerned about this case/area;
what you want to know about this case/area by the end of the semester;
what action you think someone (specify who) should be taking on this issue;
what help you foresee needing in order to do the research; and
who the audience for your research report might be.
- Then you can begin moving around the Evaluation Clock defining what you
already know, what you need to investigate, and what can only be revealed by
the evaluation you'll be designing. I think of the clock as something you move
back and forwards among later and earlier steps while you are developing your
ideas about what educational change you envisage resulting from the evaluation
you're designing, who would be making that change, and what information you'd
need to present to influence them to change. The technical steps (3-6) can be
formulated more readily when the audience and focus of the evaluation are
clear.
- Begin background library, WWW, and phone research to find out who's done what
before/ who's doing what (through writing & action) that informs your
project.
- Read chapters 1 and 2 from Elbow on reserve regarding the interplay of the
creative and the critical in thinking and writing.
Work-in-progress notes: Pull together your thinking to date and present
it in a form that will elicit useful comments from me. The work-in-progress
notes might correspond to your thinking about half to two-thirds of the
Evaluation Clock. I suggest that you include in the notes:
A revised statement of your project
Annotated list of readings done or proposed, contacts made, and interviews
conducted**
Plans and timetable for their completion
A map of the larger suite of issues in which your specific one is embedded.
See the example on following pages from a 1999 Ed. Admin. course on action
research, an approach that is not so focused on evaluation, but shares the
concern with achieving effective educational change.
**The primary goal in asking for annotations is for you to check the
significance of the item against your current project definition and
priorities. Annotations, therefore, should indicate the relevance of the item
to your topic. Annotations also make you to a) compose sentences that may find
its way into your writing, and b) have your citations already typed in (use the
format/citation style you intend to use for your final report). Focus is more
important than quantity. Don't pack or pad this with zillions of references
you've found in your searches, but instead use the assignment as a stimulus to
your clarifying whether and in what ways an article is relevant to your
project.
Work-in-progress presentations: When students prepare for their
presentations, especially when they design visual aids, and when they hear
themselves speak their presentations, it usually leads to self-clarification of
the overall argument underlying their research and the eventual written
reports. This, in turn, influences prioritizing of research for the remaining
time. These presentations will necessarily be on work-in-progress, so you'll
have to indicate where additional investigation and clarification is needed and
where you think it might lead you. Aim for 10 minutes presentation, followed
by 5-10 minutes of discussion. We'll give feedback (see Sense-making feedback
on next page), but you may wish to design your own form of feedback
(evaluation). Earlier presenters can expect to be rougher than later, and we
should give feedback accordingly.
There may not be time for extensive discussion, so, to allow for more feedback,
the rest of us will use notecards to make suggestions.
I recommend using visual aids, of which the simplest to use are overhead
transparencies. Tips: include only key words or prompts to what you're going
to say; 15-20 words only on any sheet; text should be 1/2 inch high or more.
Get some transparencies and borrow pens from me, or bring material to copy
center to be xeroxed onto overhead transparencies. Return spare pens and
overhead sheets in the envelope outside my office door.
Drafts: The complete draft should reflect your thinking about the
entire Evaluation Clock. Some answers you will be getting from your
investigations, and others would only be revealed by the evaluation you're
designing.
Although this is a design project, not an answers/results project, a key thing
I still look for in drafts is whether you Grab readers' attention, Orient them,
and move through Steps so that they appreciate what you, the writer, has
proposed.
I recommend reading Elbow chapters. 4 & 5 on Direct Writing & Quick
Revising, then doing this for 90 minutes with the goal of completing an
extended narrative outline or short draft (say 4-5 pages). After completing
this, read Elbow section III on revising, and then prepare the draft of your
report.
After the complete draft is ready I ask you to pair up and comment on another
student's draft. Take Elbow's chapters 3 & 13 in mind when you decide what
approaches to commenting you ask for as a writer and use as a commentator. In
the past I made lots of specific suggestions for clarification and change in
the margins, but in my experience, such suggestions led only a minority of
students beyond touching up into re-thinking and revising their ideas and
writing. On the other hand, I believe that all writers value comments that
reassure them that they have been listened to and their voice, however
uncertain, has been heard.
Final report: These should be 1500-2500 words (plus references). For
the final project report to be counted as final, you must have revised in
response to comments on previously submitted drafts. Allow time for the
additional investigation and thinking that may be entailed.
Cite references consistently (an annotated bibliography is not needed). For a
guide on technical matters of writing scholarly papers, I recommend Turabian,
K. L. (1996). A Manual For Writers of Term papers, Theses, and
Disertations. Chicago: University of Chicago Press.
I will keep the reports to show future students, so please make another copy to
keep for yourself. I will send you a compilation of the final reports if you
send me a check for xeroxing and postage.
Omitted from web version: Map of "Our learning as anti-racist leaders" (from
Ed. Admin. action research course, 1999
Brenda Dervin, in the Department of Communication at Ohio State, has developed
a "Sense-Making" approach to the development of information seeking and use.
Given its diverse applications and implications (Dervin 1999) one might even
call it a paradigm. Her thinking leads to recommendations about forms of
response that presenters learn most from -- and listeners also. The format
below both acknowledges different voices and facilitates connections.
Ref: Dervin, B. (1999). "Chaos, order, and sense-making: A proposed theory
for
information design," in Robert Jacobson (ed.) Information Design.
Cambridge, MA: MIT Press. Also available (as of 21 May 1996) at
http://edfu.lis.uiuc.edu/allerton/95/s5/dervin.draft.html.
Presenter:
Commenter:
a) I appreciated...
b) I learned...
c) I wanted to know more about...
d) I struggled with...
e) I would have been helped by...
f) My project connects with this in the following way(s)...
g) I disagreed with...
h) I think the presenter should consider...
Omitted from webversion: Example of Process review
Evaluation and On-going Development of this Course
"I see my own teaching and advising as active and multifaceted
processes, involving constant monitoring and steps to improve my practices."
(From my application for the CCT position.) To involve you in this, I have
included here the official course evaluation from the 1999 course, a blank copy
of my own evaluation form, and a commentary that I prepared for a departmental
review of my first year's work. I also include the report of the research I
conducted during a Fall 1999 faculty seminar. That research focused on the
revise and resubmit system. In addition to continuing to develop my ability to
promote dialogue around written work, I plan this semester to work on the
problems of alternating b/w teacher & facilitator. This problem is
conveyed in the introduction to a workshop I will be running at the meetings of
the International Association of Facilitators in Toronto in late April:
Alternating between teacher and facilitator: An Exploratory Workshop
Participation in a group process or workshop is easily stifled when
participants perceive a facilitator to prefer some ideas and outcomes over
others. When insights emerge from the participants themselves, they become
more invested in the process and the outcomes. But what should teachers who
use facilitation techniques some of the time in their classes do to be
perceived as facilitating neutrally, not directing the class. Conversely, what
should facilitators of any kind of group do when they see that a crucial
insight is not emerging?
I have designed my IAF 2000 workshop so that the participants--including
myself--can learn from each other's experiences and insights regarding the
tension between facilitating and teaching (broadly construed), and the
difficulties alternating between the two roles. The workshop consists of a
series of activities for group interaction and intrapersonal reflection that
should bring the experience and insight of teacher-facilitators to the surface.
(A secondary aim of the workshop is that participants in their own work will
explore further the interaction and reflection activities used during the
workshop that were new to them.)
[For more details, see http://www.faculty.umb.edu/peter_taylor/IAF2000.html]
Omitted from webversion: Official evaluation from 1999
Omitted from webversion: Blank copy of 1999 evaluation
Issues in Educational Evaluation (CCT685) (now CCT693, Evaluation of
Educational Change) [excerpted from
http://www.faculty.umb.edu/peter_taylor/portfolio99-TOC.html]
Initial Goals
Although I had experience in social research and statistics, evaluation
of educational change was a new area for me as a teacher. I designed the
course so that I learn as much as possible by leading students to digest the
texts for themselves and for each other, coaching the students in
mini-projects, and facilitating participatory planning and other group
processes. This last aspect would serve two functions: the syllabus could be
adjusted according to students' background and interests, and students would be
introduced to the larger endeavor of working with other people in implementing
and improving educational changes. In this spirit, I chose texts that
emphasized the relationship between evaluator and sponsor from the formulation
of questions onwards needed if outcomes are to be taken up in changes in
practice and policy.
The mini-projects were based on clippings and short articles I had collected
concerning evaluations undertaken or needed.
I decided not to schedule a sequence of classes on quantitative methods but to
encourage students to formulate questions based on the articles they were
reading and to coach them in securing statistical advice from skilled
practitioners.
Challenges and Responses
The first challenge was that many students thought the course would
focus on assessment of students and some of them took several weeks to
appreciate the actual topic--evaluation of educational changes. In response, I
have arranged for the course name to be changed to Evaluation of Educational
Change. I also had to help several students sort out what is going on in the
name of testing and standards. Different kinds of evaluation are routinely
conflated by teachers, administrators, and policy makers, namely, evaluation
of: students' knowledge; the new curricular frameworks as a means to improve
students' knowledge; the performance of teachers; the performance of schools;
and the performance of school districts. Students also needed to more time
than I had expected to get the hang of the "evaluation clock," which divides
evaluation into 12 questions (from "Why is evaluation needed?" to "What do
sponsors do differently as a result?") and emphasizes re-visiting earlier
questions in light of later ones. I modeled this to students, but have
identified a simpler case to use for future introductions.
Time spent early on orienting students to the endeavor of evaluation led me to
eliminate the second mini-project on the politics of evaluation (what things
are subject to evaluations and what not, which evaluations are paid attention
to, etc.) and truncate this topic. The politics of evaluation is an important
issue, but CCT students, I have learned, are less interested in general
obstacles to innovation than they are in creating new educational approaches
and finding situations in which to try them out. At the same time, the
practitioners I invited to class emphasized the importance of collaboration and
patience in getting one's plans implemented, and students were very impressed
by these presentations. Indeed, it seems to me that the topic of evaluation is
best presented in terms of students becoming teacher-researchers or
practitioner-researchers. I began to consult with GCOE colleagues teaching
evaluation and action research to clarify whether to push this theme further in
CCT685, or whether to direct receptive students to action research courses
taught by other faculty. I plan to continue this consultation.
Some students were disconcerted for a while by the diversity of mini-project
topics and by the changes made to the syllabus as we went along, but from my
experience with this first class I believe I can in the future minimize
syllabus changes and anticipate potential sources of confusion. Some students
commented on the number of evaluation and group process tools I introduced and
suggested working with fewer in greater depth. At the same time, the final
projects indicated that students were trying out many of the tools in their
work even without extended training. I hope I can continue to introduce a
range of tools, because in the future at least some students will have done
other courses with me and so fewer of the tools will be new.
The method I used to acquaint students with the texts, namely, not lecturing
but having students prepare briefings for each other on the different chapters,
worked quite well. I realized, however, that it would be better to use a more
recent text and one with a focus on evaluation in educational contexts. I have
identified a strong candidate, Evaluation by Carol Weiss, but am still
reviewing possibilities.
Students raised fewer questions about quantitative methods than I had expected,
even when I assigned a quantitatively-oriented manuscript of a colleague and
had him come to class to discuss his work with the students. I have identified
among the clippings some cases that will allow me to convey when students need
to secure statistical advice from skilled practitioners.
Some students in this course (and also in CCT698 and CCT601) wanted a rubric
for A vs. A- vs. B+ grades and one student prepared an elaborate model. This
motivated me to prepare my own streamlined version and implement it my Fall
1999 courses.
Future Plans
I have mentioned some of my future plans in the preceding section. I
have a larger "to do" list stimulated by the formative and summative
evaluations of students in the course, and their participation in revising the
course as we went. My other major goals for the future are to:
--expand some of the clippings into well developed cases, especially in the
areas of science education;
--consult with other GCOE faculty with a view to differentiating the evaluation
and research courses we offer; and
--build on the mini-project of one student last spring to push for more
productive forms of course evaluation in GCOE.
From "dialogue around written work" to "taking initiative"
http://www.faculty.umb.edu/peter_taylor/citreport.html