Program Evaluation Tools for Campus Conflict Resolution and Mediation Programs
-- Overview

Evaluation Packet Overview

Mediation has rapidly become a useful approach for addressing concerns on college and university campuses across the country.  It not only is an alternative to contentious and litigious dispute resolution methods, it also teaches valuable communication, listening, and problem solving skills.  The process of mediation holds great promise for assisting students, staff, faculty, and others in developing appropriate solutions to a wide range of concerns. 

Do you wish to start or expand a conflict resolution program on your campus?  Do you know if the service is needed?  Who should you target?  If you already have a program in place do you know whether things are being implemented as planned?  How many people do you serve with your conflict resolution program?  Are people satisfied with the results?

This packet is intended to address these types of questions and concerns.  The materials included in this packet were developed by a group of researchers and campus mediation program staff and volunteers that gathered together at Wayne State University in Detroit, Michigan, from April 4-6, 2001.  While we came from a variety of institutions of higher education -- large and small, public and private -- our shared aim was twofold:
(1) to improve the quality of campus conflict resolution program evaluation efforts, and;
(2) to increase the frequency of these efforts. 

Rather than develop a standard, fixed set of materials to be used by everyone, the group chose to develop a set of tools and approaches that can be adapted to meet the specific local needs of programs at various stages of development. 

To this end, the materials have been divided into four (4) modules, each of which provides insight into and guidance for assessing a different aspect of program development. 

These include: 

• Module I:        Needs Assessment
• Module II:       Process Monitoring
• Module III:      Mediation Outcomes and Impact
• Module IV:      Evaluation of Training

This evaluation packet is part of a larger effort to establish a web-based resource for campus-based conflict resolution service programs.  The Campus Conflict Resolution Resources website (www.campus-adr.org) provides information and links to, for example, bibliographic references, funding sources, mediator training, and conferences, in addition to resources for evaluation efforts. 

We need your help.  Your feedback on the usefulness of these materials is critical to their success (see packet evaluation form).  We also need samples of data collection instruments (e.g., surveys, interview questions) and of evaluation reports to include on the Resources website.

We also hope you will share some of the results of your evaluation efforts, either by posting new or modified data collection instruments you have developed, or some results from your local efforts to study and evaluate campus conflict resolution processes and services.  Look for more information on how to get involved by visiting the website (www.campus-adr.org), click on the Program Evaluation Office found in our online Conflict Resolution Services Building.

A note about terminology – Throughout this packet we use the terms conflict resolution and mediation interchangeably.  We also use program and service interchangeably when referring to conflict-related efforts.  Lastly, although we sometimes only mention college or university campuses, these materials are appropriate for use by programs on all institutions of higher education.

Why Do Evaluation Research?

Research is not often a high priority for people engaged in the exciting, challenging, and sometimes tiring work of setting up and maintaining a campus conflict resolution service.  The very term, research, conjures up images of the scientist in the laboratory forever worrying over some obscure concoction.  Nevertheless, research, particularly evaluation research, is an important key to long-term program success.  The still emergent field of conflict resolution in higher education will only truly mature to the extent that we begin to study and share broadly what works, what does not work, and why.  Programs that find ways to include research as a basic part of what they do are in a very good position to help move both themselves, and the field of conflict resolution, forward. 

Evaluation research can also be considered one of the ethical responsibilities of competent mediation service providers.  The Model Standards of Conduct for Mediators, jointly developed by the American Arbitration Association, the American Bar Association, and the Society for Professionals in Dispute Resolution in 1995, addresses this issue directly.  One of the standards, Obligations to the Mediation Process, indicates: "mediators have a duty to improve the practice of mediation."  This clearly implies a responsibility on the part of both individuals and programs to evaluate and refine the practice of dispute resolution. 

In general, evaluation research can serve quite a number of related purposes: 

Justifying and explaining the program – Providing credible proof to skeptics can help increase the sustainability of programs.  Evaluation data are often useful for justifying, defending, or explaining a program.  Sponsors may wish to know details regarding caseloads, types of conflicts, and outcomes.

Program planning and decision-making – Research can help determine how best to allocate time and resources now and in the future.  Data can also be used to refine public relations or outreach strategies or to make decisions about training needs.  Questions regarding duplication of services can also be explored.

Improving services – Research can help practitioners understand what is working and what is not and identify larger patterns that may go unnoticed day to day.  For instance, systematic follow-up contact with disputants might reveal that the compliance rate for certain kinds of agreements goes down a few months after mediation.  This finding could suggest needed changes in mediation procedures.

Addressing a specific problem area – Having good data on areas of concern can help the mediation program respond more effectively.  For instance, the director may have a concern about the limited use of the service by diverse cultural groups.  Part of the monitoring might include recording the ethnicity of parties using the service.

Assessing mediator needs and impact – Evaluations can be used to assess the needs, concerns, and effectiveness of mediators.  Questions relating to the effectiveness of the training, ongoing mediator professional development needs, and disputant satisfaction with mediators assigned to them or the process used can be quite helpful.

Identifying, at the start of a project, the purposes likely to be most important will increase the chances of gathering data that will be both credible and a good match for program needs.  It also provides the opportunity to develop standardized forms and practices that will endure as the program matures and office personnel change. 

Types of Evaluation Research

There are two broad categories of evaluation research: formative and summative.  Formative evaluation, also known as program or process monitoring, is used to keep track of how a program is being implemented, assess whether a project is going according to plan, document problems or new possibilities arising as the program grows, and provide useful strategic information for program developers.  Summative evaluation, also referred to as outcome evaluation, examines the outcomes of the program to see if they meet expectations.  It focuses on determining whether the program is meeting its stated objectives, whether it is worth continuing or expanding, and how effective it is.  Most programs end up doing some combination of the two.  It is preferable to gather formative information as the program develops, making adjustments as necessary, rather than proceed without gathering data until it becomes time to answer critical questions of continued fundability.

Information Gathering

There are many potential sources of useful information, some covered in this packet, and others left to your ingenuity and imagination.  Much of the information for formative evaluation comes from office forms (often referred to as monitoring records) that are set up and maintained by staff and volunteers.  These include, but are not limited to, telephone and office contact logs, case intake forms, and public relations activity logs that track events and the number of participants involved.  Organizing the data collected by a monitoring system can be accomplished in any number of ways depending on data processing capabilities and needs.  In small programs, a simple paper filing system and periodic (monthly or by semester) manual tabulation procedures might be adequate.  In a larger program, data might be converted for analysis on a personal computer, perhaps using a mediation case management software package or a general statistical or graphing program. 

In addition to data from monitoring records, programs may develop questionnaires, surveys, focus groups, and observation or interview guides to gather other kinds of information deemed useful for evaluation purposes.  Also, when permissible and available, secondary or archival data analysis of records prepared by related programs or services may be valuable.  For instance, it can be useful to note changes in complaint or case patterns at other programs or offices that in some way relate to the mediation effort.

Some examples of the kinds of data that program coordinators might be interested in collecting and related data gathering tools are listed in Table 1.  In each evaluation module in this packet we provide suggestions on how and when you might want to collect the relevant data. 

Table 1: Sample Data Gathering Needs and Sources

Desired Information	Possible Data Sources
Patterns of Referral	Referral forms Intake or request-for-mediation forms Case files indicating referrals made to other offices Post-mediation report forms completed by mediators Interviews with referral sources
Patterns of Program Use	Telephone and contact logs Demographic information gathered on intake forms Pre- or post-mediation questionnaires Key informant interviews
Mediators' Skill Development	Mediator observation forms (during training & mediations) Pre- and post-training questionnaires Post-mediation disputant satisfaction surveys Post-mediation self or co-mediator evaluation/debriefing forms
Scope & Extent of Outreach	Public relations (PR) event log Log of number, kind, location of PR materials distributed Web site "hit" counters "How did you hear about center" questions at intake Media clippings file
Program Efficiency	Time from first contact to resolution (from Intake form) Length of mediation session (from post-mediation report form) Percentage of cases reaching agreement (from case files) Costs per case - time, money, comparison to other processes
Disputants' Experiences	Post-mediation surveys Follow-up telephone interviews
Case & Problem Categories	Intake forms Case files Generic case summaries prepared by mediators

A note about participation -- All forms of data collection should be accessible to any mediation participant, so particular care needs to be paid to ensure that materials and methods are available in a variety of formats.  To ensure that evaluation is conducted appropriately and respectfully, interviews and questionnaires should be reviewed for accessibility to all possible respondents and users of these materials.  For example, you many need questionnaires available in large-print, Braille, and in languages other than English.   For interviews, ensure they are conducted at accessible facilities and interviewers are prepared to accommodate a wide range of languages and language skills as well as cultural contexts.  Regardless of the data collection method, sufficient time should be provided to allow for completion of any evaluation material. 

A note about confidentiality -- To ensure that participants provide open and honest responses to evaluation instruments, it is incumbent on program managers to safeguard the anonymity of the individuals and the confidentiality of their responses.  Provide assurance to participants on every written questionnaire and as part of every structured interview or observation.  Prior to undertaking any evaluation, agencies must determine which data to collect and how that data will be treated.  For example, is any of the material open to public scrutiny?  If the evaluation process calls for follow-up contacts with participants, how will an identifying code be developed and where will the list that links the code to specific cases be kept?  All policies related to the data collection effort should be made available to participants at every stage of the evaluation.

Using the Evaluation Results

After the various forms of data have been collected, they need to be analyzed and prepared for presentation.  Software packages for large amounts of quantitative (SPSS, SAS) and qualitative (The Ethnograph, ATLASti, NUD*IST) data are readily available on most campuses for use in data analysis.  Some specialized survey programs that help streamline data analysis are also available.  Excel is a basic, yet workable, option as well. 

Once the findings are compiled, strategies for reporting the results need to be developed that will appeal to the intended audience (e.g., sponsor, clients, mediators).  Several reports may need to be developed to provide different kinds and amounts of information depending on the audience.  If the information will be presented to the media, a press release may be needed as well.  Some readers like lots of charts and figures; others prefer text.  For most audiences, evaluation reports should include implications of the findings for: (1) program operation and maintenance; (2) expansion, redirection, and sustainability; and (3) recommendations for future short- and long-term actions that can be taken to improve the program.

It can be quite helpful to review a draft of the results of the evaluation with core staff, advisers, and volunteers before completing the report.  A meeting designed to discuss the findings can be held to get input and interpretations.  These additional perspectives can strengthen the final report and convey respect for the groups most likely to be influenced by the release of the report.  An evaluation is essentially a device for program staff and volunteers to use to make adjustments and improve program services.  Taken in this light, a report should not be presented as the final word on the program but rather as part of a continuous and evolving process of development and refinement.

<-Back to Eval Home -- Up To Contents -- Continue to Next Unit->