Summative Evaluation:
Assessing Outcomes and Impact

This module is designed for individuals or teams who want to review an existing mediation program.  It is intended to help collect information useful for determining the program’s future directions.  Specifically, you will be able to: 

• collect information on the effectiveness of mediation for different types of conflict;
• gain insight into the performance of mediators during a mediation session;
• gauge satisfaction with and durability of mediated agreements;
• determine who is contacting and utilizes the services of your mediation program; and,
• identify areas for improvement.

The module focuses on three aspects of mediation services:  the disagreements, the disputants, and the mediation itself.  A questionnaire-based survey is the suggested format to be used to collect relevant information, although other approaches (e.g., focus groups, interviews) could be used.  Surveys are particularly useful for summative evaluation because they take little time to complete, do not require extensive training to administer, allow for the collection of information on a wide range of topics at once, and provide findings that may be summarized and presented easily and clearly.  Included below is a discussion of how and to whom to distribute surveys and some guidance on the analysis of the information collected. 

What is Summative Evaluation?

Summative evaluation is one of two broad categories of research.  [The other category is called process or implementation research.  See module II for details on how to conduct this type of evaluation.]   Summative research examines the outcomes of the program to see if they meet expectations.  It focuses on determining whether the program is meeting its stated objectives, whether it is worth continuing or expanding, and how effective it is.

How Do You Use the Tools?

You will find three research tools in this module.  Within each tool there are a number of suggested survey questions, which reflect current standards and practices in the evaluation of mediation.  They are intended to serve as a model or template for program managers; you may wish to adapt or modify them to more closely reflect the features specific to your program and/or the culture of your campus.  Although the questions are arranged by topic, please note that when you create a survey you may wish to intersperse the questions from the different topic areas rather than leaving them clustered by topic.  Each tool is preceded by a description, intended to help your team plan for the evaluation, that includes the following information: (1) the questions that can be addressed; (2) how to use the tool; and, (3) insight into what the results may provide. 

Be advised that in most college settings, collecting information from human participants requires that you submit the survey, your purpose for using it, as well as a description of what you plan to do with the data, to an Institutional Review Board (IRB) to get ethical clearance.  Check your university's IRB policies before beginning to distribute this survey.

Research tools in this summative evaluation module include:

1. Case Characteristics -- information about the type of cases and how they are managed.

2. Participant Characteristics -- information about who utilizes your mediation services. 

3. Does Mediation Work? -- information about the outcomes and process of mediation sessions and the performance of your mediators. 

   A note about survey distribution:  It is recommended that, whenever possible, program managers survey the participants in all mediated cases.  This relates to the issue of sample size.  If an evaluation project has only a handful of completed questionnaires to analyze, the themes or trends in the data reflect only a few participants' attitudes, and these may not be representative of the typical participant.  As a general guideline, evaluators should strive to get as many surveys completed as possible.  This provides a fuller picture of the effectiveness of your program. 

   Questionnaires may be distributed in person or by mail.  Each method has its benefits and drawbacks.  While it is most cost-effective to ask the participants to complete the questionnaire before leaving the mediation session, this may lead to responses which are influenced by the short-term emotional state of the participant, for example, feelings of accomplishment or feelings of exhaustion.  Another potential drawback stems from the choice of who distributes the questions -- if the mediator hands them out at the conclusion of a session, participants may be concerned that the mediator will see their responses and be reluctant to comment candidly about the mediator's services.  This may be addressed by having the mediator provide an envelope along with the survey, with instructions on how to return it to the mediation program manager.

   Questionnaires distributed by mail avoid the complications of those distributed in person, but are more costly due to mailing expenses.  When mailing questionnaires, it is strongly advised that programs include a pre-stamped, pre-addressed envelope in which to return the completed questionnaire.  A potential drawback of mailed questionnaires -- especially follow-up questionnaires administered a few months after the mediation -- is a low response rate.  A cover letter emphasizing the importance of the evaluation process will help to increase the response rate, but even with this letter, mailed questionnaires seldom receive the response rate that those distributed in-person receive.  Often, a reminder phone call or postcard will encourage responses from individuals who have not returned their surveys.

   An alternative approach is the use of online surveying.  There are many advantages to online questionnaires, including the savings related to mailing costs and the time required to process and compile the responses.  Once a mediation provider or education agency has a page on the World Wide Web, the development of an online questionnaire is not difficult.  A potential downside of relying on online resources is that some individuals may not have easy access to the Internet.

   A note about survey data analysis:  Most of the information collected by mediation surveys is attitudinal -- that is, the survey instruments are designed to gauge the perspective or feelings of the respondent.  A common form of attitudinal survey is the ordinal rating scale, which presents responses in some order like bad-to-good or dissatisfied-to-satisfied.  Social scientists usually order the responses from negative to positive as a means to ensure that a respondent is reading through the various degrees of opinion, instead of simply checking the first box.

   To facilitate accurate data gathering, surveys should have code numbers next to each response.  Without these numbers, recording survey results is tedious and prone to error.  In the sample questions presented in this module, note that every response has such a code: 

                                       [  ] 1 Very dissatisfied

                                       [  ] 2 Somewhat dissatisfied

                                       [  ] 3 Somewhat satisfied

                                       [  ] 4 Very satisfied

   The numbers themselves are only placeholders and do not have a numeric value in relation to each other.  Thus, a "4" is not twice as good as a "2", as demonstrated in the examples provided below. 

   There are numerous software programs available in which to record the data collected, although any database or spreadsheet will serve the same purpose.  There are also many ways to analyze the data from rating scales, some of which are complex and require a high degree of sophistication with statistical analysis.  As this module is primarily concerned with identifying opportunities for improvement, extensive statistical analysis will typically not be justified.  A simple analysis of tabulated responses will reveal those program areas that require attention.  Consider the following two response sets, based on a survey of thirty-four mediation participants.

   Example: 

For Question 1, the responses are mostly positive, with 30 respondents answering either "Somewhat satisfied" or "Very satisfied," and only 4 answering "Somewhat dissatisfied" or "Very dissatisfied." Contrast that distribution with the responses to Question 2, which yielded 15 responses of "Somewhat satisfied"or "Very satisfied," but 19 responses of "Somewhat dissatisfied" or "Very dissatisfied." The high proportion of negative responses demonstrates that the program area addressed in Question 2 needs attention and improvement.

<-Back to Eval Home -- Up To Contents -- Continue to Next Unit->