Evaluation
Determining Impact
Evaluations of collaborative projects that focus on the outcomes of project activities demonstrate accountability. The results can also be used to garner support for program sustainability and funding, and demonstrate the effectiveness of the collaboration. Evaluations provide important information that enables stakeholders, partners, and others to learn from the successes and challenges of working together. Collective agreement on how collaborative success or effectiveness will be measured ensures that partners remain aligned in their shared vision and are held accountable for project outcomes.
What does Success Look Like?
Evaluating the worth of a bridge may be relatively straightforward; for example, counting the number of people using the bridge. Collaborative work is more difficult to measure but successful project outcomes and the number of people positively impacted would be starting points for review and evaluation.
Identify an Evaluation Process
The evaluation process begins with the identification of key, mutually agreed upon outcomes. Evaluation can occur at the completion of a project or at set times during implementation. The approach the Project team used, referred to as developmental evaluation, provided team members with an opportunity to respond and adapt as project needs evolved.
Developmental evaluation is an evaluation strategy that supports ongoing innovation and growth by providing real-time feedback in dynamic environments. In developmental evaluation, the goal is to actively shape the course of development by providing key insights to facilitate learning and inform decision making. Patton (2006) provides a helpful comparison of traditional and developmental evaluation, shown below.
Traditional evaluations |
Complexity-based, developmental evaluations |
| Render definitive judgments of success or failure. | Provide feedback, generate learnings, support direction or affirm changes in direction. |
| Measure success against predetermined goals. | Develop new measures and monitoring mechanisms as goals emerge and evolve. |
| Position the evaluator outside to assure independence and objectivity. | Position the evaluator as an internal, team function integrated into action and ongoing interpretive processes. |
For more information about developmental evaluation and to take a self-assessment, see:
http://innoweave.ca/en/modules/developmental-evaluation
Traditional evaluations |
| Render definitive judgments of success or failure. |
| Measure success against predetermined goals. |
| Position the evaluator outside to assure independence and objectivity. |
Complexity-based, developmental evaluations |
| Provide feedback, generate learnings, support direction or affirm changes in direction. |
| Develop new measures and monitoring mechanisms as goals emerge and evolve. |
| Position the evaluator as an internal, team function integrated into action and ongoing interpretive processes. |
For more information about developmental evaluation and to take a self-assessment, see:
http://innoweave.ca/en/modules/developmental-evaluation
Holly Christian, Northern Health (left), and Megan Klitch, Canadian Cancer Society, BC and Yukon Division (right), are sharing the results of the POWERPLAY program evaluation at the Inaugural Physical Activity Summit in Prince George.
Holly Christian, Northern Health (left), and Megan Klitch, Canadian Cancer Society, BC and Yukon Division (right), are sharing the results of the POWERPLAY program evaluation at the Inaugural Physical Activity Summit in Prince George.
Stories and Sharing
The evaluation component tested our ability to be flexible in the face of change. At times we struggled, such as when team leadership shifted. Fortunately, we had enough expertise in the group to continue with the evaluation. And that’s what made the difference. In addition, having a small, diverse, consistent, and dedicated working group was essential.
Evaluation Questions
Evaluation questions should be discussed early in the planning phase of the project and should commence prior to project completion. Project evaluations may require data from multiple time points.
Indicators of successful or effective health promotion collaboration have been identified in reviews of the current research.
Population health or project outcomes
Assessment of project end goals, and
Collaborative capacity
Assessment of how well the partners are working together, and functioning as a team.
Examples of evaluation questions that can be used to assess each of these outcomes are included below.
click to view the questions
Specific health or project outcomes
Was project implementation completed?
Did the project serve its purpose?
How have the project objectives been met?
Does the project have a legacy (e.g., policy change)?
Were behaviours of the target group influenced?
Were health indicators in the target population modified?
Have health services or programs been improved?
click to view the questions
Collaborative capacity
How did members participate in implementation?
How satisfied were members with the collaboration?
How did the partnership contribute to the objectives?
Do members perceive ownership of project outcomes?
Have members skills or knowledge been increased?
Did member costs justify the benefits?
How have relationships changed among members?
This word cloud displays the most frequent responses to a question about what the team felt needed more discussion. The larger the word, the more frequently it appeared in team members’ responses.
A Snapshot of Developmental Evaluation
As part of a developmental evaluation to assess efforts to work together, the Harmonization Project team completed individual anonymous online surveys to gather input about topics people may have been reluctant to discuss openly.
Stories and Sharing
A developmental evaluation process that was flexible enough to adapt to the real world circumstances of the project, and the competing realities of partner organizations’ different mandates was crucial. As we moved forward we were able to build on our successes and learn from challenges.
Strategies for Supporting Evaluation
A number of strategies can be used to make the evaluation process as smooth as possible:
Foster a culture of evalution
Reinforce the importance of creating opportunites for the team to provide input into evaluation strategies and plans. Ensure team members who join mid-project are aware of the project evaluation plan and how it relates back to the success of the whole.
Seek expertise
Identify evaluation experts during the team building phase.
Designate an evalution team
Form an evaluation team with a diverse subgroup of project members.
Clarify the purpose
Clearly convey to all project team members that evaluation is an essential component of project accountability.
Collect data
For evaluation, consider using a variety of tools such as questionnaires, interviews, and focus groups. Collect data at regular, pre-determined intervals.
Keep records
Diligent recordkeeping and note taking will help when reflecting on decision making and other collaborative processes. Ensure all data is stored securely and kept confidential. It is advisable to remove identifers when sharing data related to collaborative processes with team members.
Dissemination of results
Prior to dissemination, evaluation results should be reviewed by the project team, and vetted by the communication departments of partner organizations. At the beginning of the project, a clear understanding and agreement related to sharing the results of evaluations can be helpful.
Why Sharing Results Matters
There are a number of reasons why it is important to conduct an evaluation and to communicate results.
Demonstrate accountability
Funders, stakeholders, and executives from partner organizations will all want to know whether their contributions were worthwhile. Communicating results is a key component of accountability.
Continue to engage the target population
Feedback for the community and other groups that were part of the project may be needed in order to foster ongoing engagement. Participants often want to know the final results of a project. Results can be communicated within a community through newspapers, local news, community forums, or social media.
Contribute to greater knowledge
Evaluations of projects have the potential to contribute to the greater pool of knowledge on a given subject. When made public, contributions can assist and inform others.
Capitalize on results
Projects that demonstrate the success or benefit of collaboration can use results to advocate for continued investment or further collaboration on future projects. Alternatively, negative results can illustrate the need for a change in direction.
Useful Tools
Evaluation
- The Collaborative Assessment Tool is one example of a short activity that can be completed by members of a collaborative and discussed as a group. It is intended to help identify (as a baseline) areas that can be strengthened as well as provide a gauge of progress. This tool is available online: http://www.preventioninstitute.org/component/jlibrary/article/id-193/127.html
- The Wilder Collaboration Factors Inventory is another free online tool that includes 42 questions. Responses to these questions are used to calculate scores on 20 factors (such as leadership) that research has indicated are important for collaborative success. This inventory can be used periodically to assess how these collaborative factors are changing over time. https://www.wilder.org/Wilder-Research/Research-Services/Pages/Wilder-Collaboration-Factors-Inventory.aspx
- The Evaluation Plan Builder at Innovation Network can be used to guide the creation of flexible questions to fit real-life evaluation needs. Free online registration is required. http://www.innonet.org/index.php?section_id=64&content_id=185
Living Lab: Sharing Our Experience
One of the ways we made sure evaluation was included in our work was to weave activities and tools into the everyday activities of the project. Including evaluation questions in team meetings established a recorded and mutually shared set of data in the minutes.
We agreed to have a reflection question on every meeting agenda. That didn’t work too well at first because it was always last and often overtaken by more pressing matters on the agenda. However, we were soon able to work in evaluation as the primary topic for many key meetings, which provided us with valuable information.
Knowing that sensitive issues or critical incidents may not always be discussed during our meetings, we also provided other ways for our team members to share their experiences related to the collaboration.
Regular meetings were an important part of keeping the collaboration evaluation questions front and centre for the harmonization team.