Considering evaluation as part of your engagement plans
- Caitlin Ryan
- November 30, 2016
- 6 minute read
At the core of citizen engagement is a commitment to the public to include their input in decision making and priority setting. Engaging the public means establishing a trust based relationship where citizens trust you will consider their input.
With citizens putting so much trust into your engagement efforts, doesn’t it make sense to consider accountability and transparency as a large part of your efforts? Similarly, how do you determine whether your initiative has been a success? Evaluation can help measure who you’re reaching, what they think and help determine if your initiative is making its intended impact. Without a well thought out plan for evaluation, your initiative can end up missing the mark; or even worse, erode the trust that citizens have in your organization and diminish the success of any future citizen engagement initiative.
While evaluation can be an iterative process, this post is more about providing you with some general tools and ideas rather than prescribing a ‘one size fits all’ evaluative framework. For a great read about the evaluation of digital citizen engagement check out this report from the World Bank.
Types of evaluations
Citizen engagement evaluations can be defined as either process or outcome based. Process evaluations evaluate how well the engagement initiative was run, while outcome evaluations evaluate the impacts of the engagement initiative.
Process evaluations evaluate the effectiveness of the public engagement process. Process evaluations do not evaluate whether the initiative had a discernible impact on decision making or priority setting, rather are an examination of the initiative itself. This type of evaluation is very important to ensure that the method of engagement was appropriate and that participants have the necessary information, opportunities and interest to participate. Process evaluations are also valuable tools to gain insight into how participants view their experience. These are important considerations for any organiser when organising future citizen engagement opportunities.
Process evaluations can be carried out a number of ways, but are best done through participant evaluation. Providing participants with a way of submitting a Likert scale survey or questionnaire is a convenient way of evaluating participant experience. It may also be important to include one or two open ended questions so that participants can provide more detailed feedback that can be used to gain even further insight into things such as user experience or reading level of engagement materials.
Below are some valuable considerations for any process evaluation framework:
- Representativeness. How often have you participated in a public meeting that concerns the whole community only to run-in into the same participants who seem to attend every public meeting? Representativeness is important in order to measure who participated in your initiative. For a good idea of why this is an important concept, check out this blog post on demographic profiling. Unrepresentative citizen engagement initiatives can often be the public’s number one complaint regarding engagement. Understanding who needs to be at the table means understanding who your decision will or can impact. Reporting on who participated in your community engagement initiative can often be an audacious task for organizers, however one that pays dividends and can lead to better participation in the future. Representativeness can be measured using many different characteristics, including gender identity, age, geographic area (very important for planning issues), or profession. Socioeconomic status can often be difficult to collect and is not appropriate in every case. However, indicators such as education level or neighbourhood can be less intrusive and are just as effective.
- Access and availability of appropriate resources. Without appropriate information, citizens may not be able to participate meaningfully in any citizen engagement process. Having accessible information at an appropriate reading level will also help ensure the involvement of a broader mix of citizens.
- Control. Control can refer to the fairness, independence and/or equitibatility of the process. This is important as it concerns the ideas of transparency and accountability, which are both fundamental to the practice of citizen engagement. This means any citizen engagement initiative should be run in an unbiased way, where participants are treated fairly and have an equal opportunity to provide input. This criterion also helps to measure whether the method used to engage citizens was appropriate in the project context.
- Objective and purpose. Clearly outlining the purpose and scope of the public engagement initiative should help to avoid any confusion about this issue, however it is important to ask participants how they felt about the objective or purpose in order to help find out if they felt the initiative was overly ambitious or fell short of their expectations.
Outcome evaluations can often be more resource intensive and taxing than process evaluations. They require agreement from all parties about what success of your engagement process means to you. Often, these types of metrics are difficult to define. When you think of a successful outcome for your initiative, such as the influence of input on your decision making process, it may be more difficult to measure than imagined (as many decision makers can emphasize, the decision making process is not as straight forward as imagined!). Further complicating this type of evaluation is that these outcomes may take months, or even years, to realize.
Interviewing decision makers or key stakeholders after the completion of your citizen engagement initiative can be an important component of any outcome evaluation. Other measurable outcome measures can be the incorporation of public values into decision making, improved public trust, reducing conflict and/or public education and learning.
Involving citizens in outcome evaluation can also be beneficial. Citizens can lend legitimacy to the results of the engagement initiative and can make the findings the more robust. Furthermore, closing the loop with participants is an important consideration so that participants feel involved and can feel confident that the initiative was a worthwhile use of their time.
The key considerations for an outcome evaluation are outlined below:
- What were the effects of the citizen engagement initiative? Did the initiative have its intended impact on decision making or priority setting? Did senior decision makers heed the advice of citizens? This is a difficult criterion to measure and one that may take a longer time to evaluate. Nonetheless, this should be a consideration for all organizers and should be reported back to citizens in a timely manner.
- Will citizens or stakeholders participate in a similar citizen engagement initiative again in the future? Judging public opinion about your initiative can be an arduous task. Following up with participants and simply asking if they would consider participating in a similar initiative again in the future is a good indicator of public satisfaction and trust.
Evaluations of citizen engagement initiatives are by the very nature iterative and are rarely straight forward. While the field of evaluation is growing, so is the confusion for organizers.
For organizers, while the point is not to craft the most up to date, robust evaluation plan; it is important to incorporate several core tenets of evaluation discussed here and include the results of your evaluation in your planning process for future initiatives!