Throwback: It’s hard to believe I wrote this post four years ago!
I’m so grateful for the incredible professional development support I received through YouthREX: Since I shared that post back in 2017, YPI’s Evaluation Plan has served us well, and we have relied on it to collect information about the outcomes and impact of our program in order to communicate these with our key stakeholders: students, teachers, and administrators at our participating schools; representatives from charitable organizations that participate in YPI; funders of our work; and mission peer organizations and partners.
COVID and Evaluation
The pandemic has now affected two academic years for students around the world. Like many other organizations, we needed to completely overturn our program model in response to COVID-19. Our adaptations have included offering virtual student workshops this year instead of targeted in-person visits; and offering flexible alternatives to our standard programming for teachers to facilitate YPI. Since the process has been so different, our process evaluation has evolved as well: We have developed different administrative forms and data collection processes and have tried our best to streamline these for teachers who are under an immense amount of pressure supporting their students and school communities.
While our activities have had to change during this time, we have targeted the same outcomes with our programming; so, we have continued to evaluate the same outcomes as we did pre-pandemic. Our intention is to generate useful comparisons about program adaptations we have made in response to the pandemic: What are the emotional outcomes of participating in YPI for students who were able to experience in-person engagement compared with the those for students who completed the program online? Did students develop different skills as a result of our adapted resources? What can we learn from these results to inform our programming decisions post-COVID?
Looking Ahead
Programs and organizations should evolve over time, and evaluation needs to keep up.
At YPI Canada, our amazing team (which has grown considerably since this original post!) is currently undertaking deep reflection about equity and inclusion in our program model and resources, and about how we might deepen discussions with our stakeholders about social justice and human rights in our program. We are just beginning this process, with a long way to go.
Here are some of the many implications and questions this process has introduced for our evaluation program, and to my journey as an evaluator:
- I must regularly, critically acknowledge and interrogate my own biases and identity as an evaluator: As a white settler, an able-bodied, cis-gendered, straight person of class privilege, I hold enormous privilege and power; and I benefit from white supremacy culture every day. How does this truth influence this work? What biases do I carry to creating evaluations and analysing data, and how can I mitigate these?
- As I interrogate my own position in systems of power and privilege personally and professionally, YPI considers its place in the landscape of education – a system designed to oppress some students – and YPI’s place in it. YPI has participated in, contributed to, reflected, benefited from, this system of oppression. How do we move forward?
- In an education system where people who look like me continue to define student success, what are the desired outcomes of our program, and who defines them? Our existing logic model draws from peer-reviewed research in positive youth development; but we never validated any assumptions we made with our key stakeholders.
- Defining new outcomes that center equity, in collaboration with our stakeholders, will mean reimagining our logic model/theory of change: With different outcomes targeted, there may be different activities required to get there.
- We need to re-think what kinds of evidence that we value: Quantitative data makes for good bite-sized comms pieces about outcomes; but more deep listening to stories in our program could help us better understand our stakeholders and their experiences in YPI. And to problematize some long-held evaluation dogma: These stories may not be representative samples of all the students we engage with, but that makes them no less valid.
Some of these topics and many more are reflected in the work of Equitable Evaluation Initiative. I’m just starting to dive into their work, and look forward to learning much more.
Four years and one pandemic later, a lot has changed. I continue to be inspired, motivated, and challenged by this work and look forward to sharing progress with our community as we progress. Thank you for reading and learning with me!
I’m extremely grateful for the leadership of my colleague Noor El-Husseini, our Education Program Manager and Equity Lead, for leading us in critical conversations about anti-oppression, anti-racism, power, and privilege in our work. Special thanks also to my colleague Lauren Pragg, who had the great idea to share this throwback post and challenged me write this reflection.
