Want to start evaluating your engaged reporting? Here’s what I did

I am on a mission: I want evaluation to be a newsroom norm, not something that would be nice to have. Weaving evaluation into my projects has been a game changer, giving me the data I need to understand the impact of my work and communicate it’s value to funders.

That’s why I want to share a case study of how I activated Capital Public Radio leaders to support a robust assessment of a recent cross-platform documentary project. This is not a treatise on the value of evaluation, or a how-to guide. It’s an explanation of how my planning process brought in money, secured colleague and community support and delivered a plan we could execute and replicate.

I hope my experience helps jumpstart evaluation in your newsroom as we collect and share the data we need to grow the engaged journalism field.

Rope your bosses into brainstorming

Since I joined Capital Public Radio as the senior community engagement strategist five years ago, I have experimented with different ways to gauge outcomes from our documentary series The View From Here. For example, here’s our theory of change, an early assessment matrix, and some guiding values. Our most recent production, Place and Privilege, was my biggest assessment challenge yet.

Not only did the project aim to tackle one of the thorniest topics in California — housing affordability and residential segregation — the team also wanted to produce the station’s first podcast series, test out a new marketing strategy and pilot different kinds of in-person engagement experiences. As we got underway, I wondered: How the heck are we going to assess all of this?

To begin answering that question, I whipped out my Impact Pack, a set of playing cards developed by Dot Connector Studio to help teams prototype their media engagement plans. I roped TVFH Senior Producer Catherine Stifter and Chief Content Officer Joe Barr into our conference room for 30 minutes of out-of-the-box planning.

Rearranging the cards and playfully challenging each other’s ideas, we laid out the project goals, audiences, platforms and assessment strategies. By making it a game, I pulled us out of our rote ways of thinking. After a good amount of laughter and some poker faces, we arrived at a shared vision. Take a look at our layout to see how we placed TVFH’s core value at the top and the community impact we wanted at the bottom. In between we located what we wanted to do, why, how and for which audiences.

By involving key internal decision makers in the initial prototyping process, I built alignment from the get-go and ensured that that no one felt blindsided as I moved the evaluation process forward.

Take a chance — ask for outside help

Given the grand scope of our project vision, I knew we would need serious funding to evaluate it. While CFO Jun Reina and other newsroom leaders were mulling over my pitch for financial support, I had a flash of inspiration: Would a big evaluation firm do some pro bono work to get us started? I knew that Harder+Company, a national research group with a branch in Sacramento, had recently conducted an evaluation of community engagement in the arts across our region. Could I entice those evaluators to help by convincing them that it would be an easy lift to collaborate on some initial planning? And by highlighting the value of associating with a high profile public radio project focused on California’s capitol?

I could! All Harder+Company agreed to facilitate two pro-bono evaluation planning sessions that included CapRadio leaders, TVFH producers, and Place and Privilege community partners including the Sacramento Housing Alliance, UC Davis Center for Regional Change, Sacramento Public Library and AARP. Harder+Company valued their work in designing those sessions, leading them and reporting back learnings at $10,000. This number not only raised eyebrows (in a good way!), it lent gravitas and validation to the process.

Get your colleagues and community excited

At the first planning session, Harder+Company helped the group take stock of TVFH’s community engaged journalism model and reflect on the value it brings to the station, our audience, community partners and wider public. Then we discussed the benefits of evaluation: What did the group hope to discover and share through evaluating Place and Privilege? What resources were available to do this?

By the end of that first meeting, we had mapped out evaluation goals, desired impacts, existing data sources, and internal capacity. The bigger impact, however, was this: by inviting station leaders, project producers and community partners to be co-creators from the start, I demonstrated that evaluation was a process and a product we could all benefit from.

During the second session, Harder+Company presented a draft evaluation plan for Place and Privilege, which the group reviewed and modified. It centered on three big-picture goals:

• Document the value that the community engaged documentary model brings the public, project partners, advisors, and Capital Public Radio.
• Build evaluation capacity to assess the implementation of the community engaged documentary model by Capital Public Radio and project partners.
• Support internal learning and information sharing with funders and community stakeholders (e.g., the public, journalist community, public broadcasting).

The plan articulated a set of research questions to guide us.

Outcomes

• How does the community engaged documentary model inform the content and work of CapRadio?
• How does the community engaged documentary model shift perceptions of and connection to CapRadio (e.g., among partners, new/non-traditional audiences, etc)?
• To what extent do content and events increase public awareness, empathy, and involvement on the topic?
• What are the impacts of Place and Privilege for communities, including new and/or strengthened relationships?
• What works about CapRadio’s Place and Privilege project, and what doesn’t, and what does that mean about what we should do next?
• To what extent were CapRadio’s guiding principles for community engagement followed, what difference did that seem to make and should we adjust anything?

Informing the Work

• How does the community engaged documentary model inform the content and work of CapRadio?

Outputs/Reach

• How many people attended the events? What stakeholder groups are represented? How engaged were there.

This draft plan laid out possible measurement tools and indicators for each question in a color-coded matrix. I was in evaluation nerd heaven!

Even better, I saw CapRadio staff and community stakeholders get excited about evaluation. I had delighted leaders who had the power to invest in evaluation at CapRadio going forward. I also got community buy-in. Having those partners at the table from the beginning meant they were more willing to participate in evaluation activities down the line by filling out surveys and doing phone interviews.

A note about consultants: I thought working with Harder+Company would take a huge chunk of work off my plate. Wrong! We still needed a knowledgeable staff member to closely manage the process to get accurate and relevant results. I ended up spending upwards of 40 hours crafting the planning sessions and materials (and up to 25 percent of my time in later parts of the process). Outside experts bring fresh eyes and new skills, but on-the-ground staff can best frame and implement plans. You need both.

Work in a team, if you can

Harder+Company’s planning process generated a shared understanding not only about the value of doing evaluation, but the level of resources needed to make it happen. It became clear that this wasn’t something I could jam into my current job description and work plan. I was given a $20,000 budget and given the go-ahead to bring in an evaluator.

Enter Lindsay Green-Barber of Impact Architects. Lindsay and I had collaborated on evaluating a previous TVFH project — she already understood CapRadio’s approach to community engaged reporting. Together we revised Harder+Company’s initial framework to align more closely with the station’s needs, resources and timeline. Here’s the evaluation plan we came up with. It details key indicators that speak to each of our research questions, as well as how we’d collect the data. The Impact Architects’ scope of work included producing interim memos to inform ongoing planning and working with me to design assessment tools. That built in feedback loops for strategy and learning as well as refining my ability to lead evaluation work on future TVFH projects.

But, evaluation nerd that I am, I wanted more and different input to the process. I carved out funding to include a journalist and social scientist and in this evaluation effort. Jessica Clark of Dot Connector Studio and Yve Susskind of Praxis Associates gave critical feedback on the evaluation plan by reviewing tools and commenting on reports.

Yve taught me about development evaluation and challenged me to articulate the principles that guide TVFH (more on that in a future blog post) while Jessica shared key impact assessment research and helped position my work within larger national trends. Meanwhile, Lindsay kept me focused on what was working and what wasn’t so that I could improve and replicate our engaged journalism model.

Lindsay, Jessica and Yve are all linked to different national journalism initiatives. They brought best practices and emerging needs from the field into our planning and implementation. Even better, they plugged our dialogues and data into national reports and convenings which, in turn, raised Capital Public Radio’s profile in the engaged journalism space. Working with them made me a better engager and evaluator.

Check out our Impact Report for details on how we implemented the evaluation, what we learned and why it matters.

Fruits of the labor

You might be thinking, “Evaluation sounds like a lot of time and money I don’t have.” The Place and Privilege project evaluation was resource-intensive, but I believe its lessons are valuable regardless of your budget:

1. Invite your supervisor to brainstorm with you about the benefits of evaluation. This could be just one or two meetings.

2. Ask for outside help. Even if this just means pulling in examples of evaluation wins from others, gather support that lends weight to your endeavor.

3. Get your colleagues and community excited and involved. They will be more willing to participate and share data down the line.

4. Work in a team if you can. This is tied to getting outside help. When imagining how to evaluate the efforts of your own organization, perspectives from other disciplines are critical.

Thanks to these early steps, our evaluation process was a real success. We got feedback that improved the way CapRadio does community engagement. We generated data that we could share back with project partners in their effort to address the housing affordability crisis. We boosted internal interest in, and evidence of, the value of engaged journalism. As a result, our next engagement effort got funded from the get-go.

The engaged journalism field is emerging. We are all looking for ways to articulate the value of what we do, illustrate our methods, deliver concrete results and have a way to prove it. Evaluation is key to making all that happen.

Huge thanks to Olivia Henry for her wise and substantive edits on this piece. You can read more of her amazing work here.

jesikah maria ross produces participatory media projects that generate public dialogue and community change. She is the Senior Community Engagement Strategist at Capital Public Radio, Sacramento’s NPR affiliate and part of the inaugural #50WomenCan Journalism cohort. @jmr_MediaSpark, jesikahmariaross.com