The evaluation utilized both quantitative and qualitative data collection methods to holistically assess the CHAS PB process. Data sources included survey questionnaires, participant observations, and participant interviews. The evaluation team also incorporated data from City of Denver reports, news articles, and other sources to contextualize the process and its results.

The TMHAS Evaluation Committee was comprised of seven individuals, including a graduate student from the University of Colorado Denver, staff members from the other Denver PB process, a graduate student from Portland State University, a graduate student from the University of Colorado Boulder, and an undergraduate student from the University of Colorado Boulder. The committee included researchers, activists, and practitioners, and they worked collaboratively to create data collection instruments (e.g., survey questionnaires, interview protocols, and observation protocols) in accordance with national standards of PB research established by Public Agenda (PBPadmin, 2015).

Background Research

The research team collected data about the city budget, the history and demographics of Cole neighborhood, and PB to understand how the CHAS process might impact participant knowledge, skills, attitudes, and practices related to public engagement. Data sources included the American Community Survey, the U.S. Census, the City of Denver's Cole neighborhood profile fact sheet, and the City and County of Denver Mayor's 2018 Budget.
 

Quantitative Data Collection

PB participants completed surveys across three phases of the CHAS process: idea collection, project development, and voting. Survey respondents included steering committee members, lead change makers, project idea submitters, and PB voters. All surveys were administered both online and on paper to generate as high of a completion rate as possible.

The survey questionnaires were designed to collect demographic information for PB participants across all phases. Steering committee and budget questionnaires were designed to assess levels of prior civic and community engagement. Idea submitter and voter questionnaires included questions about attitudes related to PB and public officials, prior civic engagement, and how respondents learned about the PB event where they participated.
 

Qualitative Data Collection

Qualitative data was collected from in-depth interviews and participant observations. To ensure multiple viewpoints were collected, a variety of stakeholders were interviewed for the purpose of the evaluation, including steering committee members, idea collection participants, lead change makers, and PB organizing staff. All interviews were conducted individually face-to-face, over the phone, or through email. Each interview lasted approximately 30–60 minutes.

Interview questions were tailored for each stakeholder group and were developed in collaboration with members of the TMHAS Evaluation Committee. In general, the interviews were designed to understand how people participated in PB, how PB affected relationships between residents and government officials, and how PB contributed to attitudes and practices of community engagement outside of the process.

Fieldnotes were recorded across all the stages of the CHAS PB process. Observation protocol for fieldnotes was designed to document and assess the quality of public deliberation at events. Researchers were attentive to points of negotiation, conflict, forms of inclusion, and how authority operated in meetings and public outreach events.

These participant observations allowed for the generation of direct, experiential knowledge that documented how CHAS unfolded through real-time interactions with participants. Fieldnotes included quotes from participants as well as thoughts, ideas, and experiences of researchers as they occurred at PB events. Field observations provided a more nuanced, detailed view of the PB process that could not be captured solely through surveys.

References

PBPadmin. (2015, September 13). 15 key metrics for evaluating PB. Participatory Budgeting Project. https://www.participatorybudgeting.org/15-key-metrics-for-evaluating-pb/