Creating an Evaluation Tool for the Circles Weber County Anti-Poverty Initiative by CodiAnne Stevens Nye A proposal submitted in partial fulfillment of the requirements for the degree of MASTER OF EDUCATION with an emphasis in FAMILY LIFE EDUCATION WEBER STATE UNIVERSITY Ogden, Utah April 24, 2024 Approved Cass Morgan, Ph.D. Daniel Hubler, Ph.D. Charles Dunn (Dec 6, 2024 14:37 MST) C. Ryan Dunn, Ph.D. CIRCLES IMPLEMENTATION ASSESSMENT 2 Nature of the Problem Literature Review Introduction Definition and Characteristics of Poverty Definition and Importance of Social Capital Program Implementation Assessment Fidelity Quality Adaptations Conclusion Overview of the Circles Initiative RE-AIM Framework Reach Effectiveness Adoption 4 5 5 5 7 8 9 9 10 10 11 13 13 13 14 Years 14 Implementation Maintenance Method Participants Evaluation Measures Essential Components of Weekly Meetings/Interventions Session 1: What is Circles Session 2: Your Story Session 3: Poverty and the Trauma of Poverty Session 4: Expectations and Bias Session 5: Dream Boards Session 6: Personal Finance Part 1 Session 7: Personal Finance Part 2 Session 8: SMART Goals Session 9: Building Your Support Network Session 10: Communications and Strong Relationships Session 11: Work Relationships and the Rules of the Workplace Session 12: Congratulations and Next Steps Analysis Results Fidelity Adaptations 15 15 16 16 16 21 22 22 22 22 22 23 23 23 23 23 23 24 24 24 25 25 CIRCLES IMPLEMENTATION ASSESSMENT 3 Discussion 27 Limitations Implications Recommendations Future Directions References 27 27 28 28 29 CIRCLES IMPLEMENTATION ASSESSMENT 4 Nature of the Problem Approximately 37.9 million people, including nine million children, are currently living in poverty in the United States (U.S. Census Bureau, 2022). Individuals experiencing poverty lack access to many basic needs, including adequate shelter, food, education, and healthcare (Wagmiller & Adelman, 2009). Furthermore, Duncan and Holzer (2023) estimate that nearly one-third of children who grow up in poverty are more likely to experience it again as adults, creating a cycle of intergenerational poverty. A pervasive lack of resources, limited access to job and educational opportunities, and a lack of social capital make this cycle difficult to break (Jensen, 2009). Many anti-poverty programs currently aim to address or reduce poverty in impacted communities. One such program is CirclesUSA, which has demonstrated success in its mission to end poverty by building community. With 25 years of supporting research, the program has effectively used skill-building workshops and increased social connections to help build financial, emotional, and social resources (CirclesUSA, 2023). These resources are practical tools for breaking the cycle of intergenerational poverty. With 78 Circles chapters operating across the United States and Canada, it is clear that the program is making a difference. In fact, Circles effectiveness is evident in its most recent impact report, which shows 76% of participants experience increased income after 18 months (CirclesUSA, 2023). However, despite providing local chapters with a list of fidelity agreements, CirclesUSA currently has no standardized evaluation tool to assess program delivery, which is a crucial next step. While monitoring program outcomes is a routine practice, assessment of program implementation is often overlooked (Durlak & DePre, 2008). This project aims to fill the gap by exploring the importance of program fidelity and the impact on achieving desired outcomes of CIRCLES IMPLEMENTATION ASSESSMENT 5 the Circles USA curriculum: building skills and social capital that will lead to economic stability. The goal of this project is to create an evaluation tool that measures program implementation at the Circles Weber County chapter in Ogden, UT. Literature Review Introduction Assessing implementation is essential to ensuring the effectiveness of any program. This project will serve as a lens for examining the extent to which the Circles program is delivered as intended at the Weber County chapter. By identifying specific delivery methods and curriculum components, this assessment will help program staff understand the connection between implementation and outcomes, thereby enhancing the program’s overall effectiveness. Definition and Characteristics of Poverty Poverty is a state of deprivation in which people cannot access the resources necessary to meet their basic needs (Payne et al., 200)). While poverty exists in various forms, there is a distinction between general economic hardship and the more nuanced manifestation of generational poverty (Payne et al., 2009). Distinguishing between the types of poverty is crucial to formulating effective strategies for poverty alleviation. For instance, anti-poverty programs include Temporary Assistance for Needy Families (TANF), Supplemental Nutrition Assistance Program (SNAP), Women Infants and Children (WIC), health insurance coverage such as Medicaid, Medicare, and Children’s Health Insurance Program (CHIP), and childcare assistance; all commonly referred to as welfare. These programs reduce poverty by supplementing a household’s resources, which does reduce poverty at a point in time. However, without increased skills or knowledge, many households may return to poverty when benefits cease, making long- CIRCLES IMPLEMENTATION ASSESSMENT 6 term effectiveness more challenging to measure for families experiencing intergenerational poverty (Pac et al., 2017). Situational poverty is a lack of resources triggered by a specific life event (Payne et al., 2009). The event could be related to death or divorce, loss of employment, natural disasters, or a recession. While this type of poverty often leads to a sudden loss of financial support, the consequences are typically temporary. Those experiencing situational poverty will often maintain ties with their social supports and preserve knowledge of middle-class systems. These social relationships and understanding of middle-class life will often help families return to stability relatively quickly (Beegle, 2003). On the other hand, intergenerational poverty refers to a complex issue where economic disadvantage is perpetuated across multiple generations. Families experiencing intergenerational poverty often develop unique belief systems around the culture of poverty and acquire skills for navigating it (Payne et al., 2009). Jenkins and Siedler (2007) describe this intergenerational persistence as the inheritance of poverty. This transmission of poverty-related behaviors contributes to the continuation of poverty across generations. Furthermore, social and economic mobility is limited for those who spend extended periods of childhood in poverty (Wagmiller & Adelman, 2009). Studies show that the longer someone lives in poverty during childhood, the more likely they are to experience poverty again as an adult (Privette-Black, 2021; Wagmiller & Adelman, 2009). Chronic poverty is related to adverse outcomes across multiple life domains (Lee et al., 2021). Physical and mental health, for example, are negatively impacted by the cycle of family poverty because of limited access to healthcare and increased vulnerability to chronic disease (Lee et al., 2021; Privette-Black, 2021). Also, children living in poverty are more likely to attend CIRCLES IMPLEMENTATION ASSESSMENT 7 under-resourced schools, experience chronic absenteeism, or drop out of high school, potentially impacting long-term success (Wagmiller & Adelman, 2009). Social capital can serve as a catalyst for alleviating poverty by fostering community collaboration, resource mobilization, and enhanced socio-economic opportunities. Definition and Importance of Social Capital Increased social capital is a core component of the Circles model. For this study, social capital refers to the collective value of an individual's social networks and relationships. Sawhill (2020) describes social capital as reciprocal trust between formal and informal group members that positively impacts individuals and communities. In addition to personal support, social capital provides resources for fostering economic mobility. In fact, Chetty et al. (2021) state that relationships between individuals from different socioeconomic backgrounds are the strongest predictor of upward economic mobility. These relationships provide opportunities for career advancements, mentorship opportunities, and the transmission of industry knowledge. Acquaah (2008) explains this concept by stating that who you know can be as important as what you know. Additionally, Aldrich and Meyer (2014) present that in times of crisis, social capital is at least as necessary as money, infrastructure, or material resources, making the case that increased social capital is valuable for communities and individuals. Building social capital requires intentional efforts to develop meaningful, formal, and informal relationships by building partnerships between individuals who share common interests (Iantosca et al., 2024). Civic engagement also provides opportunities to connect people from diverse backgrounds, expand networks, deepen knowledge, and cultivate connections between community members (Sawhill, 2020). CIRCLES IMPLEMENTATION ASSESSMENT 8 Kyne and Aldrich (2019) differentiate three kinds of social capital: bonding, bridging, and linking. These three types of connections enhance cohesion and social ties at an individual and community level. Bonding social capital refers to connections between individuals with similar backgrounds, cultures, languages, or occupations. These people will notice someone’s absence from school or relate to their struggles in the workplace. Bridging social capital is created from a weaker tie (Kyne & Aldrich, 2019). These connections extend across social boundaries to individuals with different life experiences or beliefs. Intentionally creating these types of social connections can be instrumental when looking for employment or educational opportunities (Kramarz & Skans, 2014). The third type of social capital is linking, which connects people with resources or organizations that increase an individual’s ability to improve their economic well-being (Sawhill, 2020). While bridging social capital is the most prevalent, bonding and linking social capital can be pivotal for supporting vulnerable populations, making the case that all three categories of social capital are essential to improving a family's financial circumstances (Kyne & Aldrich, 2019). To evaluate the impact of social capital’s role in economic well-being, it is necessary to integrate measures that assess how programs leverage these connections. Program Implementation Assessment Program success is often quantified through performance measures, and future funding often depends on meeting prescribed metrics (Carnochan et al., 2014). While outcome measures are important indicators of effectiveness, they do not always capture the various factors that indicate successful program outcomes. This is particularly true for human service organizations where participant progress is complex and difficult to quantify (Bach-Mortensen & Montgomery, 2018; Carnochan et al., 2014). Previous studies have identified multiple dimensions impacting CIRCLES IMPLEMENTATION ASSESSMENT 9 program efficacy, including fidelity, dosage, quality, adaptations, participant responsiveness, and reach (Berkel et al., 2011; Durlak & Dupree, 2008). This project will use the adapted RE-AIM (reach, effectiveness, adoption, implementation, maintenance) evaluation framework to focus on three elements of program implementation: fidelity, adaptations, and quality of delivery. Fidelity For the purposes of this study, program fidelity refers to how an intervention adheres to its original design. Fidelity plays a critical role in determining the success and effectiveness of a program (Carroll et al., 2007). Essentially, program fidelity ensures observance of the fundamental components and underlying principles laid out by the program designers. Fidelity ensures the attainment of consistent and reliable results as well as the achievement of desired outcomes (Breitenstein et al., 2010). Focusing on fidelity is instrumental to fostering successful programming and reinforcing the connection between design and outcomes (Carroll et al., 2007). Program fidelity encompasses core curriculum elements, dosage, and delivery methods, although it allows for some flexibility based on contextual factors (Allen et al., 2012). Many studies demonstrate a correlation between fidelity and effectiveness (Durlak & DuPre, 2008; Meyers et al., 2012). In fact, studies find that adherence to program design increases the likelihood of achieving expected outcomes, while deviations compromise program integrity, which may lead to inconsistent results (Fixsen et al., 2013). The link between implementation and effectiveness highlights fidelity's pivotal role in a program’s overall success (Breitenstein et al., 2010). Quality Quality of delivery is another important indicator of program success and includes facilitator competence as well as the processes used to deliver information (Durlak, 2016). CIRCLES IMPLEMENTATION ASSESSMENT 10 Delivery quality refers to behaviors, teaching styles, and strategies a facilitator uses (Dowling & Barry, 2020). Farahnak et al. (2020) describe a program leader’s attitude as a critical predictor of implementation success. Measuring facilitator competence involves assessing skills and knowledge that contribute to effective facilitation. While quality can be difficult to weigh, various methods measure delivery quality, including observation, checklists, peer reviews, participant feedback, and self-reflection (Goe & Croft, 2009; Kawasaki et al., 2020). Adaptations Incorporating adaptations into the curriculum can be beneficial in some circumstances. In fact, it is important to note that curriculum adaptations and fidelity adherence are not mutually exclusive ideas (Gelmez-Burakgazi, 2020). For this study, adaptations include modifications or adjustments to a system or process to better align with specific needs or requirements (Carvalho et al., 2013). Program adaptations can be useful and even necessary when they are used to meet the needs of current program participants or funding sources. Additionally, adaptations may be necessary in the face of resource limitations, such as budget, time, or technology constraints. Cultural sensitivity may also prompt variations to ensure respect for participant norms and values (Nolt et al., 2023). In some situations, adaptations may be advantageous. However, careful consideration is necessary to maintain program integrity. Conclusion Fidelity is essential to program success, and adhering to design has been shown to enhance participant engagement and program effectiveness. Additionally, a commitment to fidelity lays the groundwork for sustainability, ensuring the program can be effectively reproduced in multiple contexts. This specific fidelity evaluation will assess the quality of CIRCLES IMPLEMENTATION ASSESSMENT 11 implementation of the Circles curriculum at the Circles Weber County chapter using an adaptation of the RE-AIM framework. Overview of the Circles Initiative Circles is a long-term initiative with a multi-generation approach to breaking the cycle of poverty (Circles USA, 2023). Participants start the program by attending a 12-week cohort of workshops to learn skills ranging from budgeting, managing debt, writing a resume, and gaining employment. These skills assist those living in poverty to become more equipped to achieve financial stability (CirclesUSA, 2023). Children of participants engage in similar skill-building activities at an age-appropriate level in the children’s program (Circles Youth Curriculum, 2021). At the end of the 12-week workshop, participants are certified as Circle Leaders and matched with a Circle Ally. Allies are moderate to high-income volunteers from the community who work with Circle Leaders to set goals, make connections, and offer insights into the unwritten rules of the middle class. This Circles match is described as an intentional friendship and aids Circle Leaders in establishing a social network and building social capital (CirclesUSA, 2023). In addition to their match, Circle Leaders and Allies may choose to participate in Circles Big View to tackle community-level issues impacting neighborhoods. Thus, the Circles model incorporates all three types of social capital: bonding, bridging, and linking. As shown in the logic model in Figure 1, the Circles program in Weber County is grantfunded through Utah’s Department of Workforce Services (DWS) and employs four full-time and four part-time staff. The program also relies heavily on volunteers to help run the weekly meetings. Each Circles weekly meeting begins with a shared meal to help foster connections between families and the community. These shared meals are instrumental to the Circles model as research strongly supports the benefits of shared family meals for overall well-being and CIRCLES IMPLEMENTATION ASSESSMENT 12 fostering community connections (Bjornwall et al., 2021; Robson et al., 2020). The Weber County Circles chapter also utilizes a resource case manager to connect participants to additional community services and resources. The 12-week training equips participants with the skills, knowledge, and confidence to combat poverty by fostering leadership qualities and increasing community engagement. Participants are required to attend at least 10 of the 12 workshops to certify. Throughout the program, participant progress is monitored using the LifeWorks Self-Sufficiency Matrix (SSM), which measures self-sufficiency across 18 domains. Domains are scored on a scale of 1-5, with one meaning in crisis, two vulnerable, three safe, four building capacity, and five empowered. This study will focus on the specific domains of income, employment, education, and network support. Families typically attend Circles weekly meetings for 12-18 months and should attend at least twice a month on Match Nights. The long-term goal is to graduate participants from the program with an income of at least 200% of the Federal Poverty Level (FPL). Figure 1 Logic Model CIRCLES IMPLEMENTATION ASSESSMENT 13 RE-AIM Framework This project will use the RE-AIM) framework to contextualize the purpose of the Circles program. The framework includes essential program elements: reach, effectiveness, adoption, implementation, and maintenance. An adaptation of the RE-AIM program assessment tool will be used to assess the program implementation of Circles in Weber County. RE-AIM is an evidence-based framework used across sectors to develop, assess, and maintain program efficacy (Glasgow et al., 2019) Reach The element of reach refers to the target population for a given intervention. It measures who is intended to benefit and who participates (Glasgow et al., 2019). Circles primarily recruits participants under 200% of the FPL who are ready to move from surviving to thriving (CirclesUSA, 2023). Circles Weber County has established a goal of starting at least four cohorts each year with 10 participants. An equal number of volunteers must be recruited to match each participant upon Circle Leader certification. In this program year, Circles Weber County has started four cohorts, with another planned for June 2024. Thirty new participants have enrolled in the Circles program, and there are currently 23 volunteers ready to match with Circle Leaders upon completion. Effectiveness Effectiveness measures an intervention's impact on participants; positive and negative outcomes can be measured. The Circles program aims to achieve the primary outcome of families moving to economic stability, typically called graduation, and measured as reaching 200% of the FPL. Since 2021, 123 individuals have started a Circles cohort, with 58 certifying as Circle Leaders. Eighteen out of 58 Circles Weber County participants who completed the 12- CIRCLES IMPLEMENTATION ASSESSMENT 14 week workshop have reached economic stability, resulting in a 47% completion rate and a 31% graduation rate, as indicated in Table 1. Additionally, to measure progress, program participants complete the Lifeworks Self Sufficiency Matrix (SSM) when they start and finish a cohort and then at six-month intervals throughout their program. This study will focus on four areas measured by the SSM: income, employment, education, and network support. As shown in Table 2, many participants have seen improvements in each of these areas since 2021. One potential negative consequence of achieving economic stability is an issue known as the Benefits Cliff or the cliff effect. This occurs when a family who previously qualified for public benefits receives a raise that makes them ineligible to continue receiving assistance. However, more than an increase in income is needed to cover the cost of benefits lost and sustain their household. Casau and Hyde (2018) refer to this as taking one step forward and then two steps back. Adoption Program adoption refers to the location where an intervention is delivered and by whom. Adoption involves the investment of the staff, setting, and systems in implementing a program. It investigates the characteristics of those who agree to implement the intervention and their Table 1 Four-Year Completion and Graduation Rates Years 2021-2024 Completion Rate Graduation Rate 58/123 = 47% 18/58 = 31% CIRCLES IMPLEMENTATION ASSESSMENT 15 Table 2 Four-Year Self-Sufficiency Domain Improvements Years 2021-2024 % of participants % of participants % of participants % of participants who increased who improved who improved who improved income employment education network support 49% 38% 28% 66% reasons (Glasgow, 2019). Circles Weber County currently has four full-time staff: a Coordinator, Coach, Case Manager, and Outreach Worker. The Circles Coach facilitates the weekly meetings. The Circles program is supported agency-wide at OWCAP by the administration, the board, front-line staff, and volunteers. Additionally, the program is supported by the community in the form of advertising, donations, volunteerism, and referrals. The program is implemented yearround at OWCAP’s central location in Ogden on Wednesday nights. Implementation The main focus of this project is the implementation aspect. Namely, how consistently interventions are delivered, how they are adapted, and potential reasons for outcome results. The evaluation tool was developed from the RE-AIM framework to include an observer and facilitator piece to examine and compare perspectives on implementation. Information collected may provide insight into how fidelity implementation impacts OWCAP’s Circles outcome measures. Maintenance Program maintenance is the aspect of RE-AIM that measures the length of time results are sustained for participants (Glasgow, 2019). Many Circle Leaders who graduate at 200% FPL continue with Circles Weber County as Allies or in some other volunteer capacity. However, CIRCLES IMPLEMENTATION ASSESSMENT 16 there is currently no system in place to follow up with participants after they graduate from the program. This may be a direction for future study. Method Participants This implementation assessment project focused on the coach’s delivery of the Circle Leader training beginning in the June 2024 cohort. The program manager observed the class using the Observer Tool during each weekly session. The Coach completed the Facilitator Tool at the end of each week’s class. The program manager and coach debriefed weekly on Thursday mornings. For the June 2024 cohort, an anticipated 10 participants will be enrolled in the program. All participants are Weber County residents with an annual income below 200% FPL. While program participants will be present during the evaluation period, they will not be observed as part of this process. Their engagement and responsiveness may be addressed and measured later. Attendance and completion rates will also be tracked as part of this project. Classes are held at OWCAP’s central location, 3159 Grant Ave. in Ogden, on Wednesday evenings from 5:30-7:30 p.m. Evaluation Measures The evaluation tool for assessing implementation was adapted from the RE-AIM framework checklist. The original tool was designed to assess programs at all levels, from planning to monitoring, and includes dimensions for reach, effectiveness, adoption, implementation, and maintenance. This project will focus on the framework's implementation dimension and collect qualitative and quantitative data from the facilitator and an observer. As noted in Figure 2, the observer used the form to evaluate the coach during each Circles session. CIRCLES IMPLEMENTATION ASSESSMENT As noted in Figure 3, the coach will use the form to self-reflect on delivery after each Circles session. The observer and facilitator will debrief weekly and discuss congruence between the forms. 17 CIRCLES IMPLEMENTATION ASSESSMENT 18 Figure 2 Observer Tool Observer: Date & Topic: Percent of Perfect Intervention Delivery Adaptations Which methods of intervention were delivered this week Delivery Methods: ● Shared meal ● New and Good ● Check-in ● In This Session ● Listening Pairs ● It’s Your Story ● Appreciations Curriculum Components: (items change weekly) ● ● ● ● Number of Interventions: Percent of Delivery: Was any aspect of the content or delivery of the content changed or modified in any way during this week’s module? If so, explain. CIRCLES IMPLEMENTATION ASSESSMENT Cost of Intervention How was dinner provided? (time & money) Total number of staff present: Total number of attendees present: Notes on Delivery (Qualitative methods to understand implementation) Behavioral Markers: ● ● ● ● Participants encouraged to ask questions Participants encouraged to engage in dialogue Coach able to connect content to participant’s lives Coach validates participant’s perspectives Delivery Method (include quantity): ● Small group discussions: _______ ● Activities: _______ ● Use of open-ended questions: _______ ● Lecture: _______ ● Other (specify): _______ 19 CIRCLES IMPLEMENTATION ASSESSMENT Figure 3 Facilitator Tool Facilitator: Date & Topic: Implementation Fidelity Which methods of intervention were used this week? Curriculum Components: ● Shared meal ● New and Good ● Check-in ● In This Session ● Listening Pairs ● It’s Your Story ● Appreciations Adaptations Was anything changed or modified in any way during this week’s module? If so, explain. Self-Reflection on Delivery What worked well? Would you change anything if you were to deliver this module again? 20 CIRCLES IMPLEMENTATION ASSESSMENT 21 Essential Components of Weekly Meetings/Interventions Each weekly session includes components to be implemented by the facilitator/coach. The key components/interventions of each week’s session are: ● Shared Meal: A meal that is donated or prepared by Circles staff. ● New and Good: Each person shares one thing from their week that is new and good. This practice helps participants celebrate the good in one another’s lives and focuses on the positive. ● Check-in: This consists of reviewing the tasks given during the previous lesson. It reinforces the lesson from the previous week and builds a habit of accountability. ● In this Session Overview: Outlines the main ideas of the session so participants know what to expect. ● Listening Pairs: Opportunities to discuss content with a neighbor. This helps participants practice listening and sharing, engage with the material, and reinforce the key concepts. ● It’s Your Story: As the session ends, participants are given a task to complete before the next week’s session. This helps participants connect outside the session and go deeper with their work at weekly meetings. ● Appreciations: Each night ends with an appreciation exercise. Each person shares one thing they appreciate about the person on their left. The receiver simply says, “Thank you” and then appreciates the person on their left. This is a way to end the session on a positive note and create a culture of generosity and gratitude. Overview of Circle Leader Weekly Training Modules Circles encompasses a variety of topics broken down into the following weekly lessons. CIRCLES IMPLEMENTATION ASSESSMENT 22 Session 1: What is Circles In this first session, participants learn about the Circles model, the Leader/Ally relationship, and what Leaders and Allies can expect from the program. Topics covered include confidentiality, the goal of Circles, commitments and barriers to participating in the program, and ground rules. Session 2: Your Story The second session starts with an exercise called New and Good. This practice continues in all subsequent weeks to encourage focusing on positive happenings. This session helps participants identify strengths, challenges, and fears as well as reflecting on areas participants want to change. It emphasizes each participant's individual story and personal journey. Session 3: Poverty and the Trauma of Poverty Session 3 teaches participants about the Federal Poverty Level (FPL) and discusses the barriers that keep people in poverty. The facilitator leads a discussion about understanding the trauma of poverty, including Adverse Childhood Experiences (ACES) and how those affect health, development, and achievement. Session 4: Expectations and Bias In week 4, participants learn about economic classes and the cultural differences between the middle and working classes. They discuss the unwritten rules of the middle class and classism, gender, and racism and how they can be linked to poverty. Session 5: Dream Boards Session 5’s focus is on dreams and wants. Dream boards are made and shared during this week’s session. CIRCLES IMPLEMENTATION ASSESSMENT 23 Session 6: Personal Finance Part 1 The first personal finance session has participants reflect on their relationship with money, calculate their income goals, and begin working on their budget worksheets. Session 7: Personal Finance Part 2 The second personal finance session focuses on completing a budget worksheet and a commitment to track spending over time. Session 8: SMART Goals Session 8 teaches participants about SMART goals and how they work. Participants will create their first SMART goals during this session. Once Circle Leaders are matched with an Ally, they will continue to set and work on SMART goals throughout their participation in the program. Session 9: Building Your Support Network In week 9, participants will identify their existing support networks and learn how to build social capital. Session 10: Communications and Strong Relationships As Circle Leaders prepare to be matched with their Circle Ally, this week’s session focuses on effective communication, understanding the stages of the Leader/Ally relationship, and avoiding the Drama Triangle. Session 11: Work Relationships and the Rules of the Workplace In session 11, participants learn about the hard and soft skills most important for gaining employment and understanding workplace etiquette. They will also consider whether education and training would be a wise investment for their employment goals. CIRCLES IMPLEMENTATION ASSESSMENT 24 Session 12: Congratulations and Next Steps In the final weekly session, participants write a second draft of their budget, review and reflect on the key points from the previous weeks' sessions, and decide how to approach their next 18 months of Circle meetings. Analysis Weekly observations and facilitator self-reflections were collected throughout the 12week cohort beginning in June 2024. The program manager and facilitator held a weekly debriefing session to discuss and compare data collected through the observation tools. In the first few weeks of implementation, this time was used to train the facilitator on using the tool and set expectations for use. Each observer tool was scored to quantify the percentage of intervention implementation and the number of adaptations made during each week’s session. The average percentage of intervention implementations and total curriculum adaptations made across the 12week cohort were calculated. At the end of the 12 weeks, the program manager conducted a final interview with the coach to discuss the entire cohort implementation. Results The June 2024 cohort included six individual participants. All were single-parent females, ages 31-37, with incomes between 0% and 100% of the FPL. Four participants had two children, one with three children and one with six children. The cohort facilitator is a 29-year-old male with a bachelor's degree in Spanish. He has two and a half years of experience as a certified coach, seven months specifically with the Circles program. Five of the six participants who started the program completed the cohort to become Certified Circle Leaders, constituting an 83% completion rate for this cohort. The five participants who completed the program also completed a pre-and post-LifeWorks Self-Sufficiency Matrix to measure changes in income, CIRCLES IMPLEMENTATION ASSESSMENT 25 employment, education, and network support, among other things. Since beginning the program in June 2024, 60% of participants have increased both employment and income, 40% have increased education by enrolling in adult or post-secondary education, and 100% indicate increased network support. The LifeWorks Self-Sufficiency Matrix will be administered to participants every six months throughout program participation. Fidelity The observer and facilitator collected data regarding delivery methods and curriculum components used during the 12-week cohort. This information was used to calculate a delivery percentage for each lesson. The observer and facilitator met weekly to discuss any adaptations used or discrepancies between the observation tools. Results showed a high level of congruence between the two tools, with only two weeks of misalignment. As indicated in Table 3, for the entirety of the program, the observer recorded intervention delivery of 85% while the facilitator recorded delivery at 87%. The two weeks of misalignment included the following weekly topics and methods: in week ten’s lesson, Communication and Strong Relationships, the facilitator indicated completing the delivery method Listening Pairs, but the observer did not; in week twelve’s lesson, Congratulations and Next steps the facilitator indicated completing the Checkin, but the observer did not. Adaptations Another area of data collection includes the recorded number and percentage of curriculum components the facilitator used in each weekly lesson. The observations show that 77% of the curriculum components were delivered throughout the 12-week cohort. Overall, the most commonly omitted component was Listening Pairs, which were implemented in five of the CIRCLES IMPLEMENTATION ASSESSMENT 26 Table 3 Observation Tool Comparison Week & Topic Observer: % of Delivery Method Used Facilitator: % of Delivery Method Used 1-What is Circles? 100% 100% 2-Your Story 43% 43% 3-The Trauma of Poverty 100% 100% 4-Expectations & Biases 86% 86% 5-Dream Boards 100% 100% 6-Personal Finance 1 86% 86% 7-Personal Finance 2 86% 86% 8-SMART Goals 100% 100% 9-Building Your Support Network 100% 100% 10-Communication & Strong Relationships 86% 100% 11-Work Relationships & Rules of the Workplace 86% 86% 12-Congratulations & Next Steps 43% 57% Average Percentage 85% 87% twelve classes. The facilitator indicated that with a small group size, it often felt disruptive to end a small group discussion that was going well in order to have participants talk in pairs. Conversely, the facilitator supplemented several of the weekly topics with additional resources or methods to provide participants with more opportunities to engage with the material. For example, in the week six budget class, the coach provided the group with additional methods to create a budget or track spending than were offered in the curriculum. CIRCLES IMPLEMENTATION ASSESSMENT 27 Discussion This study's results highlight the high level of implementation fidelity in the Weber County chapter of Circles. Additionally, preliminary self-sufficiency matrix data show participants making early gains in the areas of income, employment, education, and network support. However, the challenge of reaching 200% of FPL cannot be denied for this specific group. This section will address the limitations presented in the current cohort, policy barriers, and recommendations for future program direction. Limitations While implementation fidelity, delivery quality, and completion rates in this cohort were high, and participants experienced positive gains in the measured areas, limitations are identified in this study, primarily the small group size and similar demographic characteristics of the participants in this cohort. All participants were similarly aged, single-parent females, which indicates these results may not be generalizable to the larger population of Weber County. Additionally, although the observer and facilitator had a high degree of congruence since the collected information relies on observation and self-reflection, there is a risk of bias that could potentially influence the overall evaluation of the program. Implications This study highlights the challenge of a single-income family achieving 200% FPL. While it is clear that curriculum fidelity was relatively high in this cohort and that participants improved income, employment, and education, a single earner with two dependents would need an annual income of $51,640, or $24.82/hr, to reach 200%. It is equally important to note that a single-earner household would be more likely to be negatively impacted by the benefits cliff because an increase in single wages would likely not equal the abrupt loss of benefits. CIRCLES IMPLEMENTATION ASSESSMENT 28 Recommendations Based on the information collected during this study several recommendations emerge from the findings. First, the importance of a standardized assessment tool that could be used across chapters. A standardized tool would allow CirclesUSA to benchmark results and establish a minimum standard of fidelity, enabling improvements and consistency across chapters. Based on the feedback from the facilitator in Weber County, a standardized tool could allow for modifications based on group size and dynamics. It would also be useful to utilize outside observers who are not connected to the program. Additionally, creating follow-up or tracking systems for graduated participants could help refine strategies that enhance long-term economic stability and sustainability of measured outcomes. Future Directions Going forward, it is important to address the funding requirement of achieving 200% FPL. Unfortunately, the Circles program and the funding secured to implement the program in Weber County do not measure economic stability by the same metric. Where DWS measures success as 200% FPL, CirclesUSA defines success as a family progressing towards 200% FPL. It may be useful to measure both poverty level percentages and felt economic security. Additionally, programs could work with policymakers to understand and develop transitional support policies that ease the impact of losing benefits by promoting gradual benefit reduction and increased social service support as income increases, thereby eliminating the benefits cliff. CIRCLES IMPLEMENTATION ASSESSMENT 29 References Acquaah, M. (2008). Social capital: the benefits, potential costs, and prospects. Journal of Microfinance/ESR Review, 10(2), 4. Aldrich, D. P., & Meyer, M. A. (2015). Social capital and community resilience. American Behavioral Scientist, 59(2), 254-269. Allen, J. D., Linnan, L. A., Emmons, K. M., Brownson, R., Colditz, G., & Proctor, E. (2012). Fidelity and its relationship to implementation effectiveness, adaptation, and dissemination. Dissemination and Implementation Research in Health: Translating Science to Practice, 281-304. Bach-Mortensen, A. M., Lange, B. C., & Montgomery, P. (2018). Barriers and facilitators to implementing evidence-based interventions among third sector organisations: a systematic review. Implementation Science, 13, 1–19. Beegle, D. M. (2003). Overcoming the silence of generational poverty. Talking Points, 15(1), 1120. Berkel, C., Mauricio, A. M., Schoenfelder, E., & Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12, 2333. Björnwall, A., Mattsson Sydner, Y., Koochek, A., & Neuman, N. (2021). Eating alone or together among community-living older people—a scoping review. International Journal of Environmental Research and Public Health, 18(7), 3495. Breitenstein, S. M., Gross, D., Garvey, C. A., Hill, C., Fogg, L., & Resnick, B. (2010). Implementation fidelity in community‐based interventions. Research in Nursing & Health, 33(2), 164-173. CIRCLES IMPLEMENTATION ASSESSMENT 30 Carnochan, S., Samples, M., Myers, M., & Austin, M. J. (2014). Performance measurement challenges in nonprofit human service organizations. Nonprofit and Voluntary Sector Quarterly, 43(6), 1014–1032. Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 1–9. Carvalho, M. L., Honeycutt, S., Escoffery, C., Glanz, K., Sabbs, D., & Kegler, M. C. (2013). Balancing fidelity and adaptation. Journal of Public Health Management and Practice, 19(4), 348–356. Casau, A., & Hyde, S. (2022). The cliff effect: one step forward, two steps back. Chetty, R., Jackson, M. O., Kuchler, T., Stroebel, J., Hendren, N., Fluegge, R. B., ... & Wernerfelt, N. (2022). Social capital I: Measurement and associations with economic mobility. Nature, 608(7921), 108-121. Circles USA (2023). The Next Chapter: 2023 Impact Report. https://www.circlesusa.org/research/impact-reports/ Dowling, K., & Barry, M. M. (2020). The effects of implementation quality of a school-based and emotional well-being program on students’ outcomes. European Journal of Investigation in Health, Psychology, and Education, 10(2), 595–614. Duncan, G. & Holzer, H., 2023. Policies that reduce intergenerational poverty, Brookings Institution. United States of America. Retrieved from https://policycommons.net/artifacts/6950549/policies-that-reduce-intergenerationalpoverty/7860667/ Durlak, J. A., (2016). Programme implementation in social emotional learning: Basic issues and research findings. Cambridge Journal of Education, 46(3), 333-345. CIRCLES IMPLEMENTATION ASSESSMENT 31 Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation in program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327-350. Farahnak, L. R., Ehrhart, M. G., Torres, E. M., & Aarons, G. A. (2020). The influence of transformational leadership and leader attitudes on subordinate attitudes and implementation success. Journal of Leadership & Organizational Studies, 27(1), 98-111. Fixsen, D., Blase, K., Metz, A., & Van Dyke, M. (2013). Statewide implementation of evidencebased programs. Exceptional Children, 79(2), 213-230. Gelmez-Burakgazi, S. (2020). Curriculum adaptation and fidelity: A qualitative study on elementary teachers' classroom practices. Issues in Educational Research, 30(3), 920942. Glasgow, R. E., Harden, S. M., Gaglio, B., Rabin, B., Smith, M. L., Porter, G. C., ... & Estabrooks, P. A. (2019). RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Frontiers in Public Health, 7, 64. Goe, L., & Croft, A. (2009). Methods of Evaluating Teacher Effectiveness. Research-to-Practice Brief. National Comprehensive Center for Teacher Quality. Iantosca, M. H., Kimelberg, S. M., Lewis, D. V., & Taughrin, R. J. “We're gonna get you through it”: The role of bonding social capital in the development of bridging social capital. In Sociological Forum. Jenkins, S. P., & Siedler, T. (2007). The intergenerational transmission of poverty in industrialized countries. Chronic Poverty Research Centre working paper, (75). Jensen, E. (2009). Teaching with poverty in mind: What being poor does to kids’ brains and what schools can do about it. AScD. CIRCLES IMPLEMENTATION ASSESSMENT 32 Kawasaki, J., Quartz, K. H., Martinez, J. F. (2020). Using multiple measures of teaching quality to strengthen teacher preparation. Education Policy Analysis Archives, 28(128). Kramarz, F., & Skans, O. N. (2014). When strong ties are strong: Networks and youth labour market entry. Review of Economic Studies, 81(3), 1164-1200. Kyne, D., & Aldrich, D. P. (2020). Capturing bonding, bridging, and linking social capital through publicly available data. Risk, Hazards & Crisis in Public Policy, 11(1), 61-86. Lee, H., Slack, K. S., Berger, L. M., Mather, R. S., & Murray, R. K. (2021). Childhood poverty, adverse childhood experiences, and adult health outcomes. Health & Social Work, 46(3), 159–170. Meyers, D. C., Durlak, J. A., & Wandersman, A. (2012). The quality implementation framework: a synthesis of critical steps in the implementation process. American Journal of Community Psychology, 50, 462-480. Nolt, K. L., & Leviton, L. C. (2023). Fidelity and Adaptation of Programs: Does Adaptation Thwart Effectiveness? American Journal of Evaluation, 44(3), 322-334. Pac, J., Nam, J., Waldfogel, J., & Wimer, C. (2017). Young child poverty in the United States: Analyzing trends in poverty and the role of anti-poverty programs using the Supplemental Poverty Measure. Children and Youth Services Review, 74, 35-49. Payne, R. K., DeVol, P., & Smith, T. D. (2009). Bridges out of poverty. Texas: Process Inc, p. 304. Privette-Black, M. (2021). Intergenerational Poverty in the United States. Ballard Brief, 2021(2), 7. CIRCLES IMPLEMENTATION ASSESSMENT 33 Robson, S. M., McCullough, M. B., Rex, S., Munafò, M. R., & Taylor, G. (2020). Family meal frequency, diet, and family functioning: A systematic review with meta-analyses. Journal of Nutrition Education and Behavior, 52(5), 553–564. Sawhill, I. V. (2020). Social capital: Why we need it and how we can create more of it. Brookings Institution. U.S. Census Bureau (2022). Poverty in the United States: 2022. Retrieved from https://www.census.gov/library/publications/2023/demo/p60-280.html Wagmiller, R. L., & Adelman, R. M. (2009). Childhood and intergenerational poverty: The longterm consequences of growing up poor. National Center for Children in Poverty. Codianne Nye Project Cover Page Final Audit Report 2024-12-06 Created: 2024-12-06 By: Ellynn Raynor (ellynnraynor@weber.edu) Status: Signed Transaction ID: CBJCHBCAABAAn_GGb9opBVWJdiZWflZVYX44CXdQY2VF "Codianne Nye Project Cover Page" History Document created by Ellynn Raynor (ellynnraynor@weber.edu) 2024-12-06 - 7:16:09 PM GMT- IP address: 137.190.250.0 Document emailed to Charles Dunn (charlesdunn@weber.edu) for signature 2024-12-06 - 7:16:59 PM GMT Email viewed by Charles Dunn (charlesdunn@weber.edu) 2024-12-06 - 9:07:12 PM GMT- IP address: 104.28.48.217 Document e-signed by Charles Dunn (charlesdunn@weber.edu) Signature Date: 2024-12-06 - 9:37:56 PM GMT - Time Source: server- IP address: 172.59.152.173 Document emailed to Daniel Hubler (danielhubler@weber.edu) for signature 2024-12-06 - 9:37:58 PM GMT Email viewed by Daniel Hubler (danielhubler@weber.edu) 2024-12-06 - 10:02:30 PM GMT- IP address: 74.125.212.2 Document e-signed by Daniel Hubler (danielhubler@weber.edu) Signature Date: 2024-12-06 - 10:02:45 PM GMT - Time Source: server- IP address: 98.33.142.110 Document emailed to Cass Morgan (cassmorgan@weber.edu) for signature 2024-12-06 - 10:02:46 PM GMT Email viewed by Cass Morgan (cassmorgan@weber.edu) 2024-12-06 - 10:02:51 PM GMT- IP address: 74.125.212.2 Document e-signed by Cass Morgan (cassmorgan@weber.edu) Signature Date: 2024-12-06 - 10:05:16 PM GMT - Time Source: server- IP address: 137.190.250.0 Agreement completed. 2024-12-06 - 10:05:16 PM GMT