Creating a Culture of Evidence at UCCS

Brad Bayer, Senior Executive Director of Student Life and Leadership, explains how staff at the University of Colorado Colorado Springs used CAS to help them create a culture of evidence.

We have all studied the books, attended the trainings, contemplated the research, and developed action plans. Now, more than ever, it is important that we learn from the on-the-ground stories of those among us that have put theory into practice. Through our story, we address how one of the fastest growing universities in the west (University of Colorado Colorado Springs) successfully launched a program review process, grounded in CAS standards, encompassing 22 different program areas in Student Affairs with no increases in funding or staffing. This applied and pragmatic program assessment model has received rave reviews from HLC accreditors, external reviewers, and university leadership alike, while being positively received by staff members. Our CAS story can help readers learn how to build an authentic culture of evidence from the ground up.

THE BEGINNING
In the early summer of 2015, the Student Life and Leadership Department at the University of Colorado Colorado Springs (UCCS) initiated a self-study process using the CAS Self-Assessment Guide for Campus Activities Programs (2012 version) in an effort to gauge the quality of the overall program and to determine a course of action for the future. The UCCS Student Life and Leadership program had experienced significant growth in quantity and quality throughout the eight years prior to the self-assessment, as student enrollments on the campus increased by nearly 3,000 students during this time period.  

We followed the systematic process as outlined in the CAS Self-Assessment Guide and developed targeted outcome goals for the department. Upon completion of the self-study, I presented our findings to my supervisor, the Vice Chancellor of Student Success, and he was impressed with the results of the self-study and offered to provide the resources for us to conduct an external review of Student Life and Leadership.

The timing of this support could not have been better, as I attended the NASPA Annual Assessment and Persistence Conference in Boston in late June of 2015, and participated in a pre-conference workshop entitled "Building a Culture of Evidence: I've Got the Basics, Now What?", facilitated by Marguerite Culp, co-editor of the NASPA text, Building a Culture of Evidence in Student Affairs:  A Guide for Leaders and Practioners. The full-day pre-conference provided me with the direction and resources necessary to build a program assessment process not only for a single department, but also for an entire division. On a coffee break, I asked Marguerite Culp who would be a good external reviewer to contact for campus activities programs, and she recommended Dr. Darby Roberts from Texas A&M University. I noted this contact for future reference. Throughout the rest of the conference, I attended sessions presented by CAS representatives, began to outline the remaining steps for the Student Life and Leadership assessment process, and formulated a proposal to develop a program review process for the full division of Student Success at UCCS. 

Upon my return to campus, I met with the Vice Chancellor of Student Success and presented a proposal to adopt a broad program review process at the divisional level incorporating a step-by-step approach and utilizing best practices. He championed the idea and vetted the concept with campus leadership, gained support, and officially approved the development and implementation of the division-wide program assessment process. The outline of the 5-step program assessment plan is as follows: 
 
UCCS Student Success Program Assessment Plan
Step 1:  Self-Study

• Department(s) conduct their own self-study based on established CAS Standards for their respective area(s).  An assessment instrument/scoring rubric that is grounded in CAS standards and best practices is used by each department for the self-study.
  

Step 2:  Internal Review

• UCCS Student Success Internal Assessment Team made up of representatives from the Student Success division, one faculty member, and one staff member from another division conduct an assessment of each respective department within the division.
  
• Internal Assessment team is trained in advance based on established external review guidelines, objectives, and criteria.
  

Step 3:  External Review

• External reviewer conducts a comprehensive external assessment of the department(s) and provides feedback on what is going well and recommendations for the future.
  
• The external reviewer should be someone with highly regarded credentials and credibility within the specific area being reviewed.
  

Step 4:  Analysis of External Review Results and Goal Setting for the Future

• Departments and SSEM division review the external review results and set both short-term and long-term goals for the future in a Strategic Plan. 
  
• External review summary is shared with the Leadership Team for analysis and input.
  
• External review is saved as an institutional record in a repository to support strategic planning, accreditation, and university-wide planning.
  

Step 5:  Establish a Rotating Assessment Cycle

• Each department will rotate through the assessment cycle as noted on the program assessment calendar.
  

*Created by Brad Bayer, Senior Executive Director of Student Life and Leadership, University of Colorado Colorado Springs

Just as we gained the approval for the divisional program assessment plan, we completed the assessment process for Department of Student Life and Leadership. Dr. Darby Roberts, Director of Student Life Studies at Texas A&M University and nationally recognized as a leader in student affairs assessment, completed our external review in January of 2016 and the feedback from the external review and self-study informed the Student Life and Leadership Strategic Plan that was completed in the fall of 2016.

Also in the fall of 2016, a division-wide assessment calendar was formalized where each of the 22 different program areas in the UCCS Division of Student Success rotates through the program assessment cycle.  The current assessment cycle calendar is as follows:

UCCS Student Success Program Assessment Calendar
2015-2016

• Campus Activities Programs - Student Life and Leadership

2016-2017

• Academic Advising Programs
• Family and Parent Programs
• MOSAIC – Multi-Cultural Student Programs and Services

2017-2018

• Student Conduct Programs - Dean of Students
• Undergraduate Admission Programs and Services
• Learning Assistance Programs – Excel Tutoring Centers & First Year Experience

2018-2019

• Orientation Programs
• Financial Aid Programs
• Student Leadership Programs

2019-2020

• Lesbian, Gay, Bisexual, Transgender Programs and Services
• Registrar Programs and Services
• First Year Experience

2020-2021

• Disability Resources and Services
• TRIO and Other Educational Opportunity Programs (CSTEME)
• Transfer Student Programs and Services

2021-2022

• Veterans and Military Programs
• Alcohol and Other Drug Programs
• Fraternity and Sorority Advising Programs

2022-2023

• Civic Engagement and Service-Learning Programs
• Sexual Violence-Related Programs and Services
• Career Services

*Created by Brad Bayer, Senior Executive Director of Student Life and Leadership, University of Colorado Colorado Springs

INTERNAL REVIEW TEAM
The Internal Review Team was formed in the summer of 2016. Representatives serving on the team were selected based on their broad work-related backgrounds, leadership, expertise, and interests in assessment. The UCCS Student Success Internal Assessment Team is comprised of representatives from the Student Success division, one faculty member, and one staff member from another division. Prior to conducting internal reviews for the respective departments, the Internal Review Team received advanced training in the foundations of establishing a culture of evidence in Student Affairs, as well as in establishing the internal review guidelines, objectives, and criteria. 

The following staff and faculty comprise the Student Success Internal Assessment Review Team at UCCS (pictured at top):

• Brad Bayer, Senior Executive Director of Student Life and Leadership (Chair)
  
• Amanda Allee, Assistant Dean of Students
  
• Jacqueline Bisaillon, Associate Director of Recruitment
  
• Ellen Burkhart, Director of First Year Experience & Family and Parent Programs
  
• Brett Fugate, Director of Academic Advising
  
• Skyler Rorabaugh, Director of Campus Recreation
  
• Amy Sutz, Assistant Director Financial Aid Compliance
  
• Patty Witkowsky, Assistant Professor for Leadership, Research, & Foundations; SAHE Coordinator
  

UPDATE ON PROGRESS
In the summer of 2016, I facilitated pre-program assessment training for the Department of Academic Advising staff where the content revolved around establishing a baseline understanding along a “continuum” in terms of how each individual uses data/evidence, how individuals view their department's use of data/evidence, and how individuals view the university's use of data/evidence. Upon completion of this portion of the training, reflection and dialogue occurred and a universal understanding materialized as to why assessment, data/evidence are important to the ongoing effort of continuous improvement. Staff motivation and buy-in were enhanced through the pre-program assessment training, and it is perhaps the most critical component to incorporate when considering establishing a long-term program assessment initiative. Staff need to “buy-in” to the “why” of assessment before a formalized, systemic program review plan is adopted. Keeping these trainings light-hearted and not heavy handed is also crucial.

Throughout the rest of the summer of 2016, and after the completion of the pre-program assessment training, the Department of Academic Advising conducted their Self-Study utilizing the CAS Self-Assessment Guide for Academic Advising Programs (2015 version).

In fall of 2016, the Student Success Internal Assessment Review Team conducted the program assessment of Academic Advising utilizing the same CAS Self-Assessment Guide. The team reviewed evidence files, formulated a bank of interview questions for various groups, conducted one-day interviews with a variety of campus stakeholders, including faculty, staff, administration, students, as well as the academic advising staff members and leaders. Each of the Internal Review Team members first individually assessed and assigned a rating score to each of the twelve domains outlined in the CAS Self-Assessment Guide for the Academic Advising Program. From there, the full team met, reached consensus ratings for each assessment domain, and provided commentary specific to strengths, opportunities, and recommendations for the future. A full summary of the Internal Review Team's assessment was provided to the Director of Academic Advising in early January 2017.

External reviewers from NACADA were on campus January 26 and 27, 2017 to conduct on-campus interviews as part of the Academic Advising External Review. The reviewers also used the CAS Self-Assessment Guide scoring rubric as part of their overall process. A full summary from the external reviewers is expected by the end of February 2017. Moving forward, the Department of Academic Advising will use the information gathered from the self-study, internal review, and the external review to inform their Strategic Plan.

In collaboration with the Academic Advising Director, I will conduct strategic planning training with the Academic Advising staff utilizing the SOAR planning model and established strategic planning framework.

At present, I've met initially with the Family and Parent Program staff and reviewed the overall program assessment process, established the timeline for the various steps in the program assessment process. Immediate next steps include purchasing and reviewing the CAS Self-Assessment Guide for Parent and Family Programs, pre-training the department staff in the fundamentals of assessment, and determining the dates for both the Internal Review Interviews and External Review Interviews.

Currently, the Department of Student Life and Leadership is forging ahead with accomplishing spring semester 2017 goals as outlined in their new Strategic Plan. The overall plan incorporates a matrix which outlines specific goals over multiple semesters and years. Training in the development of learning outcomes is on the horizon. 

As an interesting side note, the University of Colorado Colorado Springs underwent accreditation by the Higher Learning Commission (HLC) in November of 2016. The overall feedback was very promising with no official findings, and co-curricular assessment received positive commendations by the accreditors. Also, the UCCS Division of University Advancement recently adopted elements of the Student Success Program Assessment Review Plan into their divisional Assessment Plan.

WHERE DO WE GO FROM HERE?

• Continue implementation of the full program assessment cycle for each of the 22 program areas within in the Division of Student Success utilizing the CAS Self-Assessment Guides. Monitor progress and make adjustments as necessary.
  
• Establish department-level Strategic Plans for each of the 22 different program areas that align with the Student Success Division's Strategic Plan and the UCCS Strategic Plan.
  
• Incorporate Student Learning Outcomes based on CAS standards into Strategic Plans and related program goals for each department respectively.
  
• Build a realistic and relevant “culture of evidence” in Student Affairs with limited resources that can be practically applied and maintained, is embraced by staff, faculty, and administration, supports student learning and development, and promotes continuous improvement.
  

---
Brad Bayer is the senior executive director of student life and leadership at the University of Colorado Colorado Springs (UCCS). He leads program assessment efforts for twenty-two different department areas within the Division of Student Success, an initiative that was launched with no additional funding or staffing. He also provides executive leadership for the office of student activities and community service, 174 clubs and organizations, the student government association, the student newspaper, sorority and fraternity life, the Chancellor's Leadership Class, the UCCSlead leadership certificate program, broad student leadership development, disability and testing services, veteran and military student affairs, the multicultural office for student access, inclusiveness and community (MOSAIC), and the LGBT Resource Center. Recently, Bayer and members of the student life and leadership staff developed a co-curricular transcript option for undergraduate students, the first of its kind at UCCS. Bayer has presented nationally, regionally, and at the state and local levels on a variety of topics related to student engagement, leadership, support, success, and professional development. He currently leads a dynamic team of 42 professional staff and 81 student employees and oversees a 4.9 million dollar budget.