Resource Guide for Institutional Transformation
to Improve Student Success at Community Colleges

October 2005

Resource Guide for Institutional Transformation
to Improve Student Success at Community Colleges

Introduction

Achieving the Dream: Community Colleges Count is a multi-year national initiative that promotes institutional change to improve student success in community colleges. Participating colleges are striving to create a culture of inquiry, evidence and accountability. They identify barriers that prevent students from advancing through college programs, and they use that information to shift institutional policies, practices, and priorities.

This Resource Guide outlines a process community colleges can follow to cultivate a culture of inquiry and evidence to improve student success. For each step of the process, the Guide provides links to practical tools, such as self-assessment inventories, templates for analyzing data, and protocols for conducting focus groups with faculty and students.

This first edition of the guide draws on the experience of the 27 community colleges that are participating in the first round of Achieving the Dream, which began in June 2004. The guide also draws on the experience of Achieving the Dream partner organizations and the coaches and data facilitators who worked closely with the demonstration colleges during 2004-05 (1). Most of the tools included here have been used by one or more Achieving the Dream colleges. All the materials may be reproduced and modified for use by colleges.

Over the coming year, the Achieving the Dream partnership will add more tools to the resource guide. New materials will be posted on the Achieving the Dream website which also provides more information on the initiative, demonstration colleges, and national partner organizations.

While this first edition of guide is written specifically for colleges participating in the Achieving the Dream initiative, most of the materials are relevant for any college seeking to assess its performance, build a culture of evidence, and improve student success. 

Overview of the Process

The process for institutional transformation which is presented here presupposes a college has certain characteristics and commitments. Its leadership must be strongly and visibly committed to the goal of increasing student success. It must be willing to undertake a process of honest self-examination. And it must be prepared to engage in a participatory planning process that includes a broad cross-section of faculty, staff, administrators, students, and members of the larger community outside the college.

The process outlined here will be familiar to anyone who has engaged in strategic planning or continuous improvement. It is a cyclical model, as depicted on the cover. After an initial round of analysis, planning, implementation, and evaluation, the process begins again, generating further goals and further improvement in student outcomes.

The six-step process presented here is a suggested path. Your college may prefer to undertake some of the activities in steps 2, 3, and 4 in a different order. Regardless of the path your college takes, the guidance and tools provide here should be helpful.

1. Begin Work

• Assess the college’s readiness for Achieving the Dream work
• Form teams & assign responsibilities
• Frame issues and develop a work plan
• Announce the initiative and begin building broad-based commitment to Achieving the Dream goals
• Begin generating a longitudinal student cohort database

2. Analyze Current Situation

2a. Gather data

• Ask questions about student outcomes and gaps among different groups.
• Use data to answer questions – and develop additional questions.

2b. Hold honest conversations among executive and senior leadership, faculty, staff, and administrators to:

• Reflect on the data
• Generate the will for change
• Choose priority issues for examination in more depth

2c. Dig deeper to understand problems and the college’s existing strategies

• Diagnose causes of poor student outcomes through further research, including discussions with stakeholders inside and outside the college.
• Examine the college's existing efforts to boost student success -- what's working, what isn't, and why.

3. Set Priorities and Measurable Outcomes

• Set priorities and measurable outcomes for increasing student success
• Place Achieving the Dream priorities at the core of institutional priorities for the coming year. Ensure that the priorities to inform and drive planning for college operations, which in turn must influence allocation of college resources.

4. Develop Strategies to Achieve Goals

• Convene faculty and staff to devise appropriate local strategies.
• Build on existing initiatives and conditions that suggest promising strategic directions.

• Plan to systematically abandon practices and policies that produce poor student outcomes.

• Research effective practices and policies at other institutions and in the literature.
• Set short-term outcomes for each strategy.

5. Implement New Policies and Practices

• Initiatives may include pilot strategies and/or institution-wide changes in policy and practice.
• When the college tests strategies on a pilot basis, it must have a plan for scaling up successful projects.

6. Evaluate and Institutionalize Effective Policies and Practices

• Evaluate pilot projects. When they prove effective, take them to scale, shifting institutional resources and policies as needed.
• Assess institution-wide changes in policy and practice. When they prove effective, integrate them into institutional plans and budgets to ensure that they continue long-term.
• Measure student outcomes, benchmarking progress over time against the college's past performance.

Step 1. Begin Work

Assess readiness

A first step in any institutional change process is to assess the institution’s strengths and weaknesses and identify challenges that will need particular attention. In Achieving the Dream, it is important to examine the college’s commitment to the initiative’s goals and process for institutional change, and in particular to gauge the institution’s readiness to develop a culture of inquiry and evidence.

The Achieving the Dream process uses a model of distributed leadership. The president or chancellor must have a vision for student success and equity and must be able to mobilize broad support for that vision throughout the college and community. The president must place student success at the center of the institution’s agenda, as reflected in annual operational plans and budgets. At the same time, involvement of faculty, staff, administrators, and trustees is essential. Members of the college community will support change most readily when they share responsibility for diagnosing the problems and crafting solutions. And better strategies will emerge when the people who are closest to the students help redesign programs, services and structures.

Institutional Transformation: Presidential Readiness and Commitment   is a list of questions to help presidents reflect on their institution’s strengths and weaknesses vis a vis Achieving the Dream priorities and process. It can help them gauge where they will need to put their energy to lead a successful institutional change effort. The inventory can be useful for other administrators as well.

The Community College Inventory is a discussion tool to help colleges assess their strengths and weaknesses in supporting student persistence, learning, and attainment. Parts I and II of the eight-part inventory are particularly relevant at this stage in the planning process. Part I addresses the institution’s vision, values and culture; Part II measures the extent to which the college has a culture of evidence. The inventory can be particularly useful if members of a team fill it out individually, tally the results, and use the results as a springboard for discussion about how the college needs to change to solidify its commitment to student success and develop a strong culture of evidence.

Principles of Institutional Transformation  is a rating scale to assess how well the college’s current culture and practice are aligned with Achieving the Dream’s goals and process for institutional change. It asks college personnel to assess, for example, the extent to which their institution currently has a student-centered vision, promotes teamwork across organizational boundaries, and encourages constructive problem solving.

Form teams

Achieving the Dream uses a team approach to institutional change. Teams are important for several reasons. They can generate energy for change, as team members reinforce one another's commitment and work together to infuse their shared vision throughout the institution. Teams bring sustainability, increasing the likelihood that the college will carry on the work even if the institution's top leadership changes. Also, several heads are better than one. When a group of people share responsibility for leading a change process, their multiple perspectives enrich the mix of ideas, and better solutions emerge. That richness occurs especially when the team gives voice to people outside the college's traditional leadership circle.

To lead the process of inquiry, planning and implementation, Achieving the Dream colleges create a Core Team and a Data Team. Suggested team composition and responsibilities recommends the kinds of people to include in each team and how to divide responsibilities between the two teams.

Frame issues and develop the work plan

An early task for the Core Team is to develop a plan of work for a 9-12 month period of inquiry, data analysis, and planning. The Sample Flow of Institutional Change Work is a timeline showing first-year activities for the core team, data team, Institutional Research office, college leadership council, and others. It incorporates required Achieving the Dream events and deadlines into a suggested plan of work. The Template for First-Year Work Plan is a chart colleges can use to plan their first year of work. It lists key activities described in this Resource Guide and has spaces where the college can indicate how it will undertake each step, who will be responsible, and when the work will be done.

Announce the initiative and begin building broad-based commitment to Achieving the Dream goals.

Early visibility can begin raising awareness of Achieving the Dream and building broad support for its goals. Many Achieving the Dream colleges have used their fall convocation and/or in-service day to announce the initiative, promote its goals, and explain the process and how members of the college community will be involved in the coming year. See Sample agenda for a fall convocation (** will be added this summer)

In introducing Achieving the Dream, it is important for college leaders to emphasize that this is not “just another project.” Rather, Achieving the Dream is an opportunity for long-term institutional transformation that will enable the college to align its practices more closely with its values. In signing on to Achieving the Dream, the college commits to making decisions based on evidence about what works to increase student success.

Generate longitudinal database

When a college joins Achieving the Dream, an early requirement is to collect and report data on student cohorts, to create a multi-year database that the college can use to analyze student outcomes over time.

When colleges analyze data on student outcomes, they typically look at each semester separately to determine, for example, how many of the students who enrolled in developmental math courses in Fall 2004 completed those courses. In Achieving the Dream, colleges look at that type of “cross-sectional” data as well as “longitudinal” data. The latter are especially valuable for benchmarking institutional performance and monitoring institutional progress, because longitudinal data allow colleges to track what happens to their students over time. The college can determine, for instance, how many of the students who enter developmental math complete the developmental sequence, advance through gatekeeper courses, remain enrolled over time, and eventually complete certificates or degrees. The college can identify key junctures where it loses students, and it can identify particular groups of students that fall out at different points.

The longitudinal database created by Achieving the Dream colleges tracks cohorts of first-time students (both full-time and part-time) who are seeking degrees or certificates. Each year the colleges collect and report data on a new cohort of entering students as well as data on prior year cohorts. In creating this database, all Achieving the Dream colleges use the same definitions and measure the same data elements. Detailed instructions for reporting data to the national Achieving the Dream database can be found in theInstructions for Data Submission.

Step 2. Analyze Current Situation

2a. Gather data

• Ask questions about student outcomes and gaps
• Use data to produce answers

Achieving the Dream colleges use data on student characteristics, course-taking patterns and educational outcomes to identify areas where students experience low rates of success. Colleges can use existing institutional data as well as data available from state systems and data generated for the national Achieving the Dream database. Both qualitative and quantitative data can contribute to the college’s understanding of problems and their causes.

It is important to begin with a demographic profile of the college’s enrollment. For example, how many students are full-time vs. part-time? How many are 18-23 years old vs. older adults? What is the mix of students by race, ethnicity, gender, income, and other characteristics? What percentage of entering students are referred to developmental courses?

Then colleges should ask questions about student outcomes. Achieving the Dream particularly urges colleges to explore the following questions:

• Which developmental courses have low success rates that prevent substantial numbers of students from advancing?
• Are there any notable gaps among different groups of students (by race, ethnicity, income, age, or other characteristics) in the rate at which they complete these courses?
• Which “gatekeeper courses” (the first-level curriculum courses that are prerequisites for higher-level courses) have low success rates that prevent large numbers of students from advancing through college programs?
• Are there any notable gaps among different groups of students (by race, ethnicity, income, age, or other characteristics) in the rate at which they complete these courses?
• What percent of course credits are completed (with a grade of C or higher) relative to the number of credits attempted by students each term?
• Are there any notable gaps in successful course completion among different groups of students?
• What percent of incoming students reenroll for a second semester? What percent reenroll for a second year?
• Are there any notable gaps among different groups of students in rates of retention from semester to-semester or year-to-year?
• What percent of students earn certificates and degrees? (2)
• Are there any notable gaps in graduation rates among different groups of students?

As noted, longitudinal data that tracks cohorts of students over a period of years is especially useful in answering these kinds of questions. The Achieving the Dream database allows colleges to examine the experience of cohorts of first-time students, starting one or two years before the college joined the initiative and for every year going forward. Some colleges also have access to institutional or state data that track cohorts going back in time. Longitudinal data on student cohort are extremely valuable for benchmarking the college’s performance and monitoring its progress. They are also useful in discerning gaps in success among different groups of students, including differences by race, ethnicity, age, sex, income, and other characteristics.

The data analysis should be presented to the college leadership teams for reflection and interpretation. In addition, data illustrating gaps in student outcomes should be used to engage faculty, staff, administrators, and students in honest conversations about why these gaps occur and what can be done to close them.

The tools listed above offer examples of types of data to look at, how to array and analyze the data, and how to present it effectively for discussion. The Example of Data Sources and Uses shows the range of data sources tapped by Tallahassee Community College, and how each source was used to explore questions about student outcomes. The Comprehensive Template for Data Analysis is a chart illustrating one way to array data by race, ethnicity, age, and other characteristics. The Example of how an Achieving the Dream college analyzed and presented data comes from San Antonio College, whose Office of Institutional Effectiveness produced short research briefs to present data findings to the wider college community. The Fishbone Exercise guides a team in discussing root causes of problems. It can be useful here and in Step 2c.

2b. Hold honest conversations among executive and senior leadership, faculty, staff and administrators to:

• Reflect on the data
• Generate the will for change
• Choose priority issues

It is imperative that college teams engage members of the college community broadly in efforts to review and understand the data. Broad-based involvement of faculty and staff is essential to defuse any defensiveness that may emerge in response to the facts about student outcomes. These conversations can begin to engage faculty and staff in more accurately defining and diagnosing the problems and setting priorities for change. These discussions are also an important opportunity for the college leaders to convey their strong commitment to addressing gaps in achievement. Discussions about the data should raise awareness of challenges and strengthen the institution's commitment to improving student outcomes and closing gaps.

Many Achieving the Dream colleges hold day-long retreats to discuss findings from the data. Through small group and plenary discussions, participants consider the implications of the data and choose priority issues to examine in greater depth. They may also begin generating hypotheses about the causes of high-priority problems. SeeSample agenda for a retreat to discuss data and set priorities. (** will be added this summer)

The data are likely to include disturbing findings, and it is essential for top administrators to model “truth-telling” behavior and encourage open, honest dialogue. Skilled facilitation may be needed to guide discussions that extract the unspoken assumptions that influence faculty, staff and administrators’ behavior, especially when potentially divisive issues related to race and class are involved. The Achieving the Dream Technical Assistance Roster (available to demonstration colleges on the initiative’s website) lists consultants who can help structure productive conversations on tough issues and train college staff to lead effective discussions.

2c. Dig deeper to understand problems and the college’s existing strategies

• Diagnose the causes of gaps in student outcomes
• Examine the college's existing efforts to boost student success

As groups of faculty, staff and administrators examine findings from the data, they should begin to generate hypotheses about the causes of weak student outcomes and gaps among groups. It is essential to understand causes of problems in order to develop effective strategic approaches.

For example, if data show that Latino, African American, or Native American students have a 25% pass rate in developmental math while white students pass at a rate of 50%, the group charged with diagnosis might postulate several possible causes, such as: different levels of pre-college math preparation; differences in faculty interaction with students from different race and ethnic groups; limited use of teaching/learning strategies that are most effective with certain student groups; or differences in the amount of time students spend studying. The group would then pose questions which could be explored through further research, for example:

• Preparation: Do Latino [or African American or Native American] students tend to take different math curricula or attend different high schools than white students? Do they tend to be older than white students and hence remember less of the math they studied in school?
• Student-faculty interaction: Are there differences in the amount or quality of student-faculty interaction among African American [or Latino or Native American] students and white students?
• Teaching strategies: Do certain instructional approaches appear particularly effective or ineffective with certain types of students?
• Time to task: Are there differences in the amount of time students devote to their class assignments among Native American [or Latino or African American] and white students?
• Open-ended: What do students of color students indicate as barriers to passing developmental math?

If the overall success rate is also low, there may be opportunities for improvement in teaching strategies.

The Force Field Analysis tool (included below in Step 4) is a useful tool to generate hypotheses about causes of problems. The Fishbone Exercise (included above in Step 2a) can also help guide discussions about root causes of problems.

After generating questions and hypotheses, the group should decide which questions to focus on and what research methods to use. Methods can include digging deeper into the quantitative data as well as canvassing students, faculty, or others through surveys, focus groups or interviews.

Focus groups with students can be a particularly valuable source of information on what works for students and what doesn’t; they can also generate ideas for strategies. Why hold focus groups with students is a Powerpoint presentation that illustrates what a college can learn from structured conversations with students. The Student Focus Group Script guides colleges in holding focus groups to explore the barriers students experience and solicit students’ suggestions for removing barriers to success.

In diagnosing problems, it may also be helpful to consult with groups in the larger community, such as feeder high schools, community organizations or social service agencies that serve low-income populations, baccalaureate institutions where students are likely to transfer, and employers. The Stakeholder Dialogues tool guides colleges in structuring meetings with community stakeholders to solicit their input on barriers to student success. Such meetings can also strengthen the college’s relationship with the community and gain community buy-in and participation in whatever student success initiatives the college ultimately plans.

Teams should also examine the institution’s existing efforts to improve student performance. Most colleges have multiple programs and services intended to help students succeed. It is important to understand the scope, strengths and weaknesses of these efforts to guide decisions about improving and expanding existing programs. Key questions to ask include the following:

• What is the nature of the support provided?

• Which students are involved?
• How many students are served?
• Is there evidence that the effort is effective? For what groups of students?
• What are the costs of the program or service?
• How is it connected to other efforts to improve student success?
• If the program or service is small-scale, what would it take to implement it college-wide?

In addition to looking at specific programs and services, there is value in examining college policies and practices that may be unintended barriers to student success. Based on such examinations, Achieving the Dream colleges have altered policies regarding registration, financial aid application, testing and placement, attendance, and other administrative issues. The Achieving the DreamPractice and Policy Checklist provides a framework to help colleges consider which institutional policies to examine.

Surveys and focus groups with faculty, staff, and students are invaluable in pinpointing the administrative policies that get in the way of student success. Several colleges have used data from the Community College Survey of Student Engagement (CCSSE) to gain insight into such issues as the extent of student-faculty interaction, use of active and collaborative learning; barriers to persistence, and the amount of help students receive on transitions to career and transfer. Other colleges have used AACC’s Faces of the Future survey to understand their students’ demographic characteristics, their reasons for attending college, and their satisfaction with the college experience including such issues as accessibility of services, quality of advising, and treatment of students of different racial and ethnic groups. Some Achieving the Dream colleges have conducted their own student satisfaction surveys to identify problems in college services ranging from financial aid to the registration process to bookstore operations.

Business Process Analysis is a useful method for examining routine college functions such as placement testing or registration, and determining how to make them more efficient and more student- and staff-friendly. The method guides the college in mapping out the steps that all the relevant players must take to complete a process such as placement testing, and it helps identify unnecessary hurdles imbedded in the process. Patrick Henry Community College used this technique in 2004-05, and the analysis convinced the college to eliminate a testing fee which was found to have a negative impact on students.

Structured individual interviews or focus groups with faculty and staff also can provide useful information about the college’s current practices. The Faculty Interview Protocol provides a guide for interviewing faculty to assess their awareness of and involvement in Achieving the Dream while also soliciting their views on current college practices. Good questions to ask about the college’s practices include the following:

• In what ways do you or your department currently try to meet the needs of students who are struggling (whether for academic, personal, or financial reasons)?
• Do you or your department identify at-risk students and encourage them to persist? If so, how? Is there evidence that your efforts make a difference for these students?
• Are there any particular techniques (modes of teaching, mentoring, advising, etc.) or clusters of techniques that you believe are most effective in boosting students’ odds for success? What are these approaches, and why do you believe them to be

Step 3. Set Priorities and Measurable Outcomes

Set priorities and measurable outcomes.

Place Achieving the Dream priorities at the core of institutional priorities for the coming year. Ensure that the priorities to inform and drive planning for college operations, which in turn should influence allocation of college resources.

The six-step process laid out in this Guide is a suggested path; some colleges may choose to change the order of certain activities. For example, some colleges will make final decisions about priority issues in Step 2 when they hold a retreat to discuss their data. Other colleges will set tentative priorities at that time, but will wait to decide on directions until they have completed their inquiry into causes of problems and current college practices. Similarly, while some colleges will be ready to set measurable outcomes in Step 3, others will prefer to develop strategies for their priority issues (Step 4) before articulating measurable outcomes.

Regardless of when in the process the college decides to set priorities and measurable outcomes, its priorities should emerge from its analysis. It should choose areas where it identifies a clear need to improve student outcomes overall and/or a need to close gaps between groups. It should seek opportunities to build on existing efforts that are effective, as well as changing policies and practices that are not working for students. It may set priorities for building the college’s capacity to use data or upgrading other institutional capacities that are inadequate as the college seeks to develop a stronger culture of evidence. Outcomes should clearly state what that colleges expects to change after two years and four years.

The Sample Agenda for a Leadership Retreat provides ideas for how to structure a meeting to set priorities based on the insights gleaned from the college’s analysis.

Here are two examples of priority areas, how they emerged from the analysis of a hypothetical college, and measurable outcomes.

Priority area: Developmental math

Rationale:

• 70% of entering credit students are placed in developmental math.
• 50% of developmental math students successfully progress to Math 101.
• 50% of developmental math students are African American, and only 25% of African Americans progress to Math 101.

Measurable change after two years:

• 55% of all developmental math students successfully complete and enroll in Math 101.
• 45% of African American students in developmental math successfully complete and enroll in Math 101.

Measurable change after four years:

• 60% of all developmental math students successfully complete and enroll in Math 101.
• 60% of African American students in developmental math successfully complete and enroll in Math 101.

Priority area: Data analysis and engagement capacity

Rationale:

• Our institutional research office consists of 0.5 FTE and is not able to meet the demand for data analysis inspired by our joining Achieving the Dream.
• Our institution is not accustomed to making decision based on student outcomes or engaging with the community in dialogue about student success.

Measurable change after two years:

• Math, English, and Developmental studies departments will submit annual budget requests based on solid understanding of their departments’ strengths and weaknesses regarding student persistence and learning.

Measurable change after four years:

• The college will use effective techniques to engage students and the larger community in identifying student needs, assessing institutional performance, and recommending areas for priority attention. All departments will submit annual budget requests based on solid understanding of their departments’ strengths and weaknesses regarding student persistence and learning.

When the college has chosen its priorities for Achieving the Dream, it is important for the college leadership to incorporate them into operational plans, strategic plans and budgets. Part III of the Community College Inventory -- which asks questions about institution’s strategic focus, planning and resource allocation -- can help a college assess how well it is integrating its priorities into institutional plans. Another tool referenced under Step 1 -- Institutional Transformation: Presidential Readiness and Commitment – may also be worth revisiting here.

Step 4. Develop Strategies

Convene faculty and staff to devise appropriate local strategies.

Build on existing initiatives and conditions that suggest promising strategic directions.

Plan to systematically abandon practices and policies that produce poor student outcomes.

Research effective practices and policies at other institutions and in the literature.

Set short-term outcomes for each strategy

Most Achieving the Dream colleges appoint work groups to research strategies. Faculty and student services staff should be central members of these work groups. They are best positioned to know what will work with students, and they must “own” reform efforts if these efforts are to be implemented successfully widely across the institution. The college may also involve other stakeholders – including students and community members – in developing strategies.

The most effective strategies will be those that take into account the forces for and against change that exist in the college and community. A good starting place for a strategy work group is to revisit the data and brainstorm the forces (attitudes, beliefs, policies, and practices) that reinforce the status quo and forces that can be harnessed in support of change. The Force Field Analysis tool can help to guide this discussion.

Findings from the force field analysis can help the group consider what types of strategies make most sense, given local circumstances. In some cases, programmatic strategies (e.g., a new student success course) may be the best approach. It is also be important to consider other types of strategies, such as modifying college policies (e.g., late registration), improving administrative operations (e.g., changing financial aid practices), or providing training and incentives to improve teaching. Here are a few examples of the kinds of problems that may be identified by the force field analysis and suitable strategic responses:

In addition to looking at existing forces and programs currently in place, the group may also decide to search the literature for evidence that particular approaches have worked at other colleges. It is important to look for convincing evidence that strategies have proven effective. For a review of the literature on practices that promote persistence and success, see the Community College Research Center’s Paths to Persistence report. This report provides a helpful perspective on how to judge the quality of evidence about the effectiveness of various interventions. In examining strategies that have worked elsewhere, work groups should seek to identify the crucial elements in the design that can make the approach successful at their own institution.

Strategy work groups may also consult with experts in areas where the college seeks to improve its practices, such as developmental education, financial aid, or student services. And they may visit other colleges that have strong track records in a program area their college is pursuing (e.g., learning communities, advising systems, etc.) Materials from the Achieving the Dream 2005 Strategy Institute may provide useful ideas about practices as well as experts and other colleges to contact.

Achieving the Dream encourages colleges to develop systemic interventions that will have a significant impact on student performance, rather than “boutique” programs or services that will benefit only small numbers of students. However, in many cases, it may be advisable for college to test new strategies on a pilot basis before implementing them system-wide. Examples of systemic activities include:

• Redesigning assessment, advising and counseling to ensure that students are properly placed in programs according to their needs and goals, and that students have a clear plan for their college study.
• Incorporating orientation to college and college success skills in developmental programs.
  

Step 5. Implement New Policies and Practices

Initiatives may include pilot strategies and/or institution-wide changes in policy and practice.

When the college tests strategies on a pilot basis, it must have a plan for scaling up successful projects.

Examples of Action Plans provides model four-year plans for implementing strategies to raise pass rates in developmental math and strategies to boost a college’s data analysis capacity and engagement with stakeholders.

Pilot programs can be a good first step in phasing in a systemic change. However, piloting needs to be done strategically so projects do not begin and end as isolated experiments with minimal impact. When colleges implement pilot programs, it is essential to have a plan for evaluating the initiative. If it is successful, the college must be prepared to take it to scale.

Step 6. Evaluate and Institutionalize Effective Policies and Practices

Assess new policies and practices. If they prove effective, expand and institutionalize them.

It is essential for colleges to measure the effectiveness of new policies and practices. As discussed in Evaluating Program Strategies (a presentation by MDRC, the organization conducing the national evaluation of Achieving the Dream), there are two kinds of program evaluation. Process evaluations examine program operations and outcomes. They measure how services are being delivered and how students are faring. Impact evaluations measure a program’s effects, to determine whether students are better off with the program than they would be without it. An impact evaluation requires some sort of comparison. The college can compare the performance of students before and after participating in the program, or it can compare the performance of students in the program with that of similar students who are not in the program. The college should consider whether an impact evaluation is needed to determine the effectiveness of a pilot program and, if so, what methodology to use.

Besides looking at student performance, it can also be valuable to compare the attitudes of students in pilot programs with those in traditional programs, to see whether the pilot results in higher levels of student satisfaction or motivation. Faculty attitudes and judgments about pilot programs are also important to measure.

When new practices prove effective, the college should shift resources and/or change policies to institutionalize these practices for all students who could benefit. Taking new practices to scale at a college ultimately means diverting resources from other activities that are not as effective in improving student outcomes. This is one reason it is essential for the college’s top leadership and trustees to work closely with faculty, staff and administrators during the analysis, planning and implementation process.

Besides assessing the effectiveness of particular interventions, the college should continue to monitor student outcomes and gaps among groups, measuring progress over time against the institution’s own past performance. To the extent that this process is integrated into the college’s ongoing systems for program review, strategic planning and budgeting, the process -- and the improvements in student outcomes that it generates -- will be institutionalized and sustained.

Benchmark student outcomes and monitor the college's progress over time.

The Achieving the Dream initiative recommends that the best way for a college to measure its progress in improving student success over time is to benchmark against its own past performance. Colleges often seek to compare themselves with other institutions. There are several limitations with that approach. First, even within the same state, every college serves a unique mix of students, with a unique set of programs and services and a distinctive set of resources, strengths and constraints. So, by comparing their own performance with that of other institutions, colleges are often comparing apples and oranges. Second, when benchmarking performance against another institution, one college may appear to improve simply because its comparison college has declined. Moreover, finding out how a college is performing compared with other institutions does not give the college any information the specific nature and causes of its gaps in performance. And finally, it is too often the case that even the highest-performing institutions may not be experiencing the levels of student success that Achieving the Dream colleges aspire to attain.

This does not mean that colleges should not be continually looking to see how well other colleges are doing, particularly institutions with similar student bodies and service areas. It is particularly helpful to benchmark performance against colleges that are “like” institutions, in terms of important characteristics like percent of low-income or minority students, percent of academically under-prepared students, etc. Over time, the Achieving the Dream database will afford opportunities for meaningful comparisons of this kind. Recognizing that it is performing less well than peer institutions can sometimes jolt a college into action. And colleges do often come up with new and better ideas for doing things by learning from other institutions. Ultimately, though, the value of any “best practice” is whether it works with your own students and within the context of your college’s own mission and goals.