Increasing Posttesting to Improve Measurable Skill Gains

Under the Workforce Innovation and Opportunity Act (WIOA), adult education programs must collect data on program participants for the primary indicators of performance. These indicators include measurable skill gains (MSG), or “a measure of a participant’s interim progress towards a credential or employment1.” The MSG indicator allows adult education programs to determine how well they are helping students make progress toward this goal and this information can be used to support program management and improvement efforts.

Although many factors affect an adult education program’s MSG outcome, a critical one is a program’s pretesting and posttesting rate. It is not possible to show an educational functioning level (EFL) gain based on testing without a posttest. Students may be making gains that are not being captured because the students are not posttested. By increasing posttesting rates, a program can ensure that it is getting credit for the work it is doing to help its students achieve academic and workplace success.

This NRS Tips summarizes strategies that two states—Maine and Rhode Island—have used to successfully increase their posttesting rates.

STRATEGY 1: KNOWLEDGE IS POWER

Local program staff may feel disconnected from WIOA accountability requirements. Staff focus on serving their students and might not understand how instruction relates to the data they are required to collect, possibly seeing data collection as a burden instead of something that can help improve the way they serve students. Teachers concentrate primarily on what they do and see in the classroom—the everyday teaching and interacting with students—rather than the testing that needs to be done. Showing teachers how small things they do in the classroom can impact the program’s data may motivate them to help increase the posttesting rate.

Maine conducted webinars to share information about how important it was to hit the target posttesting rate and how testing a certain number of students would help programs meet their MSG goals. In addition, the state’s Office of Adult Education and Family Literacy started having data discussions with adult educators to make sure that everyone was reporting, talking about, and collecting data the same way.

Rhode Island attributes its high posttesting rates to the state’s department of education informing the local programs of the exact negotiated federal targets for MSG. There is also a state policy to posttest at least 70% of students, and programs know they cannot reach the target unless they posttest at least that many students.

Data are everyone’s responsibility. By sharing information about state targets, local program staff gain a sense of that shared responsibility. They have a greater understanding that they are helping to contribute rather than being asked to meet an arbitrary goal.

STRATEGY 2: INCENTIVES ARE STRONG MOTIVATORS

Although local data contribute to a national picture of adult education, data can have an even greater effect on staff at the state and local levels. However, local staff may be unaware of their program’s funding sources and of how performance data are a factor in funding. Many states rely on performance-based funding, tying payment directly to outcomes. Some states have incentive awards based on meeting performance targets. Strong performance, as measured by data, means greater resources for instruction and for serving students, which can surely motivate staff.

A widespread example of this approach is the setting of performance standards that are tied to increased funding for success or reduced funding for shortcomings or failure. Some states set standards for measures, such as the percentage of students pretested and posttested and survey response rates, in addition to performance on NRS measures. Other methods of rewards include public recognition of the program as “high performing” and specific rewards to programs and staff members. With this approach, local staff become motivated to pay attention to program data reports that compare performance to targets.

Maine did not have Adult Education and Family Literacy Act money attached to goals under the Workforce Investment Act. Under WIOA, the state set aside a small percentage of the available funding to use as an incentive tied to three metrics: posttesting, MSG, and a program’s score on its continuation grant. Programs that met the target for two out of the three metrics would get an incentive payment, and if they met all three metrics, they would get a larger payment. For some programs, this system doubled their money. Over time, the total incentive funds available remained the same while more programs met the target requirements, so the amount a program could receive decreased. Instead of relying solely on incentive funding, the state provided recognition—another powerful motivator—through webinars highlighting the best programs.

Rhode Island implemented performance-based funding several years ago by tying a percentage of funding to EFL gains. The state started with 5% funding tied to performance the first year and then raised performance funding to 10%. Even though the percentage of funding tied to EFL gains was not high, it was effective. Just those small amounts had a large impact on the local programs and made them pay more attention to their Table 4 data and watch their performance more closely. Under the most recent request for proposals, the state used its federal MSG targets as the measure of program effectiveness. Program staff know that funding for the following year will be based on reaching their enrollment projections and reaching or exceeding the federal target for MSG again.

Programs in Rhode Island share scale scores from the posttests with students to show them how they have made progress, improved their skills, and made concrete academic gains. Some of the One-Stop training programs have grade-level-equivalent (e.g., eighth grade reading) requirements for entry, so students who want to enroll in those programs are especially interested in, and motivated by, how close they are to obtaining those scores. Knowing the importance of posttesting, students are more motivated to make sure they posttest.

One program creates incentives for students to attend and be posttested through the use of “lottery style” tickets. Students receive one ticket for regular attendance, two for perfect attendance, and three for posttesting. Then a winning ticket is drawn monthly for a gift card.

Maine and Rhode Island have used incentives successfully in motivating different stakeholders to focus on increasing posttesting rates. Although financial incentives are often the go-to, thinking creatively about what motivates each group of stakeholders and creating an innovative strategy can also lead to success.

STRATEGY 3: OVERCOMING EXCUSES

When something is hard to do, people often try to rationalize why they are unable to do it. These excuses might come from individuals’ fear of venturing outside their comfort zone or fear of failure. Such fears can be challenging if, as the state or local program director, it is your job to ask staff members to increase their focus on posttesting.

In Maine, local program staff had various excuses for why they were not able to reach their posttesting targets. Larger programs said that they were too big to reach those targets. Smaller programs said that they were too small. Other programs said that they did not have enough funding. The state debunked those excuses by finding similar programs (large, small, with limited funding) that were able to meet the targets. Identifying successful programs demonstrated that it was possible to run great programs in all those circumstances and that it was possible to reach the target and receive the incentive funds.

Rhode Island had significant buy-in from staff but had to overcome students’ excuses for not taking posttests. In many programs, the intake staff let students know from the beginning that they would be posttested and that the posttests were critically important for program funding. Staff stressed to students that they were receiving instructional services for free and they could help maintain free instruction for future students by taking posttests to help the program demonstrate success. Programs also began posttesting before students exited by testing soon after the students had sufficient hours for posttesting, rather than waiting until the last few days of the term. This change eliminated the potential challenge of students needing to return later for the posttesting.

Local programs and students can be the biggest barrier to their own success. By helping them recognize what can be possible and sharing achievements of similar programs, both local programs and students begin to understand that they too can achieve the desired outcomes.

STRATEGY 4: DON’T REST ON YOUR LAURELS

After seeing success, it can be tempting to reduce the level of emphasis being placed on posttesting. Success may lead to an attitude that the problem has been solved, and staff focus may change. This complacence may have happened in Maine. After a couple years of improvement, the state saw a decrease in posttesting between fiscal year (FY) 2017 and FY 2018, from 65% to 57% and a decline in MSG rate. This decline caused Maine state staff to recognize that just because there were gains, they could not rely on their past success to guarantee future success. Data collection and data quality require an ongoing, sustained level of effort. When staff turn their focus away from testing, it is easy to let testing fall through the cracks, and staff can slip back into old habits, even in an established culture of data use. State staff in Maine now view collecting posttest data as a work in progress. The state office published the posttesting percentages of the local programs to raise awareness again of the importance of this measure, noting the backslide. In FY 2019 the state will reinstitute a performance incentive.

Maine is also looking at the data to investigate other reasons that might have caused the decrease in posttesting. Staff have discovered that more people are entering postsecondary education and obtaining secondary credentials and achieving MSG in that way.

CONCLUSION

Improving posttesting rates is one way that adult education programs can improve their performance on MSG, but it can be challenging to do. States like Maine and Rhode Island have seen great benefits from helping their local programs understand what their targets are and why these targets are important. Having the right incentives has proven valuable as well.

Nevertheless, sometimes local programs and students continue to make excuses for why they cannot meet the targets and need support in understanding that success is possible. Once success has been achieved, program staff and students will need additional support to ensure that they maintain it.

The following table summarizes the issues, strategies, and actions the two states have taken to improve posttesting rates and, subsequently, their MSG. Achieving success in meeting posttesting targets is not a one-time endeavor. Accomplishing and maintaining success requires hard, ongoing work by both the state and local programs as well as continuous review of data.

SUMMARY TABLE