Online Solutions for Prevention Education

2/20/2002

NetResults

By Brandon Busteed, Founder of Outside The Classroom, Inc.

Student affairs, like most professions, continues to be profoundly affected by new technologies. As we learn new technological skills and embrace new services that help us carry out current tasks more effectively, we also look to technology for new solutions.

Outside The Classroom, which has partnered with NASPA over the last year, is pioneering a new use of technology in the risky-behavior prevention field with the development of AlcoholEdu, an online course which educates college students about alcohol.

AlcoholEdu is a three-hour online course, delivered through a web browser on the Internet. Students use a personal access code to log on, and then take the course individually, at their own pace. AlcoholEdu integrates into the educational program sophisticated built-in assessment surveys and tests. These evaluation capabilities have allowed the compilation of preliminary data that is invaluable to both Outside The Classroom as we improve the course and to student affairs professionals as they determine the effectiveness of and potential uses for prevention tools on their campuses.

AlcoholEdu Preliminary Findings

Evaluation results from the first cohort of students participating in AlcoholEdu provide strong evidence of the course's power. Exams measuring gains in knowledge from the course indicated significant increases in knowledge about alcohol. In addition, students showed significant changes in attitudes and behavior around alcohol use (all differences referred to in this article were statistically significant at the p<.05 level).

The students upon whom these findings are based were from 24 different colleges and universities. About half were first-year students and 59% were male. Most (87%) took the course as part of a disciplinary requirement.

Reception

Student feedback consistently has demonstrated enthusiasm about the course. Those disciplinary referrals who were unenthusiastic participants at first (understandable from a group taking the course because of disciplinary troubles) changed dramatically, dropping by 50%. One participant wrote the following in his evaluation of the course:

I learned a lot and it also made me think about how I should present myself when I am out at night. I think that this course is really helpful. I was happy with the fact that I could take this class in the privacy of my own room. This allowed me to work at my own pace and not be in the spotlight.

The majority of students found the course interesting and the information clearly presented. More importantly, most felt that AlcoholEdu was helpful.

Knowledge

Not only did the course participants report feeling that they knew more about alcohol after the course than before, their pre- and post-test exam scores confirmed the knowledge gain. Exams taken before and after the course showed a significant increase in scores after completing AlcoholEdu. The exams were based upon questions collaboratively created by experts in college alcohol issues and The Chauncey Group, a subsidiary of the Educational Testing Service (ETS).

 

Attitudes and Thoughts

Students completing AlcoholEdu demonstrated significant attitudinal changes concerning the norms of alcohol use by college students. Using a “social norms” assessment question first popularized by Allen Berkowitz and Wesley Perkins in 1996, the evaluation showed a movement away from the perception that most other students at their institutions possess lenient attitudes about alcohol use. Significantly fewer AlcoholEdu participants believed that the predominant view on campus was one that claimed “frequently getting drunk is okay if that’s what the individual wants to do.” In fact, this was a substantial drop by almost a third.

Those students completing the follow-up evaluation one to two months after completing the course continued to demonstrate this attitudinal change. Evaluation results showed a consistent drop in normative perception. In addition, they reported thinking about their own consumption, their drinking choices, and their related behaviors -- including BAC levels and learning-memory issues related to alcohol. Students also were more likely to actively plan how much they were going to drink after having taken the course.

Behavior

Seasoned educators know that it is one thing to talk about or intend to change one’s behavior, and another thing to actually do so.

 

• Abstainers in the group doubled after the course. That is, the number of students who had not had any alcohol to drink in the two weeks previous to the assessment effectively doubled between the time of the pre-test to the follow-up after the course (from 17% to 32%).

Those who were still drinking after AlcoholEdu reduced the average number of drinks they consumed when drinking, from 5.2 drinks per occasion to 4.6.

Evaluation Design

The evaluation design that allows such data gathering as what is described above includes questions measuring knowledge, attitude, and behavior. The goals are to collect both formative (information to improve the program) and summative (information about the effects of the program) feedback. Formative questions asked of students include how enthusiastic they were about taking the course before they actually took it, whether or not they experienced any technical difficulties, and what their favorite sections were. The objective of the summative portion is to measure increases in knowledge and protective factors and decreases in negative consequences and levels of consumption.

The AlcoholEdu evaluation is broken into two basic components: exams and assessments. The survey design includes an immediate pre-test, post-test, and one-month follow-up. The pre-test includes a 20-question exam created in conjunction with The Chauncey Group, a division of the Educational Testing Service (ETS). This pre-exam was developed in two equally calibrated exam forms, and is matched with two 40-question exam forms taken at the completion of the course. To prevent cheating on the exams, multiple forms were developed and a feature was built into the exams to randomize the sequence of the questions within an exam form. Therefore, no student takes the same exam, while all the data collected is scientifically valid and calibrated across different exam forms. This component enables an institution using the course to determine students’ knowledge before taking the course so that a true assessment of knowledge and behavior change can be made after the course is completed.

Within the evaluation that students complete before taking the course, they are asked to assess their knowledge and attitude about alcohol. They are also asked about behaviors, including levels of alcohol consumption in the previous two weeks. This particular question requires the student to write in a specific number of drinks consumed on each day in the previous two weeks, allowing school administrators not only to calculate a “binge” drinking rate for weekends, but also for specific weekdays. This capability, for instance, can help pinpoint high consumption days on a particular campus. Academic consequences related to alcohol are measured, including such things as whether students have attended class drunk, missed a class or a deadline, or been unfocused in class. Health consequences of alcohol use are also measured, including whether students have had a hangover, had to be prompted to remember something they did, gotten into a physical fight, vomited in public, or failed to use safe sex practices when they ordinarily would have.

When considering online evaluation tools such as those built into AlcoholEdu, it is useful to take a few steps back and examine some issues and misconceptions related to online surveying. First, student familiarity with computers is at an all-time high. The 2000 National First-year Survey Data found that 78% of high school seniors use a computer frequently while another 67% had used the Internet for academic work.

It is also important to note that studies researching differences between web-based data gathering and paper-based data gathering revealed no significant differences in results. In fact, ongoing studies will likely reveal that web-based data gathering is even more accurate than paper-based. For instance, when collecting demographic information about racial/ethnic identification, web-based surveys provide greater flexibility in allowing students to select a category or write in their own instead of simply having to choose “other.”

Although there may be greater up-front costs associated with developing web-based surveys, there is a significant savings in the long run. Paper-survey costs such as printing, paper, Scantron machines, and the staff time spent in deploying, collecting, and analyzing the data can be exorbitant. And the most important thing is that web surveys provide immediate results. When trying to cope with critical student health issues, it is an advantage to have immediate and accurate data. Paper-based surveys can take months to complete, while web surveys can produce the same results in days.

Future Directions

Some powerful effects of AlcoholEdu have been demonstrated in the preliminary evaluation results. Attitudinal change, the focus of many current “social norms” campaigns, was remarkably positive. Alcohol consumption, measured in multiple ways, showed significant decreases over time. AlcoholEdu’s approach to alcohol education even overcame resistance and won over some of the toughest critics of alcohol programs: the disciplinary referrals.

The evaluation pointed out some discrepancies in the versions of the ETS exams, and after careful analysis, these exams have been recalibrated using the additional data. The evaluation forms (pre-test, post-test, and follow-up) are being constantly revised to provide new information on the course and its effects. One area of expansion will be in examining course effects on the negative consequences of alcohol use. While there were some significant reductions in negative consequences during this pilot testing, the section as a whole proved to require some additional redesigns before all the data could be meaningfully interpreted.

A crucial consideration is that this initial evaluation of AlcoholEdu will benefit from comparison data on students who did not take the course. A control group will allow the effects of the course to be pinpointed more precisely. In keeping with the spirit of ongoing improvement and accountability, Outside The Classroom is currently pursuing a control group strategy.