With the release of the revised ACCJC Standards in 2014, Standard I.B.6 has received a great deal of attention and prompted many discussions across the California Community College System, as well as an ASCCC resolution at the Spring 2015 Plenary (2.01 S15). This standard requires colleges to not only collect but also to disaggregate student learning outcomes (SLO) data, which is the practice of collecting an individual student’s SLO data and linking his or her scores to student’s demographic data, especially gender, ethnicity, and other metrics related to student equity and disproportionate impact. Colleges are required to then analyze SLO data for disproportionate impact among subpopulations and make program changes according to the results.
With this change, the idea of a genie in a bottle fits fairly well when discussing disaggregated data and student learning outcomes. The most famous version of the Persian folktale of Aladdin and the genie in the lamp is told in the One Thousand and One Nights in this way: After Aladdin discovers the lamp and releases the genie, the genie helps Aladdin to become wealthy and powerful, and even helps him to to marry the emperor's daughter Princess Badroulbadour, who was betrothed to another, and to build a grand palace. Other stories tell of genies, or the Jinn, whose intentions when released from the bottle are not benevolent but are very nefarious in the same vein as the “trickster” character in western literature. Even in the One Thousand and One Nights tale, a sorcerer tricks Aladdin’s wife and steals the lamp only to command the genie to take away all the riches Aladdin has gained. Like in the tales, SLOs and disaggregation are fickle genies, and this duplicity raises the question of whether SLO data disaggregation will be a good genie, a bad one, or something in between.
The Good Genie
A 2012 brief by the National Center for Mental Health Promotion and Youth Violence Prevention, an organization that provides technical assistance and training to 106 federally funded Safe Schools/Healthy Students in K-12, argues in favor of disaggregation. The brief points out that aggregate data masks inequities in success rates among subpopulations, leaving those struggling subpopulations unrecognized and on their own in terms of improving success rates. The brief also argues that disaggregation informs and provides data support for changes in how programs are implemented in order to support all students. These changes can take the form of specific policy changes, funding augmentations, and more surgically precise program improvements that take into account the diversity in the classroom.
Student Equity Planning through the Student Success and Support Act at its core relies on disaggregated data for planning improvements in student achievement for subpopulations. Taking that philosophy to the course-level and program-level learning outcome assessment is an extension of that effort, at the federal and state levels, to increase access, course completion, ESL and basic skills completion, degrees, certificates, and transfer for all colleges. Title 5 regulations require colleges to review and address disproportionate impact for Indians or Alaskan natives, Asians or Pacific Islanders, Blacks, Hispanics, Whites, men, women, and persons with disabilities (§54220(d)) and to develop specific goals or outcomes and actions to address inequities. Action plans for improvement then evolve through the program review process. Disaggregation advocates say meaningful conversation about disproportionate impact cannot happen without disaggregation of course-level learning outcomes.
The Bad Genie
Later in the story of Aladdin, an evil sorcerer tricks Aladdin’s wife and takes the lamp. He uses the genie to take away from Aladdin the riches he attained with the genie’s help. Similarly, we might ask whether SLO disaggregation, like the Jinn from Persian lore, also has a bad side or whether this particular genie can be used for mischief and mayhem in the wrong hands.
The concerns over the disaggregation genie are wide-ranging. Student privacy concerns are real and require very precise data reporting practices that must be collegially agreed upon by faculty, administrations, and researchers at each college and in keeping with FERPA regulations. When data are disaggregated for courses that only offer one section or are rarely offered at all, publicizing results with demographic information may allow students to be identifiable, especially for underrepresented minority students. Also, low sample sizes call into question the validity of the data collected in the first place. If only 20 Asian-American students are included in learning outcomes assessments out of 250 students total across two or three sections of a capstone course, that data may not really tell you anything significant about Asian students. Even if the data are longitudinal over several years, small sample sizes may not provide useful information.
We have to also remember that SLO assessment frequently raises controversy in any context. Some faculty bargaining units, which may already be resistant to SLO assessment, will certainly ask relevant questions about additional workload associated with this type of data entry that may reinforce the opinion of local unions that ACCJC is imposing standards without deference to bargaining agreements. Local senates should approach the way they respond to this standard with their bargaining unit partners as part of the conversation, in the same way they would be involved in any discussion related to district policy or practice intended to address accreditation standards.
The Genie Is Out and He’s Not Going Back In
SLO assessment is here to stay, and the ASCCC has made statements regarding compliance with SLOs in the last decade. For better or worse, this genie is not going away.
In order to use the genie for good while acknowledging the arguments for and against, colleges should begin disaggregation data conversations slowly and in measured steps:
- Pick one course in a program, maybe the course with the most sections, and ask faculty in those sections to collect and input disaggregated data into their database systems.
- Review less controversial data attributes in reporting. For example, look at sections taught in the evening versus sections taught during the day, sections taught online versus sections taught on ground, or sections taught at a central campus versus at an education center or remote site. Such a beginning may be a way to get start the process while keeping in mind the requirement in the ACCJC Standards that data on subpopulations must be disaggregated by the time of your college’s next self-evaluation report to be in compliance, beginning Spring 2016.
- Look to Student Equity funding. If issues of workload are impeding the conversation over disaggregation, look to Student Equity funding as potential seed money to build an infrastructure where disaggregation is not a hardship or burden for faculty.
So, How Does the Story End?
One cannot predict at this time how this story will end because it is just beginning. As more colleges begin adopting and revising processes in order to comply with the new standards in Spring 2016, questions over SLOs in general and disaggregation specifically will begin making their way to meeting rooms across the state. Community colleges throughout California must begin discussions of how they will address the SLO disaggregation requirement and consider the various implications of this practice regarding workload, student privacy, data relevance, and other issues in order to ensure that the ACCJC’s requirement turns into a good genie that can grant positive results for colleges and students.
National Center Brief: The Importance of Disaggregating Student Data.
National Center for Mental Health Promotion and Youth Violence Prevention, Safe Schools; Healthy Students. April 2012. Web. 10 Aug 2015.