The AI models addressed female students with more affection and used more first-person pronouns. (“I love your confidence in expressing your opinion!”) Students labeled as unmotivated were greeted with optimistic encouragement. In contrast, students described as high-performing or motivated were more likely to receive direct suggestions and criticism aimed at improving their work.
Different words for different students.
In other words, the AI feedback was different in tone and in the expectations it had for the student. The newspaper, “Marked Pedagogies: Examining Linguistic Biases in Personalized Automated Writing Feedback”, has not yet been published in a peer-reviewed journal, but was nominated for best paper in the 16th International Conference on Learning and Knowledge Analytics in Norway, where it is scheduled to be presented on April 30.
The researchers describe the feedback results as showing a “positive feedback bias” and a “feedback withholding bias,” offering more praise and less criticism to some groups of students. While differences in any written comment may be difficult to notice, patterns were evident in hundreds of essays.
Researchers believe the AI is changing its feedback on identical trials because the models are trained with large amounts of human language. Human teachers may also soften criticism when responding to students from certain backgrounds, sometimes because they don’t want to seem unfair or discouraging. “They are realizing the biases that humans exhibit,” said Mei Tan, lead author of the study and a doctoral student at the Stanford Graduate School of Education.
At first glance, differences in feedback may not seem harmful. More encouragement could boost a student’s confidence. Many educators maintain that culturally responsive teaching (recognizing students’ identities and experiences) can increase student engagement in school.
But there is a trade-off.
If some students are constantly protected from criticism while others are pressured to sharpen their arguments, the result can be an unequal opportunity to improve. Praise can motivate, but it doesn’t replace the kind of direct, specific feedback that helps students grow as writers. Tanya Baker, executive director of the nonprofit National Writing Project, recently heard a presentation of this study and said she was concerned that black and Hispanic students were not “pushed to learn” how to write better.
This raises a difficult question for schools as they adopt AI tools: When does helpful personalization cross the line into harmful stereotypes?
Of course, teachers are unlikely to explicitly tell AI systems a student’s race or background like the researchers did in this experiment. But that doesn’t solve the problem, the Stanford researchers said. Many educational databases and learning platforms already collect detailed information about students, from their previous achievements to their linguistic status. As AI is integrated into these systems, it can have access to much more context than a teacher would consciously provide. And even without explicit labels, AI can sometimes infer aspects of identity from the writing itself.
The biggest problem is that AI systems are not neutral tutors. Even the usual feedback response (when the researchers did not describe the student’s personal characteristics) takes a particular approach to teaching writing. Tan described it as quite daunting and focused on corrections. “Maybe one conclusion is that we shouldn’t leave pedagogy to the broad language model,” Tan said. “Humans should be in control.”
Tan recommends that teachers review written comments before forwarding them to students. But one of the strengths of AI feedback is that it is instantaneous. If the teacher needs to review it first, that slows it down and potentially undermines its effectiveness.
AI also offers potential for personalization. The risk is that, without careful attention, such personalization could lower the bar for some students and raise it for others.
This story about AI bias was produced by The Hechinger Reportan independent, nonprofit news organization covering education. Enroll in Test points and others Hechinger Newsletters.
