Unintended Consequences of Order of Grading

There is a fascinating article I recently encountered that addresses some implicit biases in university grading. Current learning management software platforms are a great assist to instructors. They allow a central electronic depository of all course materials. They are a platform for electronic interaction among class participants. They offer simple archiving of the submitted assignments. In addition, they offer a repository for instructors’ grades assigned to the participants.

One interesting feature of most learning management systems is support for the order of grading of assignments across students. The new article examined over 30 million records from about 400,000 different student course enrollments from the Canvas learning management system in a variety of disciplines at a large state research university.

The investigation examined whether there were systematically different grades assigned to the students early in the order of grading than to those graded later. The results were consistent. Those assignments graded early in the grading process tended to receive higher grades. The results were in the range of 3-4 points on a 100 point scale. However, the pattern was quite consistent. Those assignments graded toward the end of the grading tended to receive lower grades.

The order effects were corroborated with some qualitative data. The comments provided by instructors to the students were more negative and harsher for those toward the end of the grading. Correspondingly, there were more requests by students of grade changes among those graded toward the end of the grading.

The article posited a set of causal mechanisms for the findings which, unfortunately, could not be tested with data at hand. Instructors might become more fatigued over time and become less attentive. Alternatively, instructors might become more informed about the type of errors displayed by the students and thus more alert to them as the grading proceeded. Humans may become bored. There may be a tendency to implicitly regret being generous at the start of grading and become harsher as the sequence evolves. Truth may lie in some combination of these hypotheses.

Are there exceptions to the finding? It appears that assignments for which unambiguous right and wrong answers were extant are less subject to the order effect. Subjective judgments appear more subject to order effects.

The impressive part of the study, in my opinion, is how it sought counter evidence. One possibility for the result is that the last graded exercises are indeed less meritorious. The article examines three different orderings: alphabetical by last name, reverse alphabetical, and quasi-random. In general, the order effect exists in all three. Regardless of the order, assessments are lower toward the end of the grading.

What is the default ordering in most Canvas applications? Alphabetical by last name. (The instructor has to find a somewhat obscure switch to change this.) So what does it mean for students with last names toward the end of the alphabet?

We have a nonstudent example of this.

In most journal articles in economics, the order of authors is alphabetical by last name. In citations of articles by subsequent papers, it is common (as a compact form) to cite only the name of the first author (e.g., Smith et al.). An article in the early 2000’s studied a set of 35 US economics departments. It found that faculty with names late in the alphabet had lower likelihood of garnering tenure. They were less likely to become fellows of the Econometric Society, and somewhat less likely to receive the Nobel Prize. In short, systems of that highlight attention to those with initial letters of surnames early in the alphabet can affect human behaviors in multiple and potentially long-lasting ways.

What’s an instructor to do during grading? First, know this result. Don’t let the grading order be determined by the default alphabetization. Second, take breaks in grading; grade in smaller batches. Third, if possible, review the grading and feedback in a second pass of the assignments; check whether you see depressed assigned grades toward the end of your grading that doesn’t seem justified.