Skip to main content

Crowdsourcing Mistakes and Adaptive Learning

Matthew Graham

Course: Math 300: Introduction to Higher Mathematics

Download the PDF – Graham: Crowdsourcing Mistakes and Adaptive Learning


I have been running the Introduction to Proofs course at Northwestern as a flipped/inverted course, which means that students acquaint themselves with the material normally covered in lecture outside of class and then during class we work on more difficult problems sets. The main idea is to structure the course so that the instructor is present when the students are facing the more difficult challenges. I view this as a mathematical communications course: my students need to know how to transmit math verbally and in writing as well as be able to present math to a group. The flipped structure is great at helping the students transmit math verbally, since that is most of what class time is spent on. Unfortunately, I found that this increase in verbal ability didn’t usually transmit to an increased ability to correctly convey math via writing. Additionally, my students weren’t memorizing the definitions and theorems as well as they needed to in order to write correct mathematical statements.


There were two goals that were approached independently: first, devise a method to give students a massive amount of formative assessment on their writing; and second, devise a method that would help them memorize the needed definitions and theorems.

  1. To deliver and manage the formative assessments of their writing, I used Piazza, an online question-and-answer forum that allows wiki-type answers for both students and instructors. These wiki-type answers are accessible by everyone in the class. This allows groups to crowdsource solutions to problems or questions that they are working on. Additionally, I specifically used Piazza to crowdsource mistakes, since many times a lesson is learned more effectively from a mistake than a success. Specifically, I required each student to submit rough proofs of the problems from the worksheets they work on in class to Piazza. Then, each day I would read the rough proofs and list specific edits that needed to be made for the proof to be both mathematically correct and well written. Each student was responsible for submitting one rough proof and making four edits per week. Piazza has a viewable history of edits, which means that mistakes (and successes) are not only public but also preserved for later learning.
  2. In order to help them memorize and understand the definitions and theorems in the course, I designed a simple adaptive procedure for students to take online mini-quizzes each weekday. However, I was not able to implement the adaptive strategy via Canvas. Instead, I created multiple question banks and would randomly pull questions from specific question banks to generate the online mini-quizzes. The students were able to take the quiz as many times as they liked. However, each time they took it, the quiz would randomly draw different questions from the specified question banks. I structured the quizzes so that earlier material was reviewed often to help memorization occur.

Objectives & Outcomes

I have completed a pilot quarter using these methods and have yet to analyze the data statistically. Quantitatively, there were 324 problems worked (compared to around 140 in a non-flipped course), and there were 737 student responses and 353 instructor responses handled by Piazza. In eight weeks, there were a total of 2,029 Piazza contributions (these include edits to the same posts) from 18 students, one TA, and one instructor.

Subjectively, there was in increase in my students’ writing ability and clarity even though some of my expectations were not met. It is unclear whether the memorization component was effective.


Subjectively, the students were able to stop writing poor-quality proofs in a shorter amount of time than with previous methods. By the end, most students were able to write mathematically correct proofs but only about one third of the class could fully justify all the proofs that they wrote. I expect the massive amount of formative assessment regarding the students’ written work was the cause of the increased ability of students.

This quarter we have three sections of this course running, one of which will be treated as a control group in regards to the Piazza component.

I hope that these subjective gains will be seen quantitatively as well.

Lessons Learned

Even though there were gains in my students writing ability. I found that I was not emphasizing the correct learning objectives via Piazza and in-class. The main goal of any math course is to get students to give correct answers (and know that the answers they give are correct and why they are correct). Typically the structure of math courses do not give points for the items in parentheses above even though, in many respects, these are more important than getting a correct answer. I have restructured the manner in which I use class time this quarter in order to better give formative feedback on the items in parentheses. I plan on mirroring this structure in the Piazza component of my course next time I teach this course. Additionally, I created a system to organize the large number of Piazza posts.