Think-Alouds

A strategy to assess student’s knowledge and the reasoning behind their responses⎮2 min 30 sec read

Like Comment

One of the big issues in K-12 education (and maybe higher ed. too?) during the pandemic is assessment. When we get to see some students face to face, and other students online, how can we assess their knowledge and skills?

Pre-pandemic, many teachers used “forced choice” assessment items (multiple choice, etc.) to assess students, and those formats are tough to use when students are learning from home. We can try all the fancy technology we have (e.g. lockdown browsers, Hapara, countdown clocks, etc.) but no technology is likely to be able to compete with a student, their phone, and fast thumbs. 

I worked with some teachers in my district on another option called think-alouds and it’s a technique that might work for many teachers across different contexts.

The general idea: 

  • Instead of asking students to just choose the best answer to a multiple-choice item (which a student might just Google), ask them to choose the best answer AND write out their REASONING behind that answer.
  • Require students to “think aloud” - if students explain WHY they think the right answer is the right answer, we can at least use that multiple choice item and the student’s writing, to measure if they are thinking about the question in the ways we want them to (even if they did just Google the answer). 

This isn’t a new idea: people who develop tests often use the “think-aloud” technique to confirm that multiple choice items are measuring what they are supposed to measure. Some great folks from the Stanford History Education group used this technique to gather evidence that the items on the NAEP test that are supposed to measure historical thinking probably don’t.

I have previously used this technique in my psychology class as a “test correction” technique, so here’s a possible “workflow” for think-alouds that might benefit a teacher who is working with students remotely (students who “zoom in” to class instead of being there face to face):

  1. The teacher looks at all the multiple-choice items on their “traditional” exam (the one they usually prepare for students).
  2. The teacher keeps all the “essential” multiple choice items and gets rid of some of the less important ones (rationale: using the think-aloud technique will take students more time, so you may need to reduce the total number of items).
  3. The teacher re-formats the exam to include the multiple-choice item AND space for the student to write out their thinking (e.g. a box in Google docs / Microsoft Word for each item, etc.) OR the teacher gives students instructions about how to add the think-aloud to the existing copy of the exam (e.g. teaching students how to add text to a pdf using Kami, etc. See the example below). 
  4. The teacher sends each student a copy of the test with a deadline. Students choose the best answer to each item and write out WHY they think this answer is the best answer. The test directions explain this process to the students, including the message to students that the think-aloud has to be THEIR thinking, not copied from anyone or anything else (that will be obvious to the teacher if students collaborate b/c their reasoning/wording will be very similar). 
  5. Students complete the think-aloud assessment, the teacher scores the tests, and sends feedback to students (the teacher either grades the exams or provides feedback to students, which they can use to change their responses before grading). 

Here’s an example - this teacher (thanks Jim D.!) taught his students to use Kami to do think-alouds on a pdf document. 

Limitations: 

  • Students can still “cheat” - they can still look answers up on the web. But with the think aloud technique, at least teachers can see if students are thinking through those answers correctly. 
  • In assessment terms, the think aloud technique is probably a “modification”: it measures something different from a traditional multiple choice, face to face assessment. Teachers may want to think about whether they use data from the think-aloud assessment in exactly the same way as they did traditional tests. 
  • This is more work for students and teachers. Students need to do more writing and teachers need to read through student reasoning. Teachers should think critically about the number of multiple-choice items they use with the think-aloud technique. 

I recently talked with a high school social studies teacher friend about this strategy (Thanks Jeff B.!) and he encouraged me to add more specific examples. Here’s my attempt:

  1. If your “normal” test is about 30 multiple-choice items, you might spend up front time going through those items and narrow it down to about 10 “most important” items. These are items that you think are the “best” ones: they ask about the most important information, and they do it in a way that requires students to think rather than just recall a term or a fact. That probably means you’re choosing “application” or “evaluation” items, not “recall” or “definition” items.
  2. You might make two forms of the test: the in-person students get the “old” form – the one you usually use (30 multiple-choice items). The zoom-in students get the new form of the test: the 10 “best” items with space for them to do the think-aloud technique, and directions explaining what they need to do.
  3. You get all tests back from all the students. The old form with 30 multiple choice questions is graded the usual way. But the new think-aloud form requires a teacher to look through what students wrote, however, it’s only 10 items, and you only need to look at the think-aloud part. It will be obvious from what the students wrote, whether they should get credit for the answer or not.
  4. You figure out how to make sure there are equal numbers of points for both forms (this means that each of the think-aloud items needs to be worth more than each of the “old” multiple choice items the students answered. In my example, each of the 10 think-aloud items could be worth 3 points, and each of the 30 old multiple-choice items could be worth 1 point).

Note about grading: I think grading the 30 “old” multiple-choice items by hand might take about as long as looking through the responses to the 10 new think-aloud items. But I could be wrong! (especially if you had a way to automate scoring the old multiple-choice items).

The think-aloud technique is far from a perfect solution, but I don’t think we have many perfect solutions during a pandemic. If you use this idea, please let me know, so that I can update this blog post!

Thank you in advance! 

References:

Ercikan, K., Arim, R., Law, D., Domene, J., Gagnon, F. and Lacroix, S. (2010). Application of Think Aloud Protocols for Examining and Confirming Sources of Differential Item Functioning Identified by Expert Reviews. Educational Measurement: Issues and Practice, 29: 24-35. doi:10.1111/j.1745-3992.2010.00173.x

Smith, M. D. (2017). Cognitive Validity: Can Multiple-Choice Items Tap Historical Thinking Processes? American Educational Research Journal, 54(6), 1256–1287. https://doi.org/10.3102/0002831217717949

 

 

 

Rob McEntarffer

Assessment/Evaluation Specialist, Lincoln Public Schools

No comments yet.