Web Writing: Why and How for Liberal Arts Teaching and Learning (archived site)

How We Learned to Drop the Quiz: Writing in Online Asynchronous Courses

¶ 1 Leave a comment on paragraph 1 0 Directions: Select the correct answer for the following question.
1. Why are multiple-choice quizzes often used to assess knowledge in online courses?

• ¶ 2 Leave a comment on paragraph 2 0
• A. Instructors love to write them.
• B. Taking such tests is an enriching experience for the student.
• C. Multiple-choice quizzes are the best way to assess nuanced understanding of complex topics.
• D. None of the above.

¶ 3 Leave a comment on paragraph 3 6 Pencils down.

¶ 4 Leave a comment on paragraph 4 2 What follows is the story of how we decided to embrace the implications of the above question for Hidden in Plain Sight and Virginia Studies, two online asynchronous history courses designed for practicing K-12 teachers. ¹  The result: dropping the multiple-choice quiz to place greater emphasis on historical thinking through iterative writing. The difficulty of writing good multiple-choice questions and the frustration involved needs little documentation, but participant responses also played a central role in the decision.

¶ 5 Leave a comment on paragraph 5 2 Several years ago, the Roy Rosenzweig Center for History and New Media at George Mason University, with funding from the Virginia Department of Education, designed Hidden in Plain Sight and Virginia Studies to provide quality online professional development focused specifically on history education. We started by asking ourselves a series of questions: What did we want teachers who completed the courses to learn? How could we model historical thinking in an online environment? How could we make the courses engaging? How could we incorporate writing into the process? How could we frame the feedback on writing in a way that encouraged teachers to engage deeply with primary sources and historical analysis, as well as with strategies for teaching both in the classroom? And perhaps most important, how could we measure progress and assess growth in the ability to think critically about the past and to teach historical thinking in the classroom?

¶ 6 Leave a comment on paragraph 6 0 Click to view the Hidden in Plain Sight home page in a new tab/window.

¶ 7 Leave a comment on paragraph 7 3 We structured the courses to address these questions directly. Each course begins with an introduction to historical thinking followed by a series of modules. Each module begins with an object, such as an 18th-century homespun coat, a handmade nail, or a can of instant coffee, presented without label or context. Participants are asked to form a hypothesis based on close observation—taking note of the details, form, material, and nature of the object—before drawing conclusions. In addition, teachers are prompted to think about the object in historical context, drawing on their own knowledge to hypothesize how the object fits within Virginia history or the American past. Participants submit their hypotheses and proceed to Resources to learn more about the object and explore its historical context. Additional materials in this section include maps, prints, posters, handbills, personal letters, songs, and diary entries, with accompanying text or video that draws connections between the object and the larger historical context. Videos of historians analyzing sources and modeling how to actively interpret history are also included.

¶ 8 Leave a comment on paragraph 8 0 Click to view the Virginia Studies home page in a new tab/window.

¶ 9 Leave a comment on paragraph 9 0 After exploring Resources, participants read a concluding essay and, in the initial design, completed a multiple-choice quiz to assess comprehension. After the quiz, course participants were required to rethink their initial hypothesis, reflecting on what they learned and how their thinking developed. Their original hypothesis appeared on the screen next to a space for writing a Rethink, or revised hypothesis. Participating teachers then applied their newly acquired skills and content in Classroom Connection where they developed an activity for their own classrooms drawing on the resources of the module and the lessons learned. Participants could then view and comment on the submissions of others in Wrap Up.

¶ 10 Leave a comment on paragraph 10 4 We initially included the multiple-choice quiz for several reasons: to assess basic learning, to ensure that course participants read through all of the materials in the module, and to provide immediate feedback at a specific point in the process. We created it with some ambivalence, a feeling that deepened during the development and pilot testing phases as technical, functional, and pedagogical questions arose. Should teachers, for example, be allowed to take the quiz more than once? Should they be allowed to return to Resources while taking the quiz? Doing so would allow for deeper learning and could encourage course users to spend more time with the sources. At the same time, it might lead some to skip to the quiz and then look through Resources for specific answers to quiz questions, allowing the quiz to define the learning experience. Another question loomed even larger: did the quiz questions support the overall objectives of the course? Did they facilitate historical analysis and critical thinking? Did they allow the course instructors to see growth in thinking?

¶ 11 Leave a comment on paragraph 11 1 Another drawback of including a quiz emerged while teaching the course. Course participants put far more weight on the quiz than we did as instructors. Despite extensive, personalized feedback on written responses, we received regular emails primarily expressing concern about quiz scores. This was especially notable because most participants took the course for teacher recertification credit—not a traditional graded class. One teacher, whose written responses were exemplary, emailed the instructors to reassure us that, “I read the stuff, really,” and ask “[will] quiz scores affect our recertification?” Another felt “embarrassed” by less than perfect quiz results.

¶ 12 Leave a comment on paragraph 12 6 This concern was by no means unique, as participants communicated similar worries each time the courses were offered. Some asked to retake the quiz. In response to the number and tenor of the requests for retakes, we reformatted the course settings to allow participants to take the quizzes multiple times, with some taking the quiz repeatedly until they achieved a perfect score. Clearly the quiz loomed large in participants’ minds. Given that the goal of these courses was not the retention of historical facts, but the development of historical thinking skills and their application in the classroom, the attention and mental energy that course participants devoted to the multiple choice quizzes was concerning.

¶ 13 Leave a comment on paragraph 13 1 In contrast, the written assignments (Hypothesis, Rethink, and Classroom Connection) proved very successful at engaging participants in historical thinking. Individuals taking the course demonstrated growth, often significant growth, in their ability to engage with primary sources, analyze historical contexts, develop their own interpretations, and integrate strategies for teaching students to do the same. They built on their experiences from the first module, improving their ability to hypothesize about the context and meaning of the initial object. Perhaps equally important, the written submissions provided discrete evidence of this growth in thinking, which did not necessarily correlate with the quiz results.

¶ 14 Leave a comment on paragraph 14 1 Hypothesis
The first writing activity in each module involves forming a hypothesis on the historical significance of a piece of material culture—an important first step in the iterative writing process that is central to these courses. A dishwasher, an 18th-century porcelain cup and saucer, and a boot issued to a Union soldier during the Civil War are a few of the objects used. Participants are prompted to frame their hypotheses in response to two questions: “What do you notice about this object?” and “How might this object connect to broader themes in American (or Virginia) history?” Knowing that some may hesitate to give detailed responses to this exercise, afraid of not saying the “right thing,” Hidden in Plain Sight and Virginia Studies course instructors frequently encourage participants to take risks, to use the hypothesis for observing and brainstorming. As one instructor wrote, the hypothesis is “a space to speculate and include your thoughts on the objects and their connection to broader histories—and also where there’s no such thing as a wrong answer.”

¶ 15 Leave a comment on paragraph 15 0 Sample Hypothesis object: Field Ration, U.S. Army K Ration Tin from World War II. Source: Oshkosh Public Museum.

¶ 16 Leave a comment on paragraph 16 2 In response to an image of a World War II-era can of Nescafé, for example, one participant noted its age and appearance and then hypothesized that the can was used by American soldiers during the war. To answer the question about the object’s broader historical significance, the participant thought that the can was related to “either the challenges of mobilizing an army and the growing military technology, or just the industrialization of agriculture and modern society.” After submitting the hypothesis, course participants move forward to view the modules’ resources, which feature additional primary sources and historians’ commentaries to explore the ways in which the original object connects to broader historical themes. After viewing these resources, course users revisit their original hypothesis in a written assignment entitled Rethink.

¶ 17 Leave a comment on paragraph 17 2 Rethink
Rethinks are structured writings prompted by two primary questions. The first question asks course participants how the module’s object and resources have influenced their thinking on a theme or event. In the porcelain module in Hidden in Plain Sight (pictured below), for example, the first question is “How does porcelain connect to broader themes in 18th-century history? Use specific examples.” The second question prompts course participants to interpret the materials presented in the module: What is missing? What is emphasized (or not)? How does the module connect to their existing knowledge of the subject? In the case of the porcelain cup, the second question asks “What additional information (not included in this module) would you want to know before making an argument about the causes of the American Revolution?”

¶ 18 Leave a comment on paragraph 18 0 Sample Rethink prompt.

¶ 19 Leave a comment on paragraph 19 1 A quality Rethink connects the Resources to the relevant themes and historical context, and engages meaningfully with the interpretations presented. One teacher, who took Virginia Studies, reflected on how the fence module enriched her perspective on the relationship between geography and history:

¶ 20 Leave a comment on paragraph 20 0 The fence symbolizes ownership and settlement. The shape and materials show how the settlers tried to control their environment. They used natural resources, wood, to build the fence and by building it they were taking ownership of the land, a concept that was different from the native peoples. The fence is an important part of Virginia’s history. It shows us that when the settlers arrived they were planning on staying here. Building fences began the process of transforming the landscape of Virginia.

¶ 21 Leave a comment on paragraph 21 0 I thought it was interesting that all the modules were pieces of the Virginia Studies curriculum which most teachers teach from a viewpoint of history. Yet, each module used geography as the starting point. . . Virginia’s history might be better understood and engaging to students if you use the themes of geography—location, environment, landscape, human-environment as a starting point for instruction.

¶ 22 Leave a comment on paragraph 22 1 The Rethink encourages thoughtful engagement with the material and with historical thinking through a structured writing exercise rather than a rote recounting of historical facts. In feedback, Hidden in Plain Sight and Virginia Studies instructors help course users expand on their responses, often by posing questions to prompt more critical analysis and prompting participants to include more detail in their reflections. In response to a thoughtful Rethink on the porcelain module that discussed connections between the teacup and 18th-century themes of representation, revolution, and mercantilism, the instructor encouraged the teacher to explore the consumer and political culture angle, including colonist decisions to purchase (or not purchase) imported goods as an expression of political beliefs and the decisions of American merchants who simultaneously faced public pressure to refuse imports and the potential loss of revenue from boycotts.

¶ 23 Leave a comment on paragraph 23 2 The Rethink provide the first formal chance for course participants to revisit their initial thoughts on a historical object, and to synthesize and interpret the arguments and information learned in Resources. The Hypothesis-Rethink-Feedback arc of each module provides an opportunity for iterative learning, for course users to rehash and strengthen their written interpretations over the course of the module. We see this combination of writing assignments as central to encouraging historical thinking in an asynchronous learning environment: since teachers taking the course have the chance to revisit, amend, and expand on their hypotheses based on additional information and evidence, much of the pressure to write the one “right” answer is lifted. In addition, these educators are engaging in the same exercises that they often pose for their students—and the same processes that historians use regularly.

¶ 24 Leave a comment on paragraph 24 4 Classroom Connection
After submitting their Rethink, course users move to Classroom Connection and are asked to apply what they have learned in the module by creating an activity for students. Unlike the more structured Rethink, the Classroom Connection is an unstructured writing activity. The emphasis is on developing an inventive lesson. Similar to the previous writing assignment, course instructors provide feedback, however, in the Classroom Connection participating teachers also receive feedback from each other in the form of comments on their submitted lesson ideas.

¶ 25 Leave a comment on paragraph 25 1 In Classroom Connection, participants craft a classroom activity based on the module’s content and historical thinking through primary sources. In a module on the Triangle Shirtwaist Fire, for example, one teacher asked students to investigate workplace safety in local businesses. Students compared federal (U.S. Occupational Safety and Health Administration, OSHA) and local regulations with photographs and documents from the 19th-century, before such laws were created. By studying workplace safety and regulation over time, students could examine the Triangle Shirtwaist Fire in its historical context. The teacher posed thoughtful questions for students, including the benefits and drawbacks of extensive safety regulations for businesses and for workers and what led to specific regulations and policies.

¶ 26 Leave a comment on paragraph 26 1 Participating teachers are often highly motivated to incorporate primary sources into their lessons, but sometimes struggle to connect primary source analysis with the larger historical narrative. The quizzes did not effectively address this gap, whereas the written activities in each module raised these issues to the surface and provided the opportunity for thoughtful feedback.

¶ 27 Leave a comment on paragraph 27 2 In Virginia Studies, the module on the colonial period, for example, features a source that many participants incorporate into classroom connections: a letter written by an indentured servant named Richard Frethorne describing the harsh conditions of servitude and labor. ² One teacher described a lesson in which students would read this moving letter and then write their own letter from the point of view of an indentured servant. Through feedback on Classroom Connection submissions, the instructor encouraged the teacher to incorporate other sources and direct the lesson to a larger historical question, suggesting that the teacher use advertisements designed to lure individuals to the new world. Promises by the Virginia Company could be contrasted with stories of hardship in Virginia, asking students to consider what the colonies might represent to prospective colonists and why they might choose to come despite rumors of hardship from individuals such as Frethorne. This teacher’s writing offered an opportunity to provide specific feedback on how historical evidence can help students understand the choices made by those in the past (e.g., why individuals would choose to become indentured servants in 17th-century Virginia) and address larger historical questions, such as how the Virginia colony grew in population despite a high mortality rate from disease.

¶ 28 Leave a comment on paragraph 28 1 Conclusion
The team teaching these courses met regularly during pilot testing and the first iterations of each course. Each time, we improved functionality, revised content based on feedback, and adjusted the structure slightly. We compared evaluations, student work, and instructor experiences and discussed strategies for improving the courses for future iterations. The decision to eliminate the quiz evolved over the course of these conversations and reflections.

¶ 29 Leave a comment on paragraph 29 1 The act of writing, and perhaps most importantly the iterative cycle of Hypothesis-Rethink-Feedback, both modeled and encouraged participants to go beyond descriptive, narrative history and to think of new ways to engage students in learning about the past. Through writing, receiving feedback, and revising their initial hypotheses, participants actively made insightful connections between sources and history, showed growth in their understanding of historical thinking, and applied new knowledge to their classrooms in creative ways. Success on the quiz did not demonstrate this growth, and often did not correlate with a significant development of historical thinking skills across modules.

¶ 30 Leave a comment on paragraph 30 6 Removing the quiz from Hidden in Plain Sight and Virginia Studies, however, eliminates the benefits of automatic grading and of instant feedback. This is a somewhat daunting prospect for an online, asynchronous course with open enrollment. At the same time, through trial and error, we learned that omitting the quiz allows us to keep the focus of the courses on the original goals—assessing historical thinking—and to focus all of the instructional time on the structured and unstructured writing of participating teachers. Feedback is comprehensive, ongoing, and focused on each individual’s work and progress. This regular engagement with instructors has proven to be a key feature of the course’s success as indicated by course evaluations.

¶ 31 Leave a comment on paragraph 31 1 These course evaluations have been overwhelmingly positive. Ninety-eight percent of participants either agree or strongly agree that they learned history content, teaching strategies, and tools for analyzing and teaching with primary sources while taking the course. One hundred percent reported that course structure stimulated their thinking and that they would recommend the course to a colleague. Individuals who completed Hidden in Plain Sight and Virginia Studies described the online courses as “thought provoking,” “very user friendly and engaging,” “meaningful,” and “eye opening.” One teacher noted, “This course really helps you understand the STORY behind the standard.” Formal and informal interactions with course participants throughout each semester via assignment submissions, comments, and emails confirmed the value of writing in the online course. On the same post-course surveys, the feedback on the quiz was far more equivocal, and in fact it often stood out as the only feature of the course that did not receive uniformly positive reviews.

¶ 32 Leave a comment on paragraph 32 2 Going forward, we are confident that the singular emphasis on writing will allow us to assess not only whether participants are retaining content (the original point of the quiz), but also whether they are engaging with that content in a meaningful way. It is our hope, too, that this focus on deeper understanding and critical learning can be applied to other online, asynchronous courses in place of the emphasis on rote learning and discrete facts that plagues much of open-enrollment, online education.

¶ 33 Leave a comment on paragraph 33 1 From our perspective, the online asynchronous model for courses has much to recommend it, particularly for strengthening writing in the liberal arts. Courses like Hidden in Plain Sight and Virginia Studies are available to individuals whose location or work-life schedules make attending a traditional physical course impossible. In addition, these courses have the potential to reach large numbers of students, and demonstrate that writing assignments, both structured and unstructured, can be effectively incorporated into an online learning environment. In our experience, two instructors can administer the course to about 100 students at a time. Currently these courses are offered on a semester model, but they could also potentially allow for continuous enrollment, meaning that an extra semester’s worth of participants could complete the course within a year.

¶ 34 Leave a comment on paragraph 34 1 It is this potential for online learning—the opportunity to provide meaningful learning experiences while reaching a wide audience—that provides the core focus for reflecting on and modifying the course. While online learning never has been and never will be a panacea, in this context, Hidden in Plain Sight and Virginia Studies offer exciting possibilities for liberal arts education beyond selecting between bubbles labeled A, B, C, or D.

¶ 35 Leave a comment on paragraph 35 0 About the authors:Celeste Tường Vy Sharpe is a PhD student in history and art history at George Mason University, studying the visual culture of disability in post-World War II America. She works as a graduate research assistant at the Roy Rosenzweig Center for History and New Media, and a teaching assistant for the online courses Hidden in Plain Sight and Virginia Studies.Nate Sleeter is a PhD student in history at George Mason University and graduate research assistant at the Roy Rosenzweig Center for History and New Media. He works as a teaching assistant for the online courses Virginia Studies and Hidden in Plain Sight.Kelly Schrum is the director of educational projects at the Roy Rosenzweig Center for History and New Media and an associate professor in the Higher Education Program and the Department of History and Art History at George Mason University. She has worked for more than a decade to create innovative, open digital resources and tools for teaching and learning, including Teachinghistory.org and History Matters.