Research-based assessment design in physics: including scientific practices and feedback for physics faculty

Abstract

Calls to transform introductory college physics courses to include scientific practices require assessments that can measure the extent to which these transformations are effective. Such assessments should be able to measure students' abilities to intertwine important concepts with practices in which scientists engage. In addition to evaluating student outcomes, another related goal of research-based assessments is to evaluate the efficacy of courses. To accomplish this goal, these assessments should have a mechanism to provide faculty concrete suggestions to modify their courses, beyond numerical scores. An approach to achieve this is lacking in the research literature. This motivates us to explore effective ways in which student outcomes can be reported to faculty to facilitate concrete suggestions to modify courses, i.e. actionable feedback.

Physics education research (PER) has a history of developing and disseminating research-based materials to faculty with the intention to improve student learning. However, lack of a consideration of what faculty want in the first place when developing these materials limits faculty to use these materials as developers intended. Even if these materials were adopted, faculty modify these materials to align with their needs and local contexts. There is a recent call to create partnerships with faculty when developing materials for them. In this dissertation, we provide a mechanism to develop assessment tasks that address scientific practices, provide feedback for faculty, and explore features of the external feedback that can be supportive of regular course modifications made by two physics faculty.

To design assessment tasks that can measure students' abilities to intertwine physics concepts with scientific practices, we leveraged Evidence-Centered Design and the Three-Dimensional Learning Assessment Protocol with the focal scientific practice of "Using Mathematics." We conducted video recorded one-on-one think-aloud interviews to explore how students interpreted these tasks. We articulate our design process and the analysis of students' responses using the ACER (Activation-Construction-Execution-Reflection) framework. Our assessment tasks elicited students' abilities to intertwine concepts with mathematics and written solutions elicited evidence of their abilities to intertwine them most of the time.

We present a mechanism to design actionable feedback for faculty in parallel to developing a new research-based assessment: The Thermal and Statistical Physics Assessment (TaSPA). The feedback design mechanism is rooted in the student outcomes in response to assessment tasks in a coupled, multiple-response format. This assessment task format allows online test administration with streamlined evaluation of student work. We conducted semi-structured interviews with faculty to obtain their perspectives on the developed feedback. Thematic analysis was used to explore the nuance of the faculty perspectives on the generated feedback. We then discuss the process behind incorporating these perspectives into feedback for faculty.

We conducted two case studies of physics faculty to explore the nuance of experiences associated with their course modifications. These explorations can inform identification of features of the researcher-generated feedback that can be supportive of regular course modifications made by faculty. Two case studies of faculty revealed the features - content coverage of a course, time frame for course modifications, and typical enrollment - associated with modifying courses that can be incorporated when designing feedback for them.

This dissertation provides a mechanism to incorporate scientific practices into paper-based assessment design at the introductory college level, an approach to designing explicit feedback for faculty in the context of research-based assessments, and evidence supporting why partnership with faculty when developing research-based feedback for them is important.