Applying Machine Learning Methods to Generate Understandings of Differential Item Functioning in a Flu Knowledge Assessment

Authors

William L. Romine, Wright State University - Main CampusFollow
Tanvi Banerjee, Wright State University - Main CampusFollow
Derrick Cox, Wright State University

Document Type

Article

Publication Date

2025

Abstract

Current influenza trends, including the severity of the 2025 flu season and the prevalence of H5 bird flu in livestock, necessitate efforts to better understand how to educate students about its transmission. Although validated assessments of influenza knowledge exist, these have not been evaluated for affective and demographic biases. We explore differential item functioning (DIF) effects in four items focused on specific aspects of flu transmission derived from a validated influenza knowledge assessment. In doing so, we introduce and utilize a machine learning framework for exploration of DIF which offers greater flexibility than traditional statistical approaches in terms of studying generalizability of effects within and across study sites and across different modeling approaches. Both statistical (logistic regression) and machine learning approaches revealed that the largest DIF effects—perceived complications and barriers to preventative practice—were affective in nature. Demographic factors such as gender, ethnicity, and presence of health professionals in the students’ families, tended to emerge from the algorithmic models (random forest and neural networks), whereas the data models (like logistic regression) tended to overlook these smaller effects. While not a direct replacement for statistical approaches, we encourage researchers interested in understanding equity to treat machine learning as an additional resource in our toolboxes. To better understand how to educate students about communicable diseases such as H5 bird flu, moving beyond model-specific inferential methods toward model-agnostic machine learning-based methods will enhance our ability to detect biases in our assessments, and to focus on those biases which persist across different samples and diverse modeling paradigms.

Comments

This paper was presented at the 2025 NARST International Conference, March 23-26, 2025, National Harbor, Maryland.

Repository Citation

Romine, W. L., Banerjee, T., & Cox, D. (2025). Applying Machine Learning Methods to Generate Understandings of Differential Item Functioning in a Flu Knowledge Assessment. .
https://corescholar.libraries.wright.edu/cse/675