Valerie Shalin, Ph.D. (Advisor); Debra Steele-Johnson, Ph.D. (Committee Member); T. K. Prasad, Ph.D. (Committee Member)
Master of Science (MS)
Qualitative data result from observation, video, and dialogue. These types of data are flexible and allow us to study behavior without imposing potentially disruptive data collection methods. However, subsequent quantitative analysis requires a time consuming, labor intensive initial coding process, and a second manual coding to calculate inter-rater reliability. I examined the use of machine learning algorithms to reduce the amount of manual annotation work required to perform inter-rater reliability measures on text data. By comparing machine-human and human-human raters using Cohen’s Kappa statistic and an informal analysis of the features used in machine learning classification, I identify the promise and limitations of machine rating for conducting the second coding effort used to determine reliability. I found that machine learning algorithms can be useful tools for supporting inter-rater reliability as a second coder, but there are limitations associated with the class balance of the data that may restrict their usage.
Department or Program
Department of Psychology
Year Degree Awarded
Copyright 2021, all rights reserved. My ETD will be available under the "Fair Use" terms of copyright law.