UX, Data & Research
How can we know if what we know is true?
Valid scientific claims are made based on replicable observations and, in this regard, the replication of published research is an important form of scientific validation. Although the broader scientific community is aware of this and replication has been deemed a cornerstone of the scientific method by the philosophy of science, little incentive exists to promote and facilitate the practice of replicating scientific studies as well as share replication results. This led to what has been called a “Reproducibility Crisis” in science and efforts are now underway to understand and remedy this crisis.
The goal of this thesis is to bring current research being conducted in metaresearch - the study of scientific research itself - to the field of the Digital Humanities to understand if it is also affected by issues observed in other fields, and explore topics and issues related to replication in the context of DH.
Methods:
Two studies were conducted to achieve this goal: a survey of papers published in DH and literary criticism journals in 2021 and the replication of a published DH research project that analyzed changes in the lexicon and sentiment of popular US songs.
Results:
The results of the survey indicate that DH exhibits similar transparency indicators observed in other disciplines: half of the 110 papers that relied on empirical data were available as open access; roughly a third shared the code used for data analysis and roughly two-thirds shared the data used in the study
Conclusion:
Better transparency indicators and journal policies that encourage authors to adhere to a culture of replication would facilitate replication efforts such as the one conducted for this thesis, by reducing time and energy needed to recreate data and code used to validate and extend results.
Transparency indicators of the 526 papers evaluated in the study:
Transparency indicators of the 110 papers that relied on quantitative data broken down by the journals they were published in:
Original results (bottom) and replication results (top) of five measurements originally published by Peter Meinderstma in his research on changes in lyrical diversity in Billboard hit songs:
Access the full thesis document.