Integrating measures of replicability into scholarly search: Challenges and opportunities
Video Social & Behavioral Sciences 2025 Graduate ExhibitionPresentation by Chuhao Wu
Exhibition Number 509
Abstract
Challenges to reproducibility and replicability have gained widespread attention, driven by large replication projects with lukewarm success rates. A nascent work has emerged developing algorithms to estimate the replicability of published findings. The current study explores ways in which AI-enabled signals of confidence in research might be integrated into the literature search. We interview 17 PhD researchers about their current processes for literature search and ask them to provide feedback on a replicability estimation tool. Our findings suggest that participants tend to confuse replicability with generalizability and related concepts. Information about replicability can support researchers throughout the research design processes. However, the use of AI estimation is debatable due to the lack of explainability and transparency. The ethical implications of AI-enabled confidence assessment must be further studied before such tools could be widely accepted. We discuss implications for the design of technological tools to support scholarly activities and advance replicability.
Importance
The study makes several important contributions. First, we connect with and contribute to studies on design that support scholarly search and management by empirically documenting researchers' strategies for literature search, review, and evaluation, highlighting the need for more flexible and intelligent approaches. Second, we link researchers' literature review practices with their perceptions of credibility, demonstrating opportunities for the inclusion of reproducibility and replicability metrics. Finally, we explore scholars' perceptions of an AI-driven replicability estimation tool and how to integrate the replication prediction into the literature review. We believe that the findings and discussions can motivate the research community to further ponder the design implications and ethical considerations for systems enabling automated assessment of confidence in published findings.