|Contribution title||Improving efficiency of reference screening in systematic literature reviews using the RobotAnalyst text mining application: performance assessment in a systematic review on patient safety|
|Form of presentation||Poster|
Background: Conducting systematic reviews is often time-consuming and requires significant human and financial resources. To perform systematic reviews more efficiently, the Cochrane Collaboration and several health technology assessment agencies now consider the use of text mining/machine learning tools to substitute second reviewers in the reference screening process. RobotAnalyst, a web-based tool for text mining and automatic classification, was developed by the National Center for Text Mining (NaCTeM), Manchester, UK and seems promising for semi-automatic screening in systematic reviews (www.nactem.ac.uk/robotanalyst).
Objectives: 1) To assess RobotAnalyst performance in identifying eligible references and reducing screening burden and time for reviewers. 2) To provide an account of reviewers’ experience with using RobotAnalyst.
Methods: For a systematic review on patient safety measures for older inpatients based on administrative health data, we searched 6 databases and identified 5503 references (title and abstract). Semi-automatic screening supported by RobotAnalyst is being performed by a junior and a senior researcher (R1 and R2, respectively). We currently compare its performance to manual screening by another pair of researchers (R3 and R4) using standard Cochrane methodology including the Covidence application. Yield, burden, and median decision time will be measured during screening. We will also report on the perceived usability of RobotAnalyst.
Results: Of the 5503 references, 657 (12%) were considered eligible for full text retrieval by R1. Of these, 95% (n=624) were identified after deciding on 59% of references (n=3213). It means that the use of RobotAnalyst helped save 41% of the screening workload at this stage, while retrieving the vast majority of eligible references. In addition, the median decision time was reduced from around 100 to < 20 seconds per reference after screening approximately 20% of references (n=1350). Semi-automatic and manual screening by reviewers R2, R3 and R4 are ongoing. Performance and usability assessment will be completed by August 2017.
Conclusions: Our study should help systematic reviewers decide on whether using a text-mining tool, such as RobotAnalyst, is worthwhile for complex literature searches, e.g., in health services research or public health. First results suggest that this semi-automatic approach has great potential to reduce the burden of reference screening and to speed up the review process.