World J Mens Health.  2023 Apr;41(2):342-353. 10.5534/wjmh.220128.

Need for Training in Research Methodology Prior to Conducting Systematic Reviews and Meta-Analyses, and the Effectiveness of an Online Training Program: The Global Andrology Forum Model

• Cannarella R
• Shah R
• Boitrelle F
• Saleh R
• Durairajanayagam D
• Harraz AM
• Agarwal A

• KMID: 2540966
• DOI: http://doi.org/10.5534/wjmh.220128

Abstract

Purpose
Systematic reviews and meta-analyses (SRMAs) are used to generate evidence-based guidelines. Although the number of SRMAs published in the literature has increased dramatically in the last decade, the training and the experience of the researchers performing the SRMAs are usually not explained in the SRMAs’ methodology, and this may be a source of bias. Although some studies pointed out the need for quality control of SRMAs and training in proper statistical methods, to the best of our knowledge, no study has reported the importance of training the researchers that conduct the SRMAs. The aim of this study is to describe a training program designed to impart the essential knowledge and skills required for the conduct of an SRMA and to assess the need for, and outcome of, such a training.
Materials and Methods
Researchers were trained for use of Scopus, study eligibility, assessment of the quality of evidence (QoE) through the Cambridge Quality Checklist for observational studies, as well as the Cochrane Risk of Bias tool, the Consolidated Standards of Reporting Trials (CONSORT) guidelines, and the Jadad score for randomized controlled trials (RCTs), Population, Intervention, Comparison, Outcome (PICO) questions and data extraction. A total of 35 of them were approved to join a planned SRMA. At the end of the SRMA, they were administered 43 multiple-choice questions (MCQs) on demographics, motivation for participation in the SRMA, self-perceived change in knowledge before and after conducting the SRMA, and self-assessment of performance. The senior researchers then revised the spreadsheet of the SRMA and, based on the mistakes found, organized a training focused on the correct assessment of the study design, where 43 researchers (9 joined midway) and 11 trainees with no experience in conducting SRMA attended. They all were tested through a 5 MCQ assessment that was administered before and after the training. Those scoring poorly were re-trained and re-tested, and only those scoring satisfactorily were admitted to further SRMAs.
Results
Approximately 54.3% of the participants were medical doctors (MD), 31.4% were urologists and 48.6% had previous experience with SRMAs. Joining an international collaborative study was the main motivation, chosen by 19.7% of researchers. The results of the self-perceived change in knowledge showed a significant improvement in the use of Scopus, checklists for QoE, PICO questions, data required to perform a meta-analysis, and critical reading of scientific articles. Also, the majority of the researchers ranked the quality of their work as high. The pre-test results of the 5 MCQ showed a low score, which was not different from that achieved by a group of fresh trainees (median, 2; IQR 1–3 vs. median, 1; IQR, 1–2; p=0.3). Post-training there was significant improvement in both groups (researchers: median, 4; IQR, 3–5 vs. median, 2; IQR, 1–3; p<0.001; trainees: median, 4; IQR, 3–4 vs. median, 1; IQR, 1–2; p=0.02). Out of the 44 researchers, 12 (27.3%) scored poorly (≤3). After re-training, all of them scored satisfactorily (>3) and were admitted to subsequent SRMAs.
Conclusions
At the end of our model, 100% of researchers participating in this study were validated to be included in a meta-analysis. This validation required the involvement of the MT, two meetings, a self-evaluation survey, and one or two sets of objective tests with explanations and corrections. Our results indicate that even well-trained clinicians are naïve when it comes to the methodology of SRMA. All the researchers performing an SRMA need comprehensive training that must cover each aspect of the SRMA methodology. This paper provides a replicable training program that could be used by other investigators to train the researchers to perform high-quality SRMAs.

Keyword

Meta-analysis; Research; Systematic review; Teaching; Training