J Educ Eval Health Prof.  2013;10:2.

Assessment methods in surgical training in the United Kingdom

• Evgeniou E
• Peter L
• Tsironi M
• Iyer S

Abstract

A career in surgery in the United Kingdom demands a commitment to a long journey of assessment. The assessment methods used must ensure that the appropriate candidates are selected into a programme of study or a job and must guarantee public safety by regulating the progression of surgical trainees and the certification of trained surgeons. This review attempts to analyse the psychometric properties of various assessment methods used in the selection of candidates to medical school, job selection, progression in training, and certification. Validity is an indicator of how well an assessment measures what it is designed to measure. Reliability informs us whether a test is consistent in its outcome by measuring the reproducibility and discriminating ability of the test. In the long journey of assessment in surgical training, the same assessment formats are frequently being used for selection into a programme of study, job selection, progression, and certification. Although similar assessment methods are being used for different purposes in surgical training, the psychometric properties of these assessment methods have not been examined separately for each purpose. Because of the significance of these assessments for trainees and patients, their reliability and validity should be examined thoroughly in every context where the assessment method is being used.

Keyword

Assessment; Examination; Recruitment; Surgical training

MeSH Terms

Certification
Great Britain*
Humans
Methods*
Psychometrics
Reproducibility of Results
Schools, Medical

Figure

• 

  Fig. 1 The surgical training pathway. MRCS, Member of the Royal College of Surgeons; FRCS, Fellow of the Royal College of Surgeons.

• 

  Fig. 2 Bloom’s cognitive domain taxonomy (adapted from: http://ww2.odu.edu/educ/roverbau/Bloom/blooms_taxonomy.htm).

Reference

1. van Hove PD, Tuijthof GJ, Verdaasdonk EG, Stassen LP, Dankelman J. Objective assessment of technical surgical skills. Br J Surg. 2010; 97:972–987.

2. Beard JD. Assessment of surgical skills of trainees in the UK. Ann R Coll Surg Engl. 2008; 90:282–285.

3. Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet. 2001; 357:945–949.

4. Searle J, McHarg J. Selection for medical school: just pick the right students and the rest is easy! Med Educ. 2003; 37:458–463.

5. Benbassat J, Baumal R. Uncertainties in the selection of applicants for medical school. Adv Health Sci Educ Theory Pract. 2007; 12:509–521.

6. Parry J, Mathers J, Stevens A, Parsons A, Lilford R, Spurgeon P, Thomas H. Admissions processes for five year medical courses at English schools: review. BMJ. 2006; 332:1005–1009.

7. Randall R, Davies H, Patterson F, Farrell K. Selecting doctors for postgraduate training in paediatrics using a competency based assessment centre. Arch Dis Child. 2006; 91:444–448.

8. Patterson F, Ferguson E, Thomas S. Using job analysis to identify core and specific competencies: implications for selection and recruitment. Med Educ. 2008; 42:1195–1204.

9. Gale TC, Roberts MJ, Sice PJ, Langton JA, Patterson FC, Carr AS, Anderson IR, Lam WH, Davies PR. Predictive validity of a selection centre testing non-technical skills for recruitment to training in anaesthesia. Br J Anaesth. 2010; 105:603–609.

10. Davies H, Archer J, Southgate L, Norcini J. Initial evaluation of the first year of the Foundation Assessment Programme. Med Educ. 2009; 43:74–81.

11. Welchman SA. Educating the surgeons of the future: the successes, pitfalls and principles of the ISCP. Bull R Coll Surg Eng. 2012; 94(2):1–3.

12. Pelgrim EA, Kramer AW, Mokkink HG, van den Elsen L, Grol RP, van der Vleuten CP. In-training assessment using direct observation of single-patient encounters: a literature review. Adv Health Sci Educ Theory Pract. 2011; 16:131–142.

13. McKee RF. The intercollegiate surgical curriculum programme (ISCP). Surgery (Oxford). 2008; 26:411–416.

14. Mitchell C, Bhat S, Herbert A, Baker P. Workplace-based assessments of junior doctors: do scores predict training difficulties? Med Educ. 2011; 45:1190–1198.

15. Driessen E, van Tartwijk J, van der Vleuten C, Wass V. Portfolios in medical education: why do they meet with mixed success? A systematic review. Med Educ. 2007; 41:1224–1233.

16. Tochel C, Haig A, Hesketh A, Cadzow A, Beggs K, Colthart I, Peacock H. The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No 12. Med Teach. 2009; 31:299–318.

17. Hutchinson L, Aitken P, Hayes T. Are medical postgraduate certification processes valid? A systematic review of the published evidence. Med Educ. 2002; 36:73–91.

18. Pandey VA, Wolfe JH, Lindahl AK, Rauwerda JA, Bergqvist D; European Board of Vascular Surgery. Validity of an exam assessment in surgical skill: EBSQ-VASC pilot study. Eur J Vasc Endovasc Surg. 2004; 27:341–348.

19. Iqbal IZ, Naqvi S, Abeysundara L, Narula AA. The value of oral assessments: a review. Bull R Coll Surg Eng. 2010; 92(7):1–6.

20. Ward D, Parker M. The new MRCS: changes from 2008. Bull R Coll Surg Eng. 2009; 91:88–90.

21. McGaghie WC, Cohen ER, Wayne DB. Are United States Medical Licensing Exam Step 1 and 2 scores valid measures for postgraduate medical residency selection decisions? Acad Med. 2011; 86:48–52.

22. Patterson F, Carr V, Zibarras L, Burr B, Berkin L, Plint S, Irish B, Gregory S. New machine-marked tests for selection into core medical training: evidence from two validation studies. Clin Med. 2009; 9:417–420.

23. Patterson F, Baron H, Carr V, Plint S, Lane P. Evaluation of three short-listing methodologies for selection into postgraduate training in general practice. Med Educ. 2009; 43:50–57.

24. Oosterveld P, ten Cate O. Generalizability of a study sample assessment procedure for entrance selection for medical school. Med Teach. 2004; 26:635–639.

25. Turner R, Nicholson S. Can the UK Clinical Aptitude Test (UKCAT) select suitable candidates for interview? Med Educ. 2011; 45:1041–1047.

26. McManus IC, Ferguson E, Wakeford R, Powis D, James D. Predictive validity of the Biomedical Admissions Test: an evaluation and case study. Med Teach. 2011; 33:53–57.

27. Emery JL, Bell JF. The predictive validity of the BioMedical Admissions Test for pre-clinical examination performance. Med Educ. 2009; 43:557–564.

28. Lynch B, Mackenzie R, Dowell J, Cleland J, Prescott G. Does the UKCAT predict Year 1 performance in medical school? Med Educ. 2009; 43:1203–1209.

29. Davis MH, Karunathilake I. The place of the oral examination in today’s assessment systems. Med Teach. 2005; 27:294–297.

30. Bandiera G, Regehr G. Reliability of a structured interview scoring instrument for a Canadian postgraduate emergency medicine training program. Acad Emerg Med. 2004; 11:27–32.

31. Dore KL, Kreuger S, Ladhani M, Rolfson D, Kurtz D, Kulasegaram K, Cullimore AJ, Norman GR, Eva KW, Bates S, Reiter HI. The reliability and acceptability of the Multiple Mini-Interview as a selection instrument for postgraduate admissions. Acad Med. 2010; 85(10 Suppl):S60–S63.

32. Humphrey S, Dowson S, Wall D, Diwakar V, Goodyear HM. Multiple mini-interviews: opinions of candidates and interviewers. Med Educ. 2008; 42:207–213.

33. Urlings-Strop LC, Stijnen T, Themmen AP, Splinter TA. Selection of medical students: a controlled experiment. Med Educ. 2009; 43:175–183.

34. Patterson F, Ferguson E, Norfolk T, Lane P. A new selection system to recruit general practice registrars: preliminary findings from a validation study. BMJ. 2005; 330:711–714.

35. Roberts C, Newble DI, O’Rourke AJ. Portfolio-based assessments in medical education: are they valid and reliable for summative purposes? Med Educ. 2002; 36:899–900.

36. Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007; 29:855–871.

37. van der Vleuten CP, Schuwirth LW. Assessing professional competence: from methods to programmes. Med Educ. 2005; 39:309–317.