Situational judgement test: Psychometric analysis of a pilot study for selecting post graduate medical student in residency program

Abstract

Introductions: Selection process of post graduate medical students requires combination of knowledge, cognitive abilities and skills. Situational judgement test (SJT) is one on the important tools to measure cognitive abilities and skills. This study aims to measure the internal consistency reliability of SJT tool and item quality for local validation of the non-academic attributes. It also aims to do sub-group analysis of SJT scores based on this tool to provide further evidence for validity.

Methods: This cross-sectional study was conducted at Patan Academy of Health Sciences (PAHS) in May 2017. Two sets of SJTs were developed after iterative discussion and modification to suit the local context. Set A had five options to a given situation, students had to rank these options from most appropriate to least appropriate. Set B had seven options to a given situations, students had to pick three best options. Medical officers and interns who were working at PAHS as medical officers participated in this study.

Results: One hundred and sixteen medical officers participated in the study. Thirty-nine (33.6%) participants were interns who had graduated from PAHS and 77 (66.4%) were medical officers who had graduated from various institute inside and outside the Nepal. The overall Cronbach’s alpha for 35 questions was 0.65, that for set A (19 questions) was 0.56 and that for set B (16 questions) was 0.52. Adding SJT questions improved value of Cronbach’s alpha for SJT test as 40, 50, 60 and 70 questions gave Cronbach’s alpha of 0.68, 0.73, 0.76 and 0.79. Average percentage correct of set A was 76.3% and that of set B was 60.5%. SJT scores were statistically different for medical graduates from Nepal and different universities within Nepal.

Conclusions: Locally developed situational judgement test is found to be reliable instrument for measuring non-academic construct of post-graduate medical entrance examination.