Providing a model for validation of the assessment system of internal medicine residents based on Kane’s framework

Kane’s validity framework examines the validity of the interpretation of a test at the four levels of scoring, generalization, extrapolation, and implications. No model has been yet proposed to use this framework particularly for a system of assessment. This study provided a model for the validation of the internal medicine residents’ assessment system, based on the Kane’s framework.

Through a five stages study, first, by reviewing the literature, the methods used, and the study challenges, in using Kane’s framework, were extracted. Then, possible assumptions about the design and implementation of residents’ tests and the proposed methods for their validation at each of their four inferences of Kane’s validity were made in the form of two tables. Subsequently, in a focus group session, the assumptions and proposed validation methods were reviewed. In the fourth stage, the opinions of seven internal medicine professors were asked about the results of the focus group. Finally, the assumptions and the final validation model were prepared.

The proposed tables were modified in the focus group. The validation table was developed consisting of tests, used at each Miller’s pyramid level. The results were approved by five professors of the internal medicine. The final table has five rows, respectively, as the levels of Knows and Knows How, Shows How, Shows, Does, and the fifth one for the final scores of residents. The columns of the table demonstrate the necessary measures for validation at the four levels of inferences of Kane’s framework.

The proposed model ensures the validity of the internal medicine specialty residency assessment system based on Kane’s framework, especially at the implication level.