Examining Construct Validity of the Master's UEEUsing the Rasch Model and the Six Aspects of the Messick's Framework

The purpose of the present study was to explore validity of the University Entrance Examination for applicants into English Master’s Programs in Iran (Master’s UEE) with respect to the Messickian validity framework using the Rasch model.
The data of 18821 test takers taking the 2009 version of the UEE were analyzed with Winsteps and R package eRm. An array of tests including Anderson’s (1973) test were used to check unidimensionality of the test. Since the test as a whole did not show unidimensionality, the reading, grammar, and vocabulary sections of the test were analyzed separately through Anderson''s (1973) likelihood ratio (LR) test using R package eRm. The results showed that (a) all the items in all the sections displayed good fit to the model, whereas more than 5 % of the examinees misfit the model, (b) due to small variance of item and person measures, the Rasch model explained small amount of variance in each section, namely 19.7 %, 13.8%, and 22.1 % in the reading, grammar, and vocabulary sections, respectively, (c) item measures were invariant within sections, contributing to the predictive validity (in the traditional sense of validity as types) of the test, whereas person measured did not show invariance, suggesting multidimensionality of the data hence threatening the construct validity of the test , (d) the bulk of the items did not match the bulk of the persons and there were noticeable gaps in the person-item maps, (e) small variance of person and item measures resulted in low Rasch reliability estimates for the sections, namely .53, .54, and .45 for the reading, grammar, and vocabulary sections, respectively.