Assessing the optimal method of detecting Differential Item Functioning in Computerized Adaptive Testing

Test fairness is one of the main challenges in transition from paper- pencil towards computerized adaptive testing (CAT). This study was aimed at investigating differential item function (DIF), assessing intervening factors in clarifying DIF and suggesting the optimal method for DIF in computerized adaptive testing. The empirical method was applied based on the nature of the study area. Data gathering procedure and manipulating the variables were done using simulation method. The responses of 1000 examinees (reference and focal group with equal 500 numbers) to item bank of 55 dichotomous items were simulated based on 3-parameter logistic model with 20 iterations. Fifteen items were manipulated in terms of DIF type and magnitude and test impact was evaluated based on mean difference of comparison groups. Computerized adaptive test with 30 items was administered via Firestar software package. Analysis was done by logistic regression (LR) and item response theory-likelihood ratio test (IRT-LRT) and the methods were compared based on their power and type I error rate. Type I error rate of likelihood ratio test was less than logistic regression. The power of the methods was influenced by type, magnitude of DIF and test impact. Comparing with logistic regression, Item response theory-likelihood ratio test had more power in detecting uniform DIF for the impact and no-impact conditions and it showed more power by increasing the magnitude of DIF. The two methods showed no difference in assessing non-uniform DIF and both of them were poor. Given the power and type I error rate, likelihood ratio test is an optimal approach in detecting uniform DIF. However, assessing non-uniform DIF requires further investigation.