Assessing validity of dentists evaluations in measuring Index of Orthodontic Treatment Need using modeling agreement

In assessment orthodontic treatment need, for the same group of people, dentists usually have different ideas with respect to a gold standard. The most common way of determining validity of the assessments is computing coefficients of agreement like Kappa and Weighted Kappa. In this article we first present shortcomings of the coefficients and then propose a more appropriate alternative method for evaluating of agreement. Methods & Materials: This analytic cross-sectional investigation has been done with the Index of Orthodontic Treatment Need (IOTN). This index which independently was measured on 51 study models by an expert dentist (as gold standard) and 10 other dentists (as raters), has 5 ordinal levels (no need, little/mild need, moderate/borderline need, severe need, extreme need). Diagonal parameter model and uniform association model were used for modeling agreement to evaluate validity of the assessments.

In 6 cases, diagonal parameter model (with significance level at least 0.34) and in 4 other cases, uniform association model (with significance level at least 0.36) were appropriate to the relevant data. The best assessments belong to the dentists for whom uniform association model was fitted and the worst of them to the dentists for whom diagonal parameter model was fitted.

All of the dentists tended to rate casts higher than the gold standard, so that local odds ratio for dentist who had the least(most) tendency to rate casts higher than the gold standard was 6.4(121.5), So all of them needed to be further educated to assess the index.