College of Business

Retrieval of useful digitized learning objects is a key objective for educational digital libraries, but imprecise definitions of alignment hinder the development of effective retrieval mechanisms. With over 63,000 U.S. K-12 science and mathematics education standards and a rapid proliferation of Web-enabled curriculum, retrieving curriculum that aligns with the standards to which teachers must teach is increasingly important. Previous studies of such alignment use single-dimensional and binary measures of relevance. Perhaps as a consequence they suffer from low inter-rater reliability (IRR), with experts agreeing about alignments only some 20-40% of the time. We present the results of an experiment in which the dependent variable ‘alignment’ is operationalized using the Saracevic model of relevance in which; i.e., alignment is defined and measured through ‘clues’ from the everyday practice of K-12 teaching. Results show higher inter-rater reliability on all clues with significantly higher IRR on several specific alignment dimensions. In addition, a (linear) model of ‘overall alignment’ is derived and estimated. Both the structure and explanatory power of the model differ significantly between searching vs. assessment. These results illustrate the usefulness of clue-based relevance measures for information retrieval and have important consequences for both the formulation of automated retrieval mechanisms and the construction of a gold standard set of standard-curriculum alignments.

a