Alison Wolf (Opinion, June 10) clearly indicates why comparisons of test results and educational standards over time are not normally valid.
The learning environment is changing with time, and score distributions would be expected to change for changing and unchanging questions. In either case, like is not being compared with like. Hence the rise in scores observed in the unchanging problem-solving questions in IQ tests may possibly be associated with, and even result from, increased exposure to computer games that hone problem-solving skills. Does this mean the subjects, or the population, are getting more intelligent or better at mathematics? No, IQ is multifactorial, and it is likely that communication and socialisation scores of the gamers will have decreased.
The primary recourse is the standardisation of score distributions. The only other analytical approach would be an attempt to build a multifactorial temporal model of the population of learners and the environments in which they learn so that a fully covariate-adjusted (or value-added) analysis of data could be performed. But we would probably have to be cleverer than we are, and have access to funding, to do this.
Keith Rennolls
Professor of applied statistics Greenwich University