Simeon Underwood highlights inconsistency in assessments
When all the world and the quality assessment exercise were new, one of the few enjoyable features for us administrators was the response of the key academic staff in the departments fingered for assessment. Suddenly they started to behave like bright students. The more conscientious would submit their work well before the deadlines and smile smugly, the more romantic would work into the small hours and put their essays in the pigeon-holes of their supervisors as dawn was breaking over the quadrangle. For the academics, as for their students, there was one aim: how to achieve the best possible result with the least possible effort.
I had been pondering this for some while, unsure of what it amounted to, when I came across the Don's Diary by Russell Kinman (THES, November 11). It is detailed account of assessment visits includes the following.
"Over a lavish buffet lunch, we meet the students, hand-picked by the department, presumably on the basis of their loyalty and articulacy. Articulate they certainly are, and loyal to individual departmental staff, but after a little coaxing, rather more wine, and the mention of the word 'modularisation', a floodgate (sic) of concerns is unleashed".
This passage illustrates a basic inconsistency at the heart of the quality assessment process.
The October 1993 Assessors' Handbook states that student views are an integral element of the information assessors are supposed to use in reaching their verdict. Yet although there is an elaborate "protocol" telling assessors how to deal with members of the teaching staff when assessing teaching and learning, there is no matching protocol on how to deal with the students. The information they should be seeking from students is set out in detail; but how they should extract it is not.
This gap in policy was picked up by the Centre for Higher Education Studies in its April 1994 review and evaluation of the assessment exercise. CHES recommended that assessor interaction with students should be taken further and should be related more formally to course aims and objectives. The HEFCE's Further Developments document last June stated that this recommendation had been accepted "in whole or in part": but there is nothing more specific on this point.
The lack of policy and guidance from the centre on student feedback is reflected in the assessors' reports. Student opinion rarely plays a leading role. One exception, about the mechanical engineering department at Oxford Brookes University, begins: "In discussions, students were supportive of the unit . . . Student opinion of the different courses, as expressed to the assessors, varied . . . Students on foundation were generally satisfied with their course but reported several problem areas . . . Students on the BEng course were complimentary about on-the-spot response of teaching staff."
In other reports the student contribution is less pervasive but no less profound. In perhaps the most striking report, everyone's nightmare scenario happened: "Assessors were concerned at the very negative messages that were conveyed formally and informally by some former and current students . . . low student morale . . . poor atmosphere . . . strained relations . . . concern about some teaching approaches . . . the established systems of communication were failing, leaving (the students) feeling frustrated and powerless," (Nottingham University, applied social work).
Such fullness is, however, exceptional. The majority of reports include a brief paragraph to say that students were generally "appreciative" of the course in question, the teaching, the facilities, the pastoral care etc. However, they offer little insight into how the student contribution informed the assessors' verdict.
In some cases the assessors mention areas of concern raised by students. "The supervisory system is in most cases a very high quality system for undergraduate academic support. In a number of cases, however, students expressed dissatisfaction with the quality of supervision received . . . The level of dissatisfaction expressed suggests that more care should be taken with the initial selection and training of supervisors," (Cambridge University, history). Yet the department is rated excellent.
At least a quarter of the 200-plus reports make no reference to the student contribution. Thus no matter what your carefully hand-picked loyal and articulate students say, it seems to have little bearing on the outcome. For example: "The academic management of the modular course and system necessarily requires a high concentration of bureaucracy at the centre. Although this eases some of the law unit's burden, students can feel like a number rather than a person," (Oxford Brookes, law). Yet in spite of this the department was rated as "excellent".
Conversely, a favourable assessment report on the chemistry department at UCL said: "Informal staff-student relations are good. Academic staff clearly enjoy working at UCL and this is transmitted to the students. Both current undergraduates and recent graduates expressed a high level of satisfaction with the college's courses. The pastoral environment is enhanced by the presence of a dedicated room where students and staff meet regularly during breaks and at the regular research seminars to which final-year project students contribute."
Yet in spite of this comment the department was given a "satisfactory" rating. By contrast, the history department at UCL received an "excellent" rating even though staff/student relations receive a brief mention in the final report.
The audit and assessment bodies expect institutions to be serious in obtaining and using student views.
On the evidence of Russell Kinman's article and my own reading of the assessors' reports, this is the model they use to the same end: "coaxing" by assessors; starving students suddenly confronted with a "lavish buffet" and free-flowing wine; assessors' reports which often do not show whether or not they have made use of student feedback; and a hole in the centre where a policy ought to be. One wonders what a team of quality assessors would say about it.
Simeon Underwood, former assistant registrar at the University of York, is a freelance consultant.