Features

We all know it. There are some papers that are easy and some that are hard. You may have been warned to steer clear of the legendary POLS101 if you value your grade point average, or that Biochem will endanger the mental health and social life of every first year Health Sci. As one professor commented upon overhearing another student praising my decision to change from one Arts paper to another, “the true course approval.” Far be it from Critic to meddle with this comprehensive and established system, so we decided to switch the focus to the university. Is our honourable institution aware of discrepancies in paper quality and difficulty and what do they do about it?


We can start at the beginning, at the birth of a paper. Papers can be brought into being as part of a new degree programme or can be added to an existing one. First an academic or academics within the department to which the paper will belong discuss it with their Head of Department (HOD). It will then go through a departmental process to the relevant division panel – Science, Health Science, Commerce or Humanities - followed by a central university committee, the Board of Undergraduate Studies (or the Board of Graduate Studies if it’s a post-grad paper). The prospective paper then strays outside the university walls to the Committee on University Academic Programme, a national body made up of representatives from competing New Zealand universities.
                                                              
Professor Kerry Shephard from Otago’s Higher Education Development Centre (HEDC) is one the many people involved in this process. “We start off with an academic using their professional skills, experience - their footprint - within the discipline in order to make a judgment…then it goes through the peer review process.” What are the questions being asked? “They would be thinking about what content it should have in it. Where does it fit within the degree programme? What was wrong with the last paper, for example, or, if it’s a totally new programme, why would they need this sort of area of content? At the same time, they’ve got to think about the level at which that content is addressed. That’s particularly obvious…in areas where things build on one another. You can’t study some concepts in chemistry unless you know about molecular structure and about valency and things like that.”
 
Once the paper has reached the students, the university keeps an eye on papers through regular evaluations facilitated by the HEDC. Of course, the most obvious evaluating is ad hoc; our lecturers have a pretty good view from their podiums and are not immune from distinguishing covert texting and Facebook-checking students from the bright-eyed students on the edge of their seats.  To corroborate this, there is the feedback from course representatives and the student perception forms you’ll be used to your teacher awkwardly distributing to you before leaving the classroom at the end of each semester. Some are teaching evaluations, which reach only the teacher involved, while others are evaluations of the course as a whole, which are given to a much larger group of people within the relevant department.
 
According to Shephard, students are not always the best evaluators of their courses. “All too often the teachers ask the students about the paper in terms of how difficult was it? Was it at the right level? Was the content right? Some aspects of that we shouldn’t be asking the students, we should be asking international experts, our peers, members of the academic community, but not the students, because that’s what they’re here to learn.” And that’s who they ask. The HEDC encourages peer reviews to be undertaken by other academics, HoDs and external experts brought in by departments to sit in on papers and reflect on how they’re going. External advisors are also used in adjudicating on the assessment process and in doing so often evaluate the role of the course as a whole.
 
And then there’s what could be for students something of a holy grail of paper evaluation, the Grade Comparison reports. The HEDC compiles these by looking at your results across 100, 200 and 300 level papers. It identifies in each paper the percentage of students who receive 5% higher or lower than their average grade at that level. Any paper in which more than 25% of students receive a higher or lower grade is identified as a ‘stand-out’ paper.
 
This analysis is then distributed to HoDs who can then do with it as they see fit, although in doing so Shephard tells them, “please keep this confidential. Please do not use it to enable or encourage students to choose the paper that’s going to be easiest to pass, that’s not what it’s for. That’s just using the data for the wrong purpose. If it’s used for the right paper I think it’s great data, but in the wrong hands it’s not constructive.”
 
The ‘correct purpose’ apparently is as a tool for self-evaluation by various departments, who themselves might be able to pinpoint the reasons for any dramatic, or not so dramatic, inconsistencies in results. Not everyone agrees with the usefulness of the Grade Comparison reports. The last one was circulated in 2008 and while the respondees to a survey on it mostly found it useful, the majority of HoDs didn’t respond at all. “I just throw them in the bin,” says Mark Henaghan, Dean of Law. He prefers a less bureaucratic, “external auditing” approach. “As head of department, you don’t need forms and reports. We’re never going to be perfect, but things keep ticking along. You know when teaching is good or bad because you communicate with students.”
 
Critic has nothing against a bit of bureaucracy, but wondered whether the university should really be sitting on the information it is collecting. After all, it is our grades and feedback the evaluation techniques are based on, and it is the quality of our education that they are assessing. Shephard’s response to whether students should have access to this information was “Yes and no. What I recommend, and what I think most of my colleagues in HEDC recommend, is the first lecture of a new course, the lecturer says how they’ve changed the course in relation to the feedback that they’ve had before…I know it doesn’t happen in some parts of the university and I suspect there are some parts of the university where the lecturers just bin everything, but that’s human nature…we call [telling students the outcomes of evaluations] closing the loop and it’s something that HEDC feels very strongly about.” But not when it comes to the Grade Comparison Report. “I think [it] is a really interesting instrument but there is always the danger that it will be used for the wrong purpose. I cannot help the fact that different papers develop reputations, that’s going to happen, but we don’t have to add data that supports that reputation, that for example says the average pass rate for that paper is X and the average pass rate for that paper is Y. There’s always the danger that will force all the students to go to X or Y, and then it becomes even more difficult for the HoD to support the continuation of Y and yet the profession needs Y.”
 
It’s worth asking what this all means for our education, and we don’t need to rely on specific evaluation outcomes to discuss this. Papers that have a reputation for being hard or easy may have that reputation for a number of reasons. It could be a problem that needs to be addressed. “It might be that the intended learning outcomes were just too ambitious,” says Shephard. “The level of the course could have been set too high for the students. It might have been that we’d chosen the wrong prerequisites so the students were bright enough but not experienced enough in the right way. It could be that the teacher was absolutely awful and another teacher would have been able to get all the students to pass.” Or vice versa for the ‘easy’ papers.
 
But other factors could be involved. Certain Theatre or Maori Studies papers may be seen as ‘easy’, but the passion and motivation of the types of students likely to take such papers may make result in better performance. Going the other way, a paper may simply be hard because the content is always going to be hard for most people. Mark Henaghan points out that all law papers require an understanding of the facts of the relevant cases, so areas of law that have facts that we encounter every day, such as family law, are going to be easier for most people than getting their heads around in the facts involved in a Secured Transactions paper.
 
Tony Zaharic, the co-ordinator of BIOC192, describes some of the factors that lead to this paper’s reputation; “first, biochemistry is a difficult discipline.  Second, though I would argue that physics and chemistry are more difficult, most first year students have been exposed to PHSI and CHEM at high school and thus are already familiar with the language of the discipline. One of the hardest aspects of BIOC is becoming familiar with the language of the discipline, for which there is no real world framework, and it takes time for the students (with our help) to build the real world analogies and links to BIOC-specific language that make any discipline easier to understand…Finally, I think there is an element of self-perpetuating myth. If second year and above students tell first year students that the paper is difficult, they come in with preconceived ideas.” Nevertheless, “the feedback we get from class representatives is that BIOC 192 overall is one of their favourite papers”. 
 
Perhaps this isn’t surprising. “Hard and easy are just perceptions and if you just want to pick all the papers you perceive as easy for your entire degree, then why are you here?” asks Henaghan. “You can make a paper by how much you engage with it, how much you participate in it. Or you can just become part of a degree machine.”