ÁñÁ«ÊÓƵ

Can peer review survive social science¡¯s paradigm wars?

<ÁñÁ«ÊÓƵ class="standfirst">If authors are no longer required to justify their fundamental assumptions, where does that leave referees, asks Martyn Hammersley
July 19, 2022
Footballers arguing with the referee in the centre to illustrate
Source: Getty (edited)

Peer review is essential to the operation of research communities. But what are the conditions that must be met for it to operate? One is surely that there must be some minimum level of agreement about the task of research and how it should be pursued.

Yet increasingly, in many areas of the social sciences and humanities, there are fundamental divisions not just about the nature of what is being studied and how to understand it, but even about what the product of research should be. For instance: are human beings, organisations and institutions causal agents operating in the world or are they discursive constructions with nothing lying ¡°outside the text¡±? Is the aim of research to understand the world or to have ¡°impact¡± on?it, including reducing social inequalities?

Involved here are clashes between fundamental commitments, resulting in the ¡°paradigm wars¡± whose future famously predicted back in?1989. Is peer review compatible with these conflicts? Let me illustrate the problem.

Recently, in reviewing a paper for a journal, I?faced a dilemma. The paper made some interesting points, but my view was that it relied on a range of doubtful empirical, theoretical and political assumptions that led to tendentious interpretations of rather thin data. The authors implied that any questioning of these assumptions amounted to an attack on their intellectual and social identities. But I?felt that since many readers would not share them, explicit justification ought to be provided.

ÁñÁ«ÊÓƵ

Given this, I?recommended major revisions. The other reviewers were more favourable, though they did identify various issues that needed attention.?The editors decided on minor revisions. The resubmitted paper made no substantive changes relevant to my comments. More importantly, the covering letter did not respond to me at all, only to the other two reviewers.

The editors subsequently asked the authors to address my comments, but their response was that since they had a different ¡°onto-epistemological position¡±, they did not need to offer counter-arguments to the specific points I?had made. They claimed that the differences between us were ¡°irreconcilable¡± and outside the scope of their paper, and that my ¡°standpoint¡± informed my ¡°personal opinion of ¡®valid data¡¯¡±. They insisted that their ¡°methodologies¡± had been ¡°thoroughly vetted by the academy as reliable, valid and trustworthy¡±. In other words, since they were able to appeal to a literature that shared their commitments, there was no need to justify these in the paper.

ÁñÁ«ÊÓƵ

I disagreed and recommended rejection, especially since, in my view, the paper still had fundamental defects. However, the other reviewer at this second stage recommended publication and commented: ¡°Kudos to these authors for their pushback to Reviewer?1.¡±

¡°Pushback¡± sounds like a military metaphor; the implication seems to be that mere engagement with the critical points I?had made would have amounted to surrender in the face of what needed to be repulsed. Note that what is at issue here is not that the authors did not modify their paper; peer review does not require this. The problem is that they at first refused to make any response, and when pressed, simply appealed to their own paradigm commitments. And it worked. The article will soon be published.

Of course, it could be argued that I?should have acknowledged the legitimacy of the alternative paradigm to which the authors claimed adherence and accepted that my critical comments were therefore inapplicable, simply reflecting my own paradigm. Many would argue that it is a principle of academic life that diversity in orientation should be welcomed, and I?agree up to a point. But there are clearly limits to toleration ¨C indeed, both sides of the dispute discussed here recognised this in practice. The question is: on what grounds should those limits be determined?

The authors were effectively rejecting a key assumption underpinning peer review. They adopted a version of what has come to be called standpoint epistemology, according to which some views are granted credibility ¨C and others denied credibility ¨C on the basis of the social category to which those putting them forward belong. The perspectives of members of marginalised or oppressed groups (or, rather, those who claim to speak on their behalf) are assigned epistemic privilege, while those held by members of what is taken to be the dominant group are rejected, for example as (at?best) ¡°opinions¡±.

ÁñÁ«ÊÓƵ

So on one side, the limits to what is acceptable were being defined in terms of a paradigm that the authors freely acknowledged is committed to political goals; whereas on my side, the limits were taken to derive from what is required if academic peer review is to operate. The clash arises from the fact that peer review demands that all ¡°peers¡± be treated as equal, none as epistemically privileged.

Indeed, anonymity is used to render both authors and reviewers blind to (among other things) each other¡¯s social characteristics and political views, as far as possible. All that is held to matter is that they are fellow members of the same research community; and they are required to engage with one another solely on that basis, not according to political or paradigm commitments.

Can abandoning this requirement be tolerated within peer review? It has never been perfect, of course, but if peer review is just another vehicle for paradigm warfare, can it still be justified?

Martyn Hammersley is emeritus professor of educational and social research at the Open University.

ÁñÁ«ÊÓƵ

<ÁñÁ«ÊÓƵ class="pane-title"> POSTSCRIPT:

Print headline:?Social science¡¯s paradigm wars imperil peer review

Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Register
Please Login or Register to read this article.
<ÁñÁ«ÊÓƵ class="pane-title"> Related articles
<ÁñÁ«ÊÓƵ class="pane-title"> Reader's comments (10)
Very interesting article, thank you Martyn. I'd suggest this is further complicated by the more old-fashioned* politics of, for example, supporting and promoting your faction against rivals; crushing and marginalising personal rivals; etc.. * Only kidding, I know they're still alive and fully at play here.
Good grief. I can only thank the good Lord that I'm not a social scientist. Long live Sokal and Bricmont
1) The Sokal affair paper was in a non-peer-reviewed journal. 2) Sokal and Bricmont¡®s book only sets out to address a) the perceived misuse by a select set of authors of terminology from mathematics and the natural sciences, and b) a very particular subset of critical sociology of science which they saw as going too far to be useful. Both, particularly the latter, are important and worth discussing, but neither are fundamental flaws which somehow doom the social sciences and humanities.
The Sokal paper was indeed published in Social Text, which describes itself as governed by an editorial collective. Presumably this means that more than one person read the hoax paper. Peer review by another name?
Is it common to have editors of a journal simply shrug off a reviewer's comments like that and accept that the writers were immune from criticism because they had a privileged 'onto-epistemological position'? Is the whole journal now committed to that same position? As someone once said -- I think we should be told.
In a world where tweets and tabloid headlines seem to influence public policy and opinion more than serious social science, and where arguably most academic publishing has little or no impact, other than perhaps the career trajectories of the authors, I wonder if there is a deeper problem than simply peer review. The case for standpoint-free, objective social science as the way to counter fake news, seems like a sensible idea. Yet, history would suggest that many of the established theories of social science are tainted with all kinds of ideological baggage. So, until and unless we can devise a cast iron basis for objectively determining the quality of academic publications, I think accepting and making one's standpoint offers the reader a clearer way to evaluate the work being reviewed. Hence, rather than dismissing standpoint theory, shouldn't all research be assumed to be from a particular standpoint which needs to be acknowledged?
Gurnam Singh is right, of course, but standpoints vary according to their openness and the strength with which they are held. Martyn Hammersley is defending a standpoint based on academic argument and notions of evidence which makes his views open to correction in principle at least. A suitable response would have met his objections. The authors of the article seem to be adopting a standpoint that is rooted in their personal identities and political commitments that is far less open to argument and correction: any attempt to question their conclusions must only be an unwarranted personal attack aimed at their strongly-held beliefs, probably even personally or politically motivated. The test for me would be -- is there anything at all that the authors would accept that would invalidate their conclusions?
Couldn't agree more with Darris. I may feel the earth is flat but it is not true. There are many areas of social sciences, arts and humanities that make a valuable contribution to our existence often through invaluable critique of the paths down which we tread. The issue, however, seems to boil down to what counts as research or excellent scholarship and what should be labelled personal opinion. Everybody is entitled to a personal opinion and indeed it is valuable to chronicle these in diaries and or perhaps even a Journal of Personal Opinion. Of themselves, such writings provide useful data as a reflection of the times and as new data for future scholars. The extent to which we should pay any attention to such opinions now should be determined by whether or not a set of clearly identified and agreed criteria have been applied to the collection of information, its synthesis and process of interpretation. One such basic criterion could be applied as a minimum standard is whether or not the author is able to satisfy a standard of reliability - in other words given the same data (however you define it - and here are rues around that) would the overwhelming majority of humanity come to the same interpretation and subsequent conclusion. Upon such things societies grow and exist relatively harmoniously. This doesn't mean that the majority are always correct in their attributions of causality but, using an accepted methodology, what we used to think was true can be refined accordingly when new evidence rolls in. Without demonstrated reliability in the methods there can only be (at best) minimal validity. If the opinions are focused solely through a lens of identity and/or political ideology they are likely to be characterised as being unreliable and therefore of dubious validity - but how can we be sure? Kahneman and Tversky's work on the biases and heuristics of the human information processing system clearly show how fallible we are as processors of information. Left unchecked by a clearly articulated methodology, designed to arrest our fallibility as information processors, our thought processes can only lead us in many (fruitless) directions like the preverbal headless chicken. The value in research is it should seek to answer a question and there are accepted ways of doing this. If you don't start with a clearly articulated question how can you do research? Without a clearly articulated question the activity is clearly something else and should be identified for clarity, honesty and integrity (it's ok ato ask what should the question be - this too can have an acceptable methodology). Some authors may believe passionately what they are writing is true (indeed some openly identify as activists) but in the court of informed public scrutiny it may not stack up in the collectively agreed reality and appears more as a cult or a new form of minorityreligion. Unchecked by openly articulated methods, we soon end up heading down various rabbit holes as some corners of academia demonstrate on a daily basis. At issue, surely, is how as a society, do we create new usable and meaningful knowledge some of which some might have direct application or some may used to generate new questions hopefully for a purpose with an end point in mind. Making a series of un-falsifiable statements based on untested assumptions is really not helpful nor should tax payers have to pay for it, or indeed the self-serving echo chamber from which it comes. Other areas, sadly many of which would be described by the many hues of post modernism, provide opinion dressed up, sadly, in the overcoat of respectability afforded by faculty membership. If it doesn't add value, which could be referenced in myriad ways, to those who fund it, why should it continue to be supported? Politicians and vice-chancellors, in particular, should be asked how the public is getting a valuable return on its research investment with respect to what often masquerades as research when it clearly is not.
I am very grateful to all those who have commented on my article. The discussion raises important and difficult issues that need addressing. I am especially indebted to darris for the accurate and concise summary of my position.
Did Donald Trump steal documents from the US government? That should be an answerable question as long as you have an agreed and clear definition of "steal". It should not matter on how you feel about the matter. If you do not have an agreed concept of truth, any bully can have their way and trample on their fellow citizens. At the moment, Judge Eileen Cannon is a threat not only to US security, but to all of us throughout the world. A world or unreason is a world of ignorance, pestilence, and worse.
<ÁñÁ«ÊÓƵ class="pane-title"> Sponsored
<ÁñÁ«ÊÓƵ class="pane-title"> Featured jobs