Peer review and hot cross buns

Ben Martin is going to explain how academics get their work published in peer-reviewed journals.

He is Associate Professor of Mathematics at the University of Auckland. He is also an associate editor for the Journal of the Australian Mathematical Society and has himself published a number of peer-reviewed papers.

Rachel: Let’s say, you’ve written a paper. You send it to a journal and it lands on the editor’s desk. What do they do with it?

Ben: Well, first they’ll have a quick look at it just to make sure it’s within the scope of the journal and see whether it’s written in a satisfactory sort way and not riddled with typographical mistakes or in terrible English. Generally the more prestigious the journal is, the fussier it is about matters of presentation. The editor has to make an initial decision about whether the paper is worth being sent for refereeing. Sometimes the paper will be rejected very quickly by the editor without being sent for refereeing on the grounds that it’s poorly written, or not in the right subject area for this particular journal, or the results are not considered significant enough for that journal.

Rachel: The editor has decided this paper is worth being sent off to referees. How many referees are there and how are they chosen?

 Ben: It depends on the journal. In my discipline, pure maths, it’s more common to just have one referee and referees are expected to do a very thorough job of reading the paper and checking it for correctness.  In other subjects, often things are less clear-cut than in maths: there can be several competing theories and no-one’s really sure which is correct.  So the paper might then be sent to two or three referees and the editor might choose referees with a range of different theoretical viewpoints. In particular, if the paper is debunking a paper written by somebody else, it might be sent to that other person – or someone in the same theoretical camp – for comment.

 Rachel: So the people who receive a paper for review, do they get paid anything for their review?

 Ben: Generally, no. They don’t get paid anything.

 Rachel: Does the editor get paid anything?

 Ben: A common sort of journal structure is you have an editor-in-chief, you maybe have a managing editor as well who deals with the mechanics of running the journal, printing, distribution, etc., and then there’ll be an editorial board, usually composed of academics from various Universities. The people on the editorial board don’t get paid, but the managing editor will often get paid a salary. The editor-in-chief may or may not receive some money for work they do but usually this person is an academic at some university and they’ll be receiving a salary from their University.

Rachel: How do the editors get selected?

Ben: Journals like to make sure they have a range of editors to cover all of the different topics that that journal deals with. Typically they look for people who are experts in the field and have themselves published a lot. This way they should be better at making judgements on other people’s work.

Rachel: If you become editor of a journal, is this a role for life?

 Ben: It depends on the journal. Usually this is something you do for a few years. Sometimes people will stay on editoral boards for ten or fifteen years, but probably more like five.

Rachel: The paper is now in the hands of a referee. What exactly does the referee do?

Ben: Something I should probably say is the referee is anonymous. The author who submitted the paper does not know who the referee is. Their identity is not disclosed. In some disciplines it’s common to have double blind refereeing which means the referee does not know who the author is, either. So the paper is submitted and the name of the author is blanked out which is supposed to make the process more impartial and objective.

Back to the question: the referee reads the paper, they’re supposed to decide whether the paper is suitable for publication in that journal and the kind of thing they look for is the suitability of the subject matter, the correctness of the results, the originality of the results – in the sense that they’re not the same as someone else’s results – which will involve reading other papers in the area. They also make a judgement on the significance of the results – how important are they? Are they important enough to be published in this particular journal?  The more prestigious the journal is, the higher the bar, and the more important the results have to be.  If the results are showing some established theory is wrong, for instance, then that tends to be quite important and more likely to be published than a paper that is providing just a little more evidence in support of some established theory.

The referee reads the paper and writes a report and they make a recommendation about whether the paper should be published or not. But the decision is not the referee’s: the paper goes back to the editor and the editor makes the final decision.

Rachel: How long does this referee process take from the moment the referee gets the paper to the moment they send back a report?

 Ben: It depends on the discipline. In some disciplines the referee only has a few weeks. In pure maths referees tend to spend a lot of time going through papers quite carefully and checking correctness. In other disciplines correctness might not be so easy to check – a referee can’t easily reproduce someone’s experiment – and this is one of the reasons why these other disciplines have more referees: to get a range of opinions.

Rachel: How many hours would you spend refereeing a paper?

 Ben: I’ve had papers before that I’ve easily spent 20 or 30 hours looking at. It all depends on the complexity and various other factors.

When the editor gets the recommendations they have to decide whether the paper should be published or get rejected. The options are: you reject or you can recommend publication with or without revisions. If the referee’s recommendation is very clear-cut then that makes it easier for the editor, but even then there are wider constraints. If you have a very prestigious journal, their acceptance rate is small. I don’t know what the acceptance rate for a journal like Nature is but it’s probably well under 10%. So even a paper recommended by referees might not be accepted because it’s competing with other papers also recommended by their referees.

If there’s more than one referee, they may well say different things and the editor might ask for one or more additional referee reports.

If the decision is to reject the paper the author can send it off to a different journal and the process starts from the beginning.

Rachel: What happens if a paper gets published and someone finds a mistake after peer review and publication?

 Ben: It depends on the nature of the mistake. So you might have a paper in physics for instance where the author is putting forward some new theory which matches all of the existing experimental evidence. Then somoeone goes out and conducts an experiment and gets data which contradicts the theory. In this case there’s no requirement or expectation that the author of the original paper will withdraw or retract or change anything they wrote, because what was in the paper was correct to the best of everyone’s knowledge and consistent with the existing facts and data. So what would happen then is just that the people who did the experiment would write their own paper.

It’s not uncommon for people to make mistakes in their papers. In pure maths maybe they’ll realise that one of their proofs is wrong. What you do then typically is publish an erratum: this is just a short note which will go in the same journal saying, in this paper, that was published on such and such a date, this part here is wrong and here’s how you fix it. In my experience in pure maths, these errata are not serious. It’s quite rare that you’ll get an erratum that says the whole main result is wrong. As long as the mistake was made in good faith and was reasonably minor,  you would publish an erratum. I guess if the mistake was so great that it invalidated the main result of the paper then you might withdraw it. Withdrawal or retraction is more common in cases where there’s some suggestion of malpractice – that the author has deliberately falsified the data.

Thanks very much, Ben. Ben is also an accomplished maker of hot cross buns, something he does annually on Good Friday, which is today:


The referee’s report on the hot cross buns was favourable.

39 thoughts on “Peer review and hot cross buns

  1. Interesting article. I didn’t know the finer points of peer review. The photo of the hot cross buns made me hungry. Ben’s the greatest! Lovely photo too.

      1. He probably has great buns too but that’s me being crude. Sorry, couldn’t help myself. 🙂

  2. Rachel and Ben,
    A very informative post.When did peer reviews of professional papers commence? Famously Einstein’s1905 paper on special relativity was not peer reviewed,although checked by The Editor of the Annalen der Physik ,as best he could,no doubt.
    Also,a mathematical query for Ben.This may be a commonplace question in mathematics ,but I came across it in the biography of a famous Theoretical Physician.If you take a number,say 112 and move the last number,2 to the front making the new number 211,in a progression of the two numbers thereafter, is there ever a point at which the second number is exactly twice the value of the first number from which it is formed?

  3. Hi Doug,
    Re peer review: according to Wikipedia (, peer review didn’t become standard until the middle of the 20th century.
    I’m afraid I don’t understand your maths question. What do you mean by “in a progression of the two numbers thereafter”: do you mean that you repeat the process by repeatedly moving digits from the right of the number to the left of the number?


  4. Ben,
    Yes,you repeat the process by repeatedly taking digits from the right of the number and placing the last digit in front of the remaining digits.You start with any three figure number and go on to infinity.I did not express myself well in asking the question.So for example,49,359 becomes 94,935.In that example,the second resulting number is obviously not twice the first number.Also in many examples the resulting number will be a lesser number than the first,e.g.56,512 becomes 25,651 and the second figure cannot be twice the first.
    Then we try larger numbers,1,748,318 becoming 8,174,831.Nope.And so on.
    Any thoughts?

  5. The situatiion is more complicated in some other disciplines.

    George Akerlof won a Nobel Prize in Economics primarily on the grounds of his paper, “The Market for Lemons” (on the economics of the used car market, which explained why the price of a new car drops substantially the moment it leaves the dealer’s yard), which he wrote in his first year as an assistant professor at Berkeley. He had the paper rejected by three journals, one of which asserted that it was trivial, and at another, referees claimed that it was simply wrong. Finally,The Quarterly Journal of Economics published and the Nobel eventually followed,

    This is not limited to economics, although it might be rare (or even absent) in mathematics..Physicist Max Planck once observed, “A new scientific truth does not triumph by convincing opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it… Science progresses funeral by funeral.”

    We might see if this is the case in mathematics when the first P=NP (or P~=NP) supposed proof surfaces – although, judging by the reception of Andrew Wiles’s work on Fermat’s Last Theorem, mathematicians may not set such great store by funerals as do physicists.

  6. What a lovely person Ben is! And Rachel, I hope you spent at least 20 to 30 hours conducting a thorough evaluation of the hot cross buns. He deserves a serious critique of his best work!

  7. Ben,
    All I know is that Professor Freeman Dyson says the correct answer has 18 digits.Hmm.
    Looks correct!

    1. Anything that makes the scientific process of peer review more robust and transparent is a good thing. There is a famous case of scientific misconduct – In 1998, Andrew Wakefield – former gut surgeon – published a paper in the medical journal, The Lancet (a good one I think), in which he claimed to find a link between autism and the MMR vaccine. The paper was later retracted by the journal, Andrew Wakefield was struck from the register and found guilty of more than 30 charges. The Sunday Times investigative journalist, Brian Deer, writes:

      But Wakefield’s key finding – a claimed time-link of just days between vaccination and autism – was a sham: laundering anonymized allegations for a planned lawsuit, which he had secretly been paid huge sums to back. The Sunday Times, February 2004

      But the damage was already done in the form of plummeting vaccination rates worldwide. Anti-vax groups still quote his study in defense of their argument, despite all the subsequent studies since his which find no link whatsoever, and despite the fact that his was found to be dishonest, unethical and had a clear conflict of interest.

      I can’t help but think though, that by linking to this article, you are somehow trying to discredit all peer-reviewed science. Peer review is not without its problems, something I think should be evident from my interview with Ben Martin above, but like capitalism, it’s all we’ve got at the present time. It is the peer review process that brought you modern drugs, the ability to buy stuff over the web, iphones and computers, the Higgs boson, solar power, our understanding of the natural world, space travel, our understanding of earthquakes, tsunamis and volcanoes and maybe one day, predict them.

      I had hoped, by blogging about the process, that people would see that peer-review offers a much more robust and transparent way of scientific progress, that is completely absent from blogs such as WUWT, which offers no such robustness or transparency, is quite often incorrect, biased and whose contributors often have a conflict of interest. The comments are full of vitriolic abuse whenever someone dares to challenge the status quo – I for one do not dare to comment there – and I have heard from people who do try to challenge the postings, that in some instances they become black-listed and are no longer able to comment. So I would seriously question any improvement to peer review that incorporates blogs. I’m not saying it isn’t a good idea, but just want to highlight that it too presents challenges.

      It is quite inconsistent to dismiss the more than 99% of published climate science articles that accept anthropogenic global warming, on the premise that they are fraudulent or incorrect or whatever, while simultaneously accepting the less than 1% of articles that reject global warming because, although subjected to the same level of scrutiny as all other papers, they are somehow correct. Even more inconsistent would be to accept articles and books written by people who have a conflict of interest, that have had no peer-review whatsoever and that do not provide sources for their claims.

  8. However, for the record, I am not tying to discredit peer reviewed paper. The article speaks for itself. As you acknowledge, peer review is not without its problems but you are wrong to suggest that competent and knowledgeable commentators on blogs such as WUWT cannot usefully advance scientific progress by accurate criticism as acknowledged in the article I sent you.

  9. Rachel,
    There are 2 assertions you make above which I must challenge.
    Without any attribution you claim apparently on hearsay grounds that commentators to WUWT are subjected to vitriolic abuse and are excluded from commentary and ultimately blackballed.You say you personally are afraid to comment there.Could you provide an example of such abuse and exclusion and blackballing?Could the language used by the offended party or parties be similar to the irrelevant ,and offensive posts on some of the sites you list for consideration?
    Secondly,you state that 99% of published climate science papers accept anthropogenic global warming with less than 1% rejecting the “settled science”.This is pure invention.I have never seen the figures of 99% versus 1% before.The usual ( discredited ) claim comes from Doran et al , and Naomi Oreskes who assert that the proportions are 97% to 3 % .The so-called surveys underpinning this claim arose from 75 of 77 warmist scientists who self identified out of some 10,500 approached,and answered 2 basic questions as to whether the climate had warmed and whether CO2 had contributed.I would have expected that the results to the survey should have been 100% to 0%, given the fact that no serious scientist doubts the consensus that humans put large amounts of CO2 into the atmosphere ,that some warming has resulted and that some further ( modest ) warming is to be expected.There is no scientific consensus on how much the world has warmed,or will warm,how much of the warming is natural,how much impact greenhouse gases will have on temperature,or how sea levels ,storms, droughts, floods flora and fauna will respond to warmer temperature.Nor on what mitigation steps ,if any we should take,and whether ,if at all ,such steps would have sufficient or any climatic effect.
    If you doubt the truth of what I say, I am happy to provide you with peer reviewed papers supporting each of these contentions.

    1. References are important, so fair request:

      * 99% vs 1% –

      * censoring and blacklisting on WUWT –

      * abusive comments –

      The last link provides links to specific comment threads on WUWT. I don’t want to link to WUWT directly from my blog.

  10. Rachel,
    I am replying to your last comment on Peer Review and Hot Cross Buns. You have referred me to James Lawrence Powell to establish that 99% of published papers over the last 20 years support CAGW and less than 1% are sceptical in nature. This is demonstrably false. Mr Powell mimics Naomi Oreskes and her infamous science paper of December, 2004, The Science Consensus on Climate. One commentator described this paper as among the silliest ever published in Science. I agree with that view and regard Mr Powell’s deductions and search results as total nonsense.
    You will recall my referring you to Fang et al 2011. There, five sceptical Chinese scientists state they prefer the views expressed at NIPCC over those of the UN IPCC. If you go to NIPCC, you can immediately falsify the statements of both Powell (0.17% sceptical papers) and Oreskes (0% sceptical out of 928 papers). That charlatan, Al Gore, has misinformed millions of people in his movie that Oreskes research proved there were no sceptical papers in 928 she found. Rachel, go now to NIPCC which lists hundreds of sceptical papers. In October last, I count 45 sceptical papers alone which exceeds, in one month, the total of 34 for the 20 year period identified by Mr Powell.
    You may care to read from October, 2012, one sceptical paper showing how warming of 2 degrees Celsius enhances reproduction of inter-tidal New Zealand crabs. See Van den Brink et al in The Journal of the Marine Biological Association of the U.K. 92.515-520. This interesting paper shows that warming temperatures increase the productivity of the crabs.
    The fact that Oreskes could pass peer review when any independent check by the reviewers would falsify her thesis shows how corrupted the climate science discipline has become.
    Lastly, the latest tranche of Climategate emails shows the contempt in which mainstream climate scientists hold Oreskes. On 12 November, 2009, Tom Wigley wrote to Phil Jones. He was endeavouring to do a Web of Knowledge search not dissimilar to Mr Powell’s. He ended up with a crazy result. Here is his comment:-
    “Phil, This is weird. I used Web of Knowledge, “Create Citation Report”, and added 1999 through 2009 numbers……Here are three results….
    Kevin Trenberth, 9049
    Me, 5523
    Ben Santer, 2407
    The max on their list has only 3365 cites over this period.
    Analyses like these by people who don’t know the field are useless. A good example is Naomi Oreskes’ work.
    Oreskes might not know the field in the estimation of her peers but her science paper is good enough to fool Al Gore and the public.
    If her peers treat Oreskes and Powell with contempt privately, why should I believe their nonsense?

    1. I don’t know anything about Naomi Oreskes, so can we eliminate her from this discussion please? I did not bring her into the discussion, you did, and I should only have to defend the research I have included.

      James Powell searched the Web of Science, not the NIPCC or the UNIPCC. His methodology is explained here. The Web of Science in an online academic citation index which includes 12,000 academic journals. It would be difficult to find a greater breadth of academic papers on climate change anywhere else.

      The 0.17% of articles out of a total of 13,950 since January 1991 are articles that clearly reject human-caused global warming. They are not articles that accept human-caused global warming but think it won’t be that bad or that it will be good for New Zealand crabs.

      No, I’m not going to go and read the NIPCC website which receives funding from the Heartland Institute which in turn gets funding from the fossil fuel industry. Why would I, when I can go straight to library journal databases (as James Powell has done), which are far more transparent and which bypass institutions like these that have a conflict of interest.

  11. I quote from Powell, i.e. from the site you referred Doug to, “virtually all of the global warming that Oreskes and I separately reviewed can be classified……..”

  12. Rachel,
    You are putting your head in the sand.Mr.Powell’s search process is obviously flawed,whether he purports to get it from the Web of Science or on Tablets of Stone from God via Moses.
    If you disdain looking at NIPCC,on the grounds of conspiracy theories,try this as an alternative.Mr.Powell mentions as sceptical scientists,Lindzen, Singer and Watts.Add the prominent sceptics listed on Wikipaedia, and total their published papers in the period 1991 to 2012.It won’t take you long to get past 0 (Oreskes),and 24 (Powell).My simple point is that you are being gulled by people who are clearly very intelligent, but are prepared to say anything however implausible for the “Cause”.
    Worse you now appear happy to disseminate these claims without any one of a number of simple cross checks,which would falsify them.

    1. I think it is you who is gullible and has his head in the sand. I don’t have the time or the inclination to go and find papers that reject human-caused global warming. I don’t doubt that they exist, but their numbers pale in comparison to the mountain of evidence on the other side. Even if I found 25 papers or perhaps 100 (as opposed to Powell’s 24), it is still insignificant compared with the 10s of 1000s on the other side.

      But to still reject human-caused global warming, I would not only have to dismiss this haystack of evidence and replace it with a tiny needle, I would also need to dismiss the public statements of dozens of scientific organisations around the world. And how many scientific organisations have publicly denied climate change? ZERO

      I would then need to align myself with people who receive funding from fossil fuel companies. So with due respect, I am not the one with my head in the sand.

  13. It’s clear that you’ve become a denier yourself, preferring to dismiss out of hand any conflicting research by labelling it all as part of an unworthy conspiracy led by fossil fuel research.

  14. It’s a pity Graham Refearn can’t recognize a metaphor or hyperbole (or hyper bowl according to our P.M.) when he sees it.

  15. Rachel,
    On the Arctic Ice melt theory and record cold European winters, see The Met Office report of April 2013 by Professor Julia Slingo ,Met Office Chief Scientist,,”Why was the Start to Spring 2013 so cold?”
    It finds little evidence of a difference between the cold spring of 1962 and this year.The Report also finds little evidence that “Arctic amplification ” is responsible for any increase in extreme weather that results from prolonged conditions such as droughts flooding, ,cold spells and heat waves.
    According to the report ,”Figure 13 shows the mid troposphere temperature for 1962 and 2013 over the Arctic to be almost identical and reflect the negative NAO (natural North Atlantic Oscillation) pattern.It is hard to argue that Arctic amplification had changed the Equator to Pole temperature in a systematic way to effect the circulation this Spring.”
    The whole report lists 4 possible explanations of which 3 are natural variability and the 4th is Arctic sea ice melt. The Report then effectively admits that Arctic sea ice melt or “Arctic amplification”was not responsible for the unusually cold Spring in Europe,2013.The Arctic ice melt theory is said to be “contentious”.
    I am not sure how they would rationalise a different outcome for winter 2012/2013.
    Love ,

    1. That’s a great article, thanks.

      Have you noticed though, how we can read the same article, yet come away with a completely different conclusion?

      Julia Slingo writes,
      “Preliminary and ongoing research at the Met Office Hadley Centre is providing increasing evidence that the loss and thinning of Arctic sea ice predisposes the winter and spring atmospheric circulation over the North Atlantic and Europe to negative NAO regimes, as was experienced at the start of this spring.”

      It looks to me as though the contentious hypothesis she talks about is not that Arctic melt is influencing weather in Europe, but how it is doing so.

      But you are right, she does conclude that the Arctic is just one of several possible factors, and that although the cold March was unusual, is was not unprecedented –
      “It is worth re-emphasising, however, that while changes in the Arctic are consistent with predisposing the climate system to cold weather in northern Europe, this is only one possible driver among several potential factors which could account for the cold March weather.”

      Here’s the link:

  16. Rachel,
    After your latest reply ,I thought I’d resolve the Arctic ice melt issue ,so I rang my friend Professor Klondike at the UK Met Office.He said,”Arctic Ice Melt? Unofficially,Doug,we’re buggared if we know why it’s been so bloody cold ! ”

  17. Rachel,
    This paper is now 30 years old but raises some interesting issues on peer review not only for climate science but for all disciplines of science.
    “Peer-review practices of psychological journals:The fate of published articles submitted again”,Douglas P. Peters and Stephen J.Ceci, Behavioral and Brain Sciences (1982),5,pp.187-195.
    Does this paper mean that with the “correct” titles,individuals and institutions, papers will pass peer review and be published ,but without same,they are “tragically flawed” and rejected? ( h/t WUWT,”Peer review falls for recycled manuscripts”,May 29,2013)

    1. That paper was sent to open access journals. I’m not sure whether you know the difference between the traditional journal and an open access journal?

      Over the last decade or so, academics have become frustrated by the cost of gaining access to journals. University libraries usually subscribe to them but the cost is in the thousands per year. WIth funding cuts, some libraries have had to cut subscriptions to some journals. There is a movement to make academic papers available for anyone to read for free through what is known as open access journals. In these journals, academics pay to have their paper published. Some of these journals, I believe, have worked quite well. Like PLOS journals ( However, this movement did give rise to a crazy number of shonky operators promising to publish papers and provide open access but without having any great quality control.

      Academics tend to know which journals are good and which aren’t. William Connelly writes about it in STOAT (

      The serious point, as I take it, is the murky industry of pay-to-publish journals, which threatens to either pollute the science-o-sphere with trash, and/or rip of poor authors. On the second point: well, its always been part of science to know what the credible journals are in your field, based on their reputation, and based on the papers you’ve read that they’ve already published. If you submit to journals that are filled with trash, you’ve shot yourself in the foot.

  18. Is this the legendary W. Connelly banned for six months from Wikipedia for editing articles (over 5400 revisions to be exact) on climate change? The accusation against him being that he tried to “remove any point of view which does not match his own”.
    To say academics “tend to know which journals are good” instantly dismisses the majority of the population who don’t know the difference between OA journals and non-OA journals!
    It still isn’t clear whether or not the above sting would have worked with non-OA journals.

    1. It’s hard for non-academics to know which journals are good and which aren’t. I think if you search through your local library though, then you should be fine.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s