A Guide to Writing Your First Referee Report

One day, sometimes earlier than you expect, you’ll get an email asking you to referee a paper. The standard reaction, reported with absolute consistency by the people sitting around me at the AAS 247 peer review workshop, is some version of “I have no idea what to do with this.” Well, first things first: Being asked is a professional honor. It means the community has decided you have enough expertise to be useful.

The good news is that the peer review workshop, run by Frank Timmes, Brian Jackson, and Mubdi Rahman (all scientific editors at the AAS journals) was an explicit answer to that reaction. This guide is the (not complete) practical version of what they taught us, organized as a reference you can come back to when you actually have a manuscript open.

Here is the first thing I want you to breathe in and hold onto: you are not the one accepting or rejecting the paper. Your job is to give expert input, clearly and constructively, so the scientific editor can make a decision. When you remember this, the task becomes calmer and more doable. As Frank Timmes put it: “relieve yourself of the burden of rejecting a manuscript.”

A second framing helps too: the goal is to improve the manuscript. You are not there to win, or to prove the authors wrong or anything like that. You are there to make the paper better, or to identify the specific reasons it cannot be made publishable in its current form.

Now, with those out of the way:

1. Should you accept the invitation?

Before you even decide whether to accept, take a minute to make sure the request is legitimate. Early-career researchers sometimes receive review invitations from spammy or predatory journals, and those emails can look surprisingly official.

When the email arrives and you could confirm it is legitimate, ask yourself three questions and decide based on that:

1. Do I know the topic well enough? Subject-matter expertise is the actual reason you’ve been asked. If you can’t evaluate whether the result is significant and new, decline politely and, if you can, suggest someone who can.
2. Can I make the deadline? If you cannot, decline. “I’m too busy” is a completely acceptable reason. If you can do it with a short extension, ask upfront. Communicate early.
3. Do I have a conflict of interest? Same institution, PhD advisor or advisee, current/recent collaborator, former officemate, known personal relationships, are all conflicts. Editors mostly know who’s collaborating with whom, but not always. Disclosing a conflict is your responsibility. If you have a conflict, disclose it and decline. If you are unsure whether something counts, tell the editor and let them decide.

(Also, before accepting, read the journal’s reviewer guidelines. This guide grew out of an AAS workshop, but different journals can have different expectations about confidentiality, anonymity, conflicts of interest, report structure, and review timelines. If you are reviewing outside AAS journals, or in a neighboring field, do not assume the process is exactly identical.)

2. Read the paper

A small trick that makes reviews much better: Read first, write second. 

Trying to write the report while you are still figuring out the paper usually produces scattered reviews and makes big-picture judgment really hard. Instead, do this:

Pass 1: Read for the big story

Ask yourself: what problem is the paper addressing, what did they do, what did they find, and why?

If you cannot answer those questions by the end of the first pass, that might not be a failure on your end. That is useful information for the authors! A referee is an expert reader, and if the story is unclear to you, it will be unclear to many readers.

During this first pass, take messy notes. Not polished. 

Pass 2: Read for the science

Now go back and ask more technical questions:

• Significance and novelty: Is the “new thing” genuinely new, and is it explained clearly? 
• Correctness: Are assumptions stated and justified? Are conclusions supported by evidence? 
• Context: Is the paper placed correctly in the literature? Are key references missing?
• Presentation: Does the title reflect the content? Are figures readable, labeled, and actually helpful? Are captions informative? It is worth checking figure clarity and accessibility. AAS has a public graphics guide with practical advice.
  

3. Write the report

The editors offered a clean four-part skeleton and encouraged us to follow on our first review:

1. One short summary. Start with 1–3 sentences summarizing the paper in your own words. This is not for the authors, they know it! This summary has a few purposes:  (a) it reminds the editor what the paper is about, (b) It protects everyone from misunderstandings. If you misunderstood the central claim, the authors can spot it immediately.
2. Overall recommendation. Most systems ask for a recommendation like accept, revise, or reject. Say what you think should happen next, and why. If you recommend rejection, do it only when you genuinely believe the manuscript cannot be made publishable with reasonable revisions. Even when you recommend rejection, the editor still decides.
3. Major comments. These are the comments that change the paper’s scientific value: missing tests, questionable assumptions, missing context, etc. You need to make these comments useful: Each one should be specific and actionable: “This is wrong” is not actionable. “This assumption is not justified because…” is.
4. Minor comments. These are line-level issues: Go line-by-line. Quote the line you’re commenting on, give the page or section number, and use a marker (e.g., > or –) at the start of each item so the authors can produce a tidy response.

Good reports range from a couple of paragraphs to a couple of pages. The metric is usefulness.  A one-line “looks great, accept” report is, in the editors’ words, suspicious. Even very strong papers usually can benefit from something, figure improvements, clarity or some other small notes.

4. Tone and language: Be clear and useful

This was the single most-emphasized point of the entire workshop for me: review the research, not the researcher.

When something is confusing, try to say it like this: “I got stuck in Section 3 because…” and then name the exact missing definition. It lands better than “This is poorly written,” and it gives the authors something they can actually fix!

Also try to include one genuinely positive sentence early on, not flattery, just something true like: “Figure 2 is doing a lot of work.”

One more behind-the-scenes reality: editors sometimes edit reports before forwarding them, especially if language is hostile or unprofessional. If you are tempted to write something spicy, it is likely to be removed anyway. Save yourself time and write it constructively.

5. The “notes to the editor” box

Many systems have two text boxes:

• Report to the authors. What it sounds like.
• Notes to the editor. This one is Private. The authors never see this. Most reviewers leave it blank.

That second box is where you can maybe flag a possible conflict of interest you noticed while reading, or mention things like figure-organization concerns that the authors may not have control over at submission.

The editors said anything in this box really stands out, precisely because most reports don’t use it. 

6. Anonymity or signing your report

Most referee reports are anonymous to authors by default. The AAS journals offer double-anonymous reviews, where the identities of both authors and reviewers are concealed during the review process.

You may have the option to sign your report depending on the journal. Some people like signing because it makes the process feel more human. Some people avoid it, especially early in their career, because it can create awkward dynamics. There is no universal right answer. In all cases, it is the scientific editor’s decision whether a report is sent to the authors anonymously or not.

But there is one hard boundary: do not contact the authors directly. Even if you think you know who they are, even if you want to be helpful. All communication should go through the editor and the journal system.

7. What happens after you submit your report

Most papers are not accepted after the first round. The most common path is:

You submit a report. The editor decides whether the paper is acceptable, needs revision, or should be rejected. If it needs revision: the authors have a deadline to revise and respond. They write a point-by-point response letter.

The editor sends the revision and response back to the original reviewer (you) for a second look.  If you are reviewing a revision, your job is not to re-review the entire paper from scratch. Your job is to check whether the key issues were addressed, and whether new changes introduced new problems.

8. What the workshop changed about how I’ll write my papers

I went into this workshop wanting to learn how to referee. I came out with a very different impression of what makes a paper easy or hard to review. If you want to become a stronger writer, refereeing is one of the fastest feedback loops available. And to start that, you can attend this workshop in the next AAS meeting. The next workshop is going to be at AAS248 in Pasadena. Do not miss the opportunity to learn and practice all the procedures for yourself, with AAS editors by your side to guide you through everything and many more details I did not cover here. You can sign up for the next one here.

Thanks to Frank Timmes, Brian Jackson, and Mubdi Rahman for running this amazing and practical workshop, and to the AAS for making it free to attend. I am especially grateful to Frank for reading a draft of this guide and providing helpful feedback. Any errors or oversimplifications in this guide are mine.

If you want to read more, look at these other guides:

• AAS journals: Policies
• Astrobites: How to submit a paper

Astrobites edited by: Neel Kolhe

Featured Image Credit: Niloofar Sharei (Canva)