PODC 2021 Guidelines for writing a review

Written by: Keren Censor-Hillel, PC chair for PODC 2021
Date: February, 2021

PODC2021 – Guidelines for writing a review

The goal of this document is to describe what is expected from a review for a paper submitted to PODC 2021.

Preliminary notes:

i. This document contains my personal views and preferences, and is not an official document of the steering committee.
ii. Other conferences/PC chairs may have different expectations. Journals certainly have different expectations.
iii. Anyone is welcome to use any part of this document for creating their own guidelines.

Before the review: PC members are expected to read the introduction of their assigned submissions in order to find experts according to item 1.a. in the description of the review form below, for reviewing these submissions (preferably themselves).

The review form: The review form for external reviewers for PODC2021 is as follows:

  1. Description of the reviewers point of view, split into 3 categories:
    a. knowledge of the topic of the paper
    b. depth of reading the paper (hidden from authors)
    c. year of first publication (hidden from authors)
  2. Paper summary
  3. Strengths
  4. Weaknesses
  5. Comments for authors
  6. Comments for PC (hidden from authors)

The review form for PC members also contains a merit score. After receiving an external review, the PC members are encouraged to communicate with the reviewer and obtain as much information as needed for them to decide on a score. Guidelines for scores:

  1. Reject (This paper is in the bottom 50% of submitted papers)
  2. Weak reject (This paper is in the top 25-50% of submitted papers)
  3. Weak accept (This paper is in the top 10-25% of submitted papers)
  4. Accept (This paper is in the top 10% of submitted papers)
  5. Strong accept (I nominate this paper as a candidate for an award)

Notice that a paper whose merit is above average but not in the top 25% of submitted papers should be given a score of weak reject, as we cannot accept all papers that are above average. The scores only serve as a basis for the discussions of the PC and do not determine final decisions in an automatic manner.

The goal of the review:
The primary goal of a review is to help the PC select papers for the program. The review is also used for feedback to the authors.
I recommend splitting the review into two parts: evaluation and comments. The evaluation part is used for both goals above, while the comments part is mostly for providing feedback to the authors on a more lower level (minor scientific comments, structure of the paper, readability suggestions, typos, etc.). The split between the two parts should ideally be such that the PC members do not need to read the comments part in order to make decisions. The rest of this document refers mostly to the evaluation part.

The evaluation part of the review:
This should be split into two subparts, one which is visible to the authors and another which is confidential and visible only to the PC. The latter may be empty, and typically contains information that may reveal the identity of the reviewer or that contains a comparison to other submissions. The visible part of the evaluation should contain the following main ingredients:
1. A summary of the paper: description and technical content.
2. The merit and strengths of the paper, and its weaknesses.

1. Summary:
The summary of the paper consists of a description of the paper, and possibly some of its technical content.

1.1. The summary — description of the paper:
The goal of the description of the paper is to help a PC member who did not read the paper understand what is the area of the paper and the type of result. It should be very short (typically a short paragraph should suffice), and should avoid containing long lists of exact statements that appear in the paper, or repeating the abstract in different words. Instead, it should contain the essence of the paper’s content, which would be accessible also to a PC member who does not know much about the area. A typical summary begins by indicating the area of the paper, the subarea, and so on until it zooms into the research questions addressed in the paper. Then, it indicates the paper’s answers.

1.2. The summary — technical content:
The description could be followed by some technical explanation on how an algorithm works, or how a lower bound is constructed, etc. These should be included if needed in order to help the PC understand the points listed later as strengths or weaknesses. If the reasons for the evaluation are clear without diving into technical details, then explaining the technical details is not necessary.

The summary parts are usually fact-based, and so they also provide useful feedback for the authors as to what a typical reader (the reviewer in this case) perceives as the main contributions of the paper.

Both parts of the summary should be kept as short as possible, because the PC members do not have the capacity to read lengthy descriptions. Keeping the summary short while providing the necessary information is sometimes a task that requires a non-negligible amount of work in extracting main ideas, just as one does when writing other scientific documents. Revising this part until its ratio of information per text is optimal is work that extremely helps the PC.

2. Strengths and weaknesses:
This part contains the reviewer’s assessment of the paper, based on the summary that was provided earlier. The lists below are only examples. There are many more aspects of the paper that could be strong or weak..

2.1 Typical strengths could be:
* The paper reveals something new about a fundamental question.
* The paper gives a significant improvement in the round complexity or in a lower bound. Significance does not necessarily mean “a large number”. When indicating such a strength, the review should explain why the improvement is significant.
* The paper develops a new technique.

2.2 Typical weaknesses could be (compare with the strengths above):
* The question addressed in the paper is rather narrow.
* The improvement is not significant enough. In this case try to give an example of what would be a significant enough improvement.
* The techniques are rather standard.

Remarks:

  1. Style: It is perfectly fine to comment about things that are a matter of style. However, such comments should go into the comments part, and should not be stated as strengths and weaknesses. Examples include: “The algorithm is not given as a pseudocode” (if it is given in a different, non-ambiguous, manner), “It is better to change the order of this and that part”, and so on.
  2. Explicitness of pros and cons: Strengths and weaknesses should be stated explicitly, e.g., “the improvement in the obtained complexity is not significant enough” or “the technical novelty is insufficient”, rather than as general statements, e.g., “the paper is weak”, or “the result is not interesting”.
  3. Correctness: If a paper appears to be correct, this should not be indicated as a strength. It is expected that all submitted papers are correct. If, however, it is established that the paper is incorrect, then this should be indicated as a weakness. If the reviewer has concerns about correctness they should notify the PC member as soon as they can, and the PC member should notify the chair. Sufficient time could allow the PC chair to check this with the authors, if she decides to do so.
  4. Quality of writing: An exceptionally well-written paper is a strength, and an exceptionally hard-to-read paper is a weakness. Anything in between should go into the comments part.
  5. Wording of the review: The tone of the review should be constructive when possible. In any case, the text should be polite and should avoid any wording which could be perceived as offensive or unfair or unexplained. The best way for a reviewer to check this is to imagine that they receive this review for a submission of their own.

Concluding remark: Reviewing papers is an indispensable part of the publication process. It is hard work that consumes time. I wish to thank in advance all reviewers and PC members for PODC2021 for their service to the community.