Policies Regarding Review of Conference Papers


Confidentiality

Papers under consideration for publication should not be distributed or disclosed to any individual, except with the explicit authorization of the program chair for purpose of review. Accepted papers are considered unpublished until the day of the conference, and should not be discussed outside the program committee without explicit permission from the authors. All reviewers must be made aware that submissions are unpublished works, and in no case should they duplicate them, pass them on to other people, or discuss them with people other than the members of the program committee. The program chair should keep a record of the external reviewers used to review submissions.

Reviewers grant paper authors permission to use the reviews however they wish. Reviews can also be analyzed under the auspices of SIGPLAN for research purposes such as evaluation of bias.

The title, authors, acceptance status, and other metadata (including discussion summaries, if they exist) can be shared under the discretion of the program chair, when there is an expectation of legitimate benefit for other SIGPLAN processes (such as a related review committee, or an artifact evaluation process).

Conflicts

Authors and PC members must adhere to SIGPLAN’s conflict of interest policy.

A conflict of interest is defined as a situation in which the reviewer can be viewed as being able to benefit personally from the outcome of a review, or in which the reviewer is not able to remain objective for personal reasons. If a conflict of interest exists, then the reviewer should decline to review a paper. During the program committee meeting, any committee member who has a conflict of interest regarding a paper may be asked to leave the room for the duration of the discussion.

Among the situations in which conflicts are typically considered to exist are between an advisor and an advisee (Ph.D. and post-doc; forever), between an author and a co-author (papers and proposals; for two years), between people at the same institution (branches of large companies or different locations of research institutes are considered to be the same institution; for two years after leaving an institution), between people with financial conflicts of interest, and between friends or relatives. If a reviewer feels unable to render an objective judgement for any reason, he or she should notify the program chair.

Contacts

Reviewers should not interact with authors about their submitted papers. Any interaction should go through the program chair.

Review Process

Author Anonymity

SIGPLAN conferences should implement double-blind reviewing. The primary goal of DBR is to help PC members review papers with minimal bias. The process should be such that authors are able to withhold their identity, and reviewers are able to avoid learning their identity:

  1. Submissions should not have author names, and references to previous work must be in the third person.
  2. Authors are not required to “hide” their submissions: they can put them on web pages and give talks about them.
  3. Reviewers should notify the PCC if anonymity is an obstacle to sound and objective reviewing of a paper.

Given the use of DBR, the PCC must vet any external reviewers suggested by a PC member.

Recommendations:

Provision (2) above is intended to prevent DBR from inhibiting normal dissemination of scientific ideas. Authors should not, however, take it as a license to explicitly lobby the PC or likely reviewers on behalf of their work. While SIGPLAN will not attempt to codify or police such behavior, PC chairs may wish to remind authors of their obligation to live up to the spirit as well as to the letter of DBR. A major point of contention concerns posting research results in social media or wide-reach mailing lists. PC Chairs should carefully consider to what extent such publicity conforms to their implementation of DBR and inform authors accordingly.

The two variants of DBR in current SIGPLAN use are “lightweight DBR” and “full DBR”. Under lightweight DBR, author names are revealed to reviewers after the initial review is submitted and before paper deliberations. In full DBR, author names are not revealed until the end of deliberations, and possibly only for accepted papers. These variants (and their many variations in implementation) reflect different models of the threat of reviewing bias and the benefit of knowing author identity. Lightweight DBR is based on the belief that authorship information should not bias “first impressions” of the paper’s quality, but the reviewers’ informed technical opinion will be unaffected, and even improved in depth, if author identities are later revealed. Full DBR rejects all authorship information as a bias factor.

Guardians

Guardians may be used at the discretion of the program chair. A guardian is a PC member assigned responsibility for ensuring that a paper is carefully and fairly assessed. Concretely, a guardian’s role may include: Assisting the PCC in identifying gaps in reviewer expertise. Ensuring that all reviews are of an appropriate standard. Facilitating online discussions. Writing the author-visible summary of the decision rationale.

Expert and External Reviews

SIGPLAN conferences should strive to identify expert reviewers for every submission, insofar as possible. An “expert” is a reviewer who is very well versed and current in related work in the field. Authors gain confidence in the outcome of review decisions when expert reviewers are involved. That said, well-informed, but non-expert reviews also play a significant role in acceptance decisions: they represent the majority of the future readership of a paper, and involving them mitigates against topical balkanization.

Recommendation:

Several successful formulas for obtaining expert reviews have been used in the past:

  • The PCC designates and announces an External Review Committee (ERC) in the CFP.
  • The chair (sometimes in consultation with the PC) identifies a roster of external reviewers after submissions have closed.
  • The paper’s “guardian” PC member (if applicable) identifies external reviewers, in consultation with the PCC.

The PCC should exercise some judgment in assessing expertise ratings. Some PC members consider themselves experts on most topics, while other, more modest, PC members almost never declare themselves an expert. In the end, the PCC should use his/her judgment in deciding whether the expertise on a given paper is sufficient.

External reviewers should be encouraged to participate in online discussion of the papers they’ve been assigned; however, they should not see unrelated online PC discussions.

Recommendations for Submission of Supplementary Material

Authors can be allowed to submit supplementary material (examples, proofs, software, datasets, etc.) at the time of submission. The PC is allowed, but not required, to consult this material. Supplementary material should be made anonymous if possible and reasonable. For example, proofs and datasets should always be anonymized, while source code may be too difficult to reasonably anonymize.

Evaluation Criteria and Acceptance Ratio

Conferences should strive to accept all high-quality submissions, and avoid numerical limits on acceptable papers. S Recommendation:

  • The PC should focus its deliberations on: – whether there is a genuine research contribution which may include a new insightful evaluation of previous work – whether the approach is fundamentally sound – whether the community will benefit from reading the paper – whether the paper (and supporting material) contain sufficient information for others to reproduce and build on the results
  • The committee should lean toward accepting papers that are controversial, that is, ones that, after discussion, still have both a strong advocate and a strong detractor.
  • The committee should lean toward accepting papers that explicate their results clearly.

PC Submissions

PC submissions can be problematic to manage, but, on balance, the benefits of allowing PC submissions exceed the costs. Conferences should allow PC submissions, but disallow submissions by the GC and PCC.

PC members should not be allowed to review or discuss other PC papers; hence all reviews of PC submissions must be external. The PCC will designate the roster of external reviewers for PC papers, manage the review process, and make the final acceptance decision, in consultation with the reviewers. SIGPLAN requires that PC papers be held to a higher standard than other papers.

Acceptance decisions for PC papers will be announced at the same time as other author decisions are announced; i.e., the PC will not be aware of any PC paper decisions until the PC meeting is complete.

Recommendation:

The policy on PC submission should be clearly explained to candidate PC members when their participation on the PC is solicited.

An External Program Committee (EPC) can be used to evaluate PC submissions. The EPC can be a subset of the ERC, or a separate, non-overlapping committee, or “EPC” can be used as a replacement term (or synonym) for the ERC.

Author Response Recommendations

Recent SIGPLAN conferences typically allow author responses to reviewers. Authors should be allowed to read all reviews that are available at the time of the author response period. The PCC should make every effort to complete all initial reviews before the author-response period. Additional reviews may be solicited after the author response period and it may not be possible for the authors to see these reviews before final decisions are made by the PC. Ideally, authors should be allowed at least four days to respond. Additionally:

  • Authors strongly prefer seeing the scores at author response time. The PCC should consider making the scores available as well as the reviews.
  • Authors should be strongly encouraged to be brief.
  • PC chairs may choose to state that the PC is not obliged to read or respond to rebuttals beyond a certain designated length.
  • The PCC may choose to allow authors to provide feedback on review quality, e.g., via a simple rating scale, in addition to rebutting the content of the review.