Guidelines for the Program Chair of a SIGPLAN Event


[N.b.: SIGPLAN rules are highlighted like this to distinguish them from general recommendations and discussion.]

Executive Summary

You may be an experienced Program Chair, in which case much of the advice on this page may already be familiar to you. Nevertheless, even experienced chairs sometimes forget some important points, and some policies and guidelines are specific to SIGPLAN, so please at least skim the rest of this page and the following others:

Please help us keep this document (and the rest of this site) up to date: If you notice any errors, duplications, inconsistencies, etc., please inform the SIGPLAN vice chair.

Submission and Reviewing Policies

The Program Chair (PCC) may not submit papers to the conference; SIGPLAN’s Executive Committee recommends that the General Chair also not be allowed to submit papers. If Program Committee members are allowed to submit papers, either their papers should be held to a higher standard or other EC-approved measures should be taken, to avoid the appearance of impropriety.

Conferences that use full double blind reviewing and have program committees with at least fifty members need not hold PC papers to a higher standard, as the EC has judged that these are sufficient conditions to avoid the appearance of impropriety. Here full double blind means submissions are anonymous, references to prior work are in the third person, authors’ identities are not revealed until a paper is conditionally or fully accepted, and author identities for rejected papers (that were not conditionally accepted) are never revealed.

The PC Chair should make all program committee members and reviewers aware of SIGPLAN’s policies on reviewing of conference paper submissions and on plagiarism.

The conference steering committee should be consulted for any changes to the accepted way of organizing and running the conference.

You, the Program Chair, in consultation with the General Chair, should decide on general policies relating to the program committee, such as:

  • Will you have a physical PC meeting? If so, when and where will it be held? If not, will the PC discussion be a “synchronous” teleconference or an extended discussion over email or moderated by some conference management platform? (Whichever you choose, SIGPLAN expects all PC members to participate fully in the PC meeting, except in unusual circumstances.)
  • Who pays for the travel expenses of the PC members?
  • Who pays for the conference registration of the PC members? Typically, the PC member is responsible for all of his/her expenses, including registration. If a decision is made to provide funding for some/all of the members, these expenses must of course be included in the budget.

The previous year’s Program Chair for the same conference is a good resource.

Program Committee Selection

In consultation with the General Chair, develop a list of potential Program Committee members. In formulating this list, be mindful of SIGPLAN’s diversity policy. Also, some people will decline the invitation, so your list should includes sufficient alternates. The list should be finalized at least one year prior to the meeting.

Download the PC proposal spreadsheet template and enter the requested information for your proposed PC members and alternates. Send the completed spreadsheet to the SIGPLAN Vice Chair (vc_sigplan@acm.org) for comments and approval by the SIGPLAN Executive Committee. The approval process normally takes less than two weeks. The same approval process applies to all other reviewing committees for conferences such as ERC and EPC.

At the same time, make plans for the Program Committee meeting, so that these can be communicated to potential PC members when you invite them: choose the location (if physical) and finalize the date(s).

After SIGPLAN approval is confirmed, invite the people on the list to serve. Explain what is expected of them and any relevant policy decisions, and give them the date(s) for the PC meeting. If more people than expected decline and you run out of alternates, send the SIGPLAN Vice Chair another set of candidates for approval.

Recommendations:

  • The PCC should strive to avoid the appearance of favoring current and former students, postdocs, colleagues, or collaborators for membership on the PC or ERC.
  • The size of the PC should be such that, given the expected number of submissions, will yield a reasonable reviewing load per PC member. Exceeding 25 papers per PC members is strongly discouraged.
  • If there is an ERC, the same diversity criteria listed above should apply to them, scaled to the size of each committee.
  • PC members should generally not serve on the PC more often than every four years. Exceptions may be made to enable a small overlap (<10%) from year to year to provide continuity. It is advisable that no one should serve on the PC for more than two years in a row.
  • The PCC of year X+1 can be invited to serve on the PC (with a lighter reviewing load) for year X as soon as he or she is selected.
  • The PCC should strive to avoid selecting PC members that collaborate regularly with each other or with the PCC, or PC members from the same institution in the exact same area.
  • For conferences, all PC members should have PhDs (or other appropriate qualifications) by the submission deadline of the conference. For affiliated workshops, the PC can include PhD students in their final year, as long as the total number does not exceed 10% (rounded up) the PC.

Call for Papers

Develop a call for papers. Note that you can produce a preliminary call without listing the PC members. Make sure to put the call on the web site for the conference (established by the General Chair). The web page should refer prospective authors to SIGPLAN’s Republication Policy and to ACM’s Policy and Procedures on Plagiarism. Once the conference web page is in place, establish a link to the page from the SIGPLAN conference page by contacting the SIGPLAN Information Director.

If the proceedings will be published by ACM, declare explicitly in the call for papers that authors of accepted submissions will be required to choose one of the following options: Author retains copyright of the work and grants ACM a non-exclusive permission-to-publish license (and, optionally, licenses the work with a Creative Commons license); Author retains copyright of the work and grants ACM an exclusive permssion-to-publish license; Author transfers copyright of the work to ACM. For more information, please see ACM Copyright Policy and ACM Author Rights.

Proceedings published by ACM as an issue of PACMPL are Gold Open Access, and the PACMPL editorial board recommends the first option (author retains copyright of the work, licenses the work with a Creative Commons license, and grants ACM a non-exclusive permission-to-publish license).
The PACM Steering Committee also requires conferences to collect statistics about the review process—see here for details.

If presentations are to be recorded, mention that presenters must sign a permission form in order for the video to be released online.

Mention that publication of auxiliary material is encouraged, and note the associated copyright policy (see ‘Auxiliary material’, below).

The conference web page should include information about the various benefits offered by the SIGPLAN Professional Activities Committee to SIGPLAN members attending SIGPLAN-sponsored conferences. These benefits include travel and lodging assistance for student authors and assistance for members who need a companion care-provider in order to attend.

Author Response

Many SIGPLAN conferences have included author response mechanisms as part of the paper review process, and the general feedback concerning author response has been positive (see Experiences with Author Response at PLDI and ICFP 2004, Kathleen Fisher and Craig Chambers, SIGPLAN Notices, Dec. 2004). Program Chairs and Steering Committees are encouraged to consider allowing author response or rebuttals as part of the paper review process.

Submission and Review

Set up the paper submission web site. Many services are available for this purpose; most SIGPLAN conferences these days use HotCRP (which is free for us because ACM has a bulk deal). If you choose a different one, look for support for the ABCD rating scale, author feedback, and double-blind reviewing.

Some advice: Many systems make it easy to include ratings on a large number of different scales, but please don’t do this. Most PC chairs use only two scales: an overall rating and an expertise rating. It also works best to have just two or three text fields: one for a summary of the paper (this is optional), one for comments for both the PC and the authors, and one for comments to be seen by the PC only. The ABCD scale avoids ranking papers by average: instead you can group papers by highest and lowest rating (not a linear scale, since AD neither precedes nor follows BC). For more details see: Conferences with improved management and style, Philip Wadler, SIGPLAN Notices, Feb. 2006.

Before papers are received, develop an evaluation procedure. After the submission deadline, send submitted papers and a description of the evaluation procedure to the PC members.

The PCC should identify another PC member (a “COI-PCC”) to handle papers for which the Chair is in conflict.

PC Meeting

Some SIGPLAN conferences have in-person PC meetings; others have moved to virtual meetings (to reduce carbon footprint, wear and tear on PC members, or both).

For both kinds of meetings, PC members should be allowed to participate in the discussion (both online and in-person) of papers that they didn’t review, and with which they have no conflicts. The PCC should strive to ensure, prior to the PC meeting, that every competitive paper has at least two PC reviews at least one, and preferably two expert reviews

During the online discussion period (whether or not followed by an in-person meeting), the PC should be allowed to see all papers with which they have no conflicts, and participate in the discussion. Online discussion must be actively facilitated by the PCC, and (if applicable) paper guardians.

Before the PC meeting, decide on any policies that should be in effect, such as:

  • What defines a conflict of interest?
  • What happens if a PC member has a conflict of interest?
  • What are the rules of confidentiality?
  • Do you want to have a best paper award?

In-Person PC Meetings

A physical PC meeting should be preceded by an extensive online discussion period. The online discussion period should be a minimum of one week, but two (or more) is strongly preferred.

The designated COI-PCC (see COI policies) should handle papers the PCC is conflicted with during the meeting.

At the end of the PC meeting, the PCC may want to take a bit of time to plan the sessions and the session chairs.

It is strongly recommended that the PCC have an assistant during the physical PC meeting, to take on certain administrative and logistical tasks during the meeting and allow the PCC to concentrate on leading the discussions of papers. Typical duties of the assistant would be to keep track of the disposition of papers for the PC, interface with the on-line review system, and record any relevant information about a particular paper (e.g., paper Y paper is being shepherded by committee member X, committee member X agrees to revise their on-line review, etc.). Past PCCs have used graduate students or post-docs as assistants, with good results.

Any PC meeting expenses that are to be reimbursed by the conference should be put in the conference budget. Send receipts to the ACM Representative to SIGPLAN for reimbursement (the SIGPLAN Vice Chair can tell you who this is at the moment).

Additional suggestions:

  • Experience has shown that a two day PC meeting provides adequate time for deliberation while avoiding PC burnout.
  • The GC may play the role of “COI-PCC”.
  • The PCC is encouraged to have one or two dedicated assistants to deal with timekeeping and bookkeeping in order to allow the PCC to focus entirely on the discussion rather than logistical concerns. A senior non-PC member such as the GC or next PCC can fulfill this role.
  • Instead of considering the papers in order from highest-ranked to lowest, consider the papers in a quasi-random order (see article by Kathleen Fisher in SIGPLAN Notices, 46(4):17, April 2011).
  • If the most positive reviewer for a paper is external, the chair may wish to ensure that the paper is discussed on day one of the PC meeting and the outcome summarized by the PCC so that the external reviewer can provide additional feedback prior to a final decision.
  • The PCC is encouraged to use a multi-round discussion process for papers where an initial consensus is not apparent.
  • The PCC can distinguish between contentious and marginal papers and table all marginal papers until the end of the meeting to allow them to be systematically considered in the context of the emergent acceptance standard.

After the PC Meeting

Send acceptance and rejection letters as soon as possible after the PC meeting. Authors using LaTeX should be directed to the template developed by SIGPLAN. This template was designed to allow more material in a paper while remaining easy on the eyes (typically, a paper that requires 12 pages with the standard ACM format requires 11 with the SIGPLAN format, and most readers find the SIGPLAN style easier to read).

Managing the Proceedings

The proceedings service by Conference Publishing Consulting (ConfPub) can be initiated once you decide that you want formal proceedings. Please contact the ACM Publications Coordinator, Adrienne Griscti, to begin the process. ACM will provide ConfPub with pertinent conference information. ConfPub will forward you a schedule and instructions on how to proceed.

Draft a “Letter from the Program Chair,” which should include a description of the paper selection process (number of papers submitted and accepted, etc.). Forward the letter to ConfPub along with any other front matter (perhaps a “Letter from the Conference Chair,” a List of Reviewers, etc.) for inclusion in the proceedings. ConfPub will format all front matter; their schedule will include a deadline for front matter submission. They will also compile the table of contents/author index, using the schedule of talks/advanced program that you provide to determine paper order.

Accepted authors will submit their papers directly to the ConfPub web site. Once ConfPub have collected all of the papers/copyright forms and front matter, compiled the table of contents/author index and paginated the volume, they will provide camera-ready copy for your review. The proceedings will not be printed until this has been approved. It is a good idea to send the table of contents to the contributing authors to confirm the spelling of their names and affiliations. You then will be asked to provide conference shipping information and the number of proceedings needed for the conference.

At the Conference

At the meeting, give a short summary of the paper evaluation process. Include the number of submissions, in what topical areas, and the number of accepted papers. You might want to compare the number of submissions and acceptances with the previous year (these numbers should also be in the proceedings and the final report). You should also explain the review process, e.g, how many PC members reviewed each paper and the organization of the PC meeting. You might want to talk about the geographic distribution of submissions and any unusual trends. The summary should take about 15 minutes.

After the Conference

Help the Conference Chair prepare the post-event report for SIGPLAN EC.

As former Program Chair, you may be invited to serve on the conference’s Steering Committee. For the formal rules for your particular conference’s steering committee, consult the relevant SIGPLAN conference web page. Links to these pages can be be found here.

The program chairs of the major SIGPLAN conferences (POPL, PLDI, OOPSLA, and ICFP) are automatically on the selection committee for the following year’s SIGPLAN Programming Languages Achievement Award.

Artifacts

ACM treats artifacts as “first-class” objects in the digital library, that is, each artifact has its own DOI, meta data, and landing page. AEC chairs should send a list of papers with accepted artifacts and their badge assignment to ConfPub. They will print the badges on the papers at a uniform position. ConfPub also collects the meta data from the authors and provides the data to ACM.

Auxiliary Material

The ACM Digital Library and ConfPub, who prepare many ACM proceedings, are both set up to accept auxiliary material. The SIGPLAN EC encourages SIGPLAN-sponsored conferences to accept such material as part of the publication process. (Some conferences also accept auxiliary material with submissions; this is an independent issue.)

Auxiliary material may include

  • a technical report with additional details
  • source code for software
  • proof scripts for automated theorem provers
  • test data

or anything else relevant. It is SIGPLAN policy to encourage authors to publish adequate auxiliary material to enable others to reproduce their work. Authors retain copyright of their auxiliary material, while copyright of the paper can be either assigned to ACM or held by the authors. Referees may examine auxiliary material, if it accompanies the submission, but this should not be considered part of the review process. For an example of auxiliary material in the Digital Library, see here and click on “Supplemental Material”.

Shepherding

Papers may be accepted conditionally, in which case, such papers require shepherds. The role of the shepherd is to compile a list of required changes to the paper that are required for the paper to be accepted, in consultation with all reviewers, and subject to the approval of the PCC. Authors are expected to modify their papers appropriately; in the event that the reviewers and the shepherds do not come to an agreement that a paper has been revised satisfactorily, the PCC should have the ultimate decision as to whether the paper will be accepted.

Additionally, the shepherds can ensure that outstanding work is not rejected on the basis of a problem that can easily be fixed with minor modifications to the paper.

Distinguished Papers

Up to 10% of the accepted papers may be designated for ACM SIGPLAN Distinguished Paper Awards.

  • Nominations may be solicited from the PC and ERC, and will also include the top 10% papers in terms of numeric scores. PC/ERC members should not nominate papers by authors they are conflicted with.
  • PC papers may be nominated.
  • The PC/ERC (excluding the authors of any nominated papers) should vote on this set of nominated papers. Each PC member may cast as many votes as the target number of Distinguished Papers. The usual conflict-of-interest rules will apply (e.g., PC/ERC members cannot vote on papers with which they have a conflict of interest).
  • The PCC and GC should decide which papers will receive the award. While the number of votes received by nominated papers is an important factor, the PCC and GC may exercise their judgment in making their decision.

PC Member Responsibilities

PC members must commit to reading all of their assigned papers and writing their own reviews. PC members may also suggest additional reviewers, but they should not subcontract reading or review writing duties to others. The PCC should be consulted before additional reviewers are contacted to avoid conflicts and to ensure topical balance.

If there is an in-person PC meeting, then PC members must attend the PC meeting in-person. If the meeting is virtual, PC members must participate in the manner described in the invitation to join the PC. Invited members who cannot commit to attending or participating in the meeting in person in the way the PC chair has specified should decline the invitation.

Recommendations:

The PCC should consider whether to explicitly allow PC members to share and discuss papers with their own students (subject to the usual confidentiality and COI provisions), and to incorporate information from such discussions into their review. If this is permitted, the PC member should still be responsible for writing the review. When a student is involved in this capacity, the PCC should be informed so that the student receives appropriate credit.

PC members should consult the SIGPLAN Republication Policy. If a related version of the paper appeared in a workshop, PC members should take into account whether its call for papers stated that publication in the workshop is not intended to preclude later publication.

Recommendations for Reviewing Process Management

Keeping the same reviewing form has benefit for authors and reviewers as they can more easily calibrate and interpret scores if the ratings are well understood. Nierstrasz’ Identify the Champion scoring process has stood the test of time. A typical review form in the spirit of this process has (at least) the following fields:

  • Paper score: –A: Good paper. I will champion it at the PC meeting. –B: OK paper, but I will not champion it. –C: Weak paper, though I will not fight strongly against it. –D: Serious problems. I will argue to reject this paper.
  • Reviewer expertise: –X: I am an expert in the subject area of this paper. –Y: I am knowledgeable in the area, though not an expert. –Z: I am not an expert. My evaluation is that of an informed outsider.
  • Paper summary (reviewer’s synopsis provides a useful sanity check)
  • Points in favor (encourage positive reviewing)
  • Points against (be clear about principal objections)
  • Detailed comments for authors (body of review)

Additionally:

  • It is useful for the PCC to acquaint themselves ahead of time with the conference management system (currently HotCRP), and the facilities it offers.
  • A two-phase reviewing process can be used, in order to avoid PC overload, by identifying weaker papers earlier.
  • PC members should be strongly encouraged to submit reviews as they are completed; this makes it easier for the PCC to monitor progress and identify problems early.
  • The PCC may wish to identify major reviewer disagreements and papers without sufficient expertise and seek second opinions where necessary to help resolve the disagreements.
  • The PCC should allow all reviewers to see decisions for the papers they have reviewed before decisions are publicly announced.

Important Email Addresses

SIGPLAN Chair: chair_sigplan@acm.org SIGPLAN Vice Chair: vc_sigplan@acm.org SIGPLAN Information Director: infodir_sigplan@acm.org

See Also

Some columns containing useful advice for conference organizers have been published in SIGPLAN Notices and elsewhere over the years: