Every submission is assigned a Senior Program Committee member (SPC) and at least three Program Committee members (PCs). Please note, we do not regard SPC members to be more "senior" than PC members; they just have different responsibilities. We will shuffle SPC and PC members over the coming years.

To ensure the highest quality of reviewing, we strongly recommend that SPCs/PCs adhere to the following guidelines when preparing their meta-reviews/reviews, which have been adapted from SIGIR 2023.

• As an SPC member, you are responsible for ensuring the overall quality of the review process. As reviews come in, you should encourage reviewers to provide constructive and informative reviews, flag any problems, and let the PC Chairs know if an extra emergency review is needed. After the review deadline, you must lead discussions for your assigned papers. Based on the discussion, you will write a meta-review, and make a final recommendation for acceptance or rejection to the PC Chairs. Note that you are expected to personally read and build your own opinion of every paper that is not a clear reject. More details on your tasks are included below.
• As a PC member, your job is to provide critical assessments of the papers assigned to you, based on the review form that we have prepared. For each paper assigned to you, your and your fellow reviewers’ feedback will form the basis for a discussion. Note that you must conduct each review personally. In other words, you should not delegate any of your reviews.

SPC Member Tasks

As an SPC member, you are responsible for monitoring the review process of PC members, conducting quality checks of the reviews, sending reminders to PC members when necessary, leading discussions in the discussion phase trying to reach a consensus, writing a meta-review, and making the final recommendation of accept/reject. Please inform us of any difficulties as soon as possible at any stage.

• Paper Review Management: You will be the main person to manage the papers assigned to you. Flag missing reviews close to the review deadline and inform PC chairs.
• Read Each Paper: Note that you are expected to personally read and build your own opinion of every paper that is not a clear reject.
• Review Quality Checking: You are in charge of ensuring that the review reports reach the expected quality. You should ask the PC members to rewrite or modify a review if its quality is insufficient.
• Discussions: You are responsible for initiating and leading the discussions as needed.
• Recommendation: You will write a meta-review, and make an accept/reject recommendation to the PC chairs. This also includes participating in the PC meeting to finalize decisions if necessary.

PC Member Tasks

As a PC member, your first task is to conduct objective and thorough reviews in a timely manner, answering all the questions included in the review form:

• Relevance: the relevance of this paper to SIGIR-AP
• Novelty: how original is the work compared to prior art
• Potential impact: the potential impact of the paper to the research community or the world
• Technical soundness: the quality of the technical content of this paper
• Presentation: the clarity of the presentation of this paper
• Reproducibility: whether researchers can reproduce the methods and results described in the paper with reasonable efforts
• Recognition of prior work: how the paper discusses and compares with related work
• Revision after SIGIR: how the authors have addressed the SIGIR reviewers' comments
• Overall evaluation: provide an overall score and a detailed review for the authors, including justifications for your scores and overall evaluation
• Reviewer’s confidence
• Best Paper candidate: whether this paper might be considered as a best paper candidate or a best student paper candidate
• Review forwarding to SIGIR. Let us know if you understand that, should this paper be rejected at this SIGIR-AP conference, your anonymous review may be forwarded to the next SIGIR conference. Reviewer identity will be strictly protected in this process.
• Confidential remarks for the program committee: any remarks intended only for PC members (optional)

You should assign a score for each of these questions. In the text area of “Overall evaluation”, supply a detailed rationale for these scores.

To ensure the quality of submissions, we recommend that authors observe the Guidance for Authors in preparing their papers. We kindly request PCs take into account this guidance when conducting your reviews. Additionally, we will remind you of other basic guiding principles for effective reviewing.

Your review is not just a vote for acceptance or rejection; it is an essential input to a discussion amongst the reviewers and the Program Chairs. You are assisting your fellow PC members, the SPCs, and the Program Chairs by providing arguments for or against acceptance. In some cases, there will be divergence among reviewers’ ratings of the paper; if you provide only a rating and a terse summary, without an adequate rationale, it will not be helpful.

Your review also serves as constructive input to the authors to help them understand the outcome of the review process and improve their work for future venues. Please be helpful and polite in your comments. The first author may be very junior or even a student – think about what kind of feedback would have been helpful when you were starting in the field.

Start your review with an assessment of what you consider to be the main contribution of the paper. Please do not just repeat what the authors say they did. You should provide your own summary of what you gained by reading the paper.

Whether you like or dislike a paper, please say so in a manner that is courteous and helpful to the authors, and informative to your SPC and Program Chairs. You will be asked to rewrite reviews that do not meet this expectation.

In the text area of “Overall evaluation”, please summarize your main points. It is important to point out weaknesses and validity issues, but it is equally important to identify the contribution of a submission. Ultimately, a submission’s acceptance depends on its novel contribution, not perfection. Note again that we are looking for an evaluation of the paper, not just a recommendation.

Now, let us elaborate on the reviewing criteria in detail.

Relevance: We ask the authors to identify one or more topics when submitting a paper. The Relevance criterion should not be evaluated solely with respect to the topics identified but with respect to the conference. We encourage the PC members to be open on this question: if the work is related broadly to IR, or it is potentially useful to IR (and the connection to IR is sufficiently established in the paper), then the paper should be considered to be within the scope of SIGIR-AP. If you find that a paper is outside of the scope of SIGIR-AP you can notify the PC Chairs so that we can consider a desk rejection.

Novelty: We encourage authors to take risks and tackle fresh and diverse research challenges that will interest our research community. Reviewers should give preference to an intriguing paper whose research might not be fully completed (as long as the style is good, the technical approach is sound and the main claims are supported), over a paper whose research is perfectly addressed but whose contribution is marginal.

Recognition of prior work: One issue that may arise is that authors miss some of the prior research that has been published in the area. This should be regarded as being a fatal flaw only if the missing work critically affects their conclusions; remember, authors will have the opportunity to make small editorial changes to their papers (including adding missing references) if they are accepted. If you do regard a paper as being unacceptable because of a lack of reference to prior work, you should supply sufficient detail about this prior research i.e., such as key references or descriptions that would enable the authors to explore this work, so that the authors can understand why you believe their paper should not be accepted. We encourage you to provide this level of detail for all references that you consider missing in the submission.

Revision after SIGIR: A novel feature of SIGIR-AP is that, although the review process is single-track (with a variable page limit), we have two paper types: regular and SIGIR-RR (Revise and Resubmit). Regular submissions are new, original contributions that have not been submitted elsewhere before. SIGIR-RR papers are a subset of resubmissions of papers that were not accepted at SIGIR 2023; the authors are required to explain how they addressed the reviewer comments to improve the paper. For some of these papers, the SIGIR-AP PC chairs will have access to the original anonymized reviews from SIGIR 2023, provided by the SIGIR 2023 PC chairs. The SIGIR-AP PC chairs will try to desk-reject SIGIR-RR submissions with clearly unsatisfactory explanations or revisions. While each regular paper will be reviewed by at least 3 Regular PC members, each SIGIR-RR paper will be reviewed by at least 2. During the online discussion, the Senior and Regular PC members will discuss how well the paper has been improved after SIGIR. Please be reminded that at highly competitive conferences such as SIGIR, some of the rejected papers may be just as good as the accepted ones; so, we will try to avoid negative bias. The idea is to avoid wasting the reviewer efforts of SIGIR.

Review forwarding to SIGIR. Please note that the anonymized SIGIR-AP 2023 reviews may be forwarded to the SIGIR 2024 PC chairs so that your efforts will be re-utilized there if authors of rejected SIGIR-AP papers choose to resubmit to SIGIR.

Variable paper length. Like recent ICTIRs, the paper length is variable (2-9 pages + references), and you are expected to assess for each paper whether the paper length is commensurate with contribution size. That is, for example, if a paper contains a good scientific contribution that is worth 2 pages, then that 2-page paper should be a candidate for acceptance; if a 9-page paper does not contain enough substance, then that paper is considered weak.

Reproducibility: This question is not about the use of proprietary data. It is about whether you think the authors provide sufficient details to reproduce the work. If the authors make their dataset and code available, this certainly helps to reproduce the results; but the reproducibility criterion is not limited to the sharing of data and code. Do you think that other researchers would be able to reproduce the method or results presented in the paper if they had access to the same or similar resources? Are the descriptions of the methods used detailed and accurate? Are all important technical details described? Given the resources used in the paper, or (if they are unavailable) similar resources, could researchers carry out similar experiments to verify the results? What further description could the authors provide?

Double submission or plagiarism: Pay special attention to this! If you are aware that the submitted paper is under review for another conference or journal, or is very similar to such a submission or published paper, please clearly put this in your comments. Program Chairs will double-check the case and sanctions will be imposed on any verified offending authors. Note that non-archival workshop papers and other non-archival publications such as arXiv are not considered double submissions or self-plagiarism.

Anonymity: The submissions should be anonymous. If the identity of the authors is revealed in the paper, please state it clearly in your comment. However, do not try to guess who the authors could be from evidence such as the dataset used in the experiments – using a dataset from a company does not necessarily mean that the authors come from the same company. So, mentioning a dataset or experimental environment from a company or academic institution is not considered a violation of the anonymity of submission. Similarly, it may be that the work has previously appeared in a non-archival form such as a workshop or an online repository; such prior publication is not prohibited and can be regarded as an unavoidable disclosure of identity, which we do not penalize.