The purpose of this document is to define the Review Process for the MICCAI 2024 conference. It is to be passed on each year, from successive MICCAI organizing committees, with the goal to refine and include feedback from the full set of future conference organising committees to create a continuity and consistency over the years. The document has been initiated, edited, and endorsed by the MICCAI 2017-2024 organizing committees. The MICCAI 20204 Program Committee has made some substantial changes to previous years, and in particular, updated the review process and criteria for early accept/reject as detailed below.
The policies set within are designed to continuously improve the review process, with the following objectives:
The goal of the MICCAI conference review process is to select the best papers in each submission category. This selection should be: fair, taking into consideration the specialized nature of our discipline and the size of our community; efficient, in not wasting valuable time, effort, and funding from our peers; and just, in relying on consensus of peer comments.
This summary is made public for transparency, so that all participants understand the review process and its implications on the final paper decisions.
The MICCAI 2024 review process will be overseen by the following individuals:
Marius George Linguraru
Qi Dou
Aasa Feragen
Matina Giannarou
Ben Glocker
Kitty Wong
Karim Lekadir
Julia Schnabel
The Program Committee of MICCAI 2024 consists of three bodies of participants that are essential to the review process:
The following documents supplement this guide, and will be made available on the MICCAI 2024 conference website:
The MICCAI Society maintains a database of past ACs and Reviewers. Top performing ACs and reviewers should be identified after each event. The MICCAI 2024 organizing committee will build on the list provided by earlier editions. As in previous years, MICCAI 2024 will use CMT to maintain this database. The information about the number of times a member has served as an AC will be kept indefinitely. The information on the performance of reviewers or ACs will be maintained in the system.
MICCAI 2024 will issue a call for participation in the Program Committee via the newsletter and mailshots. Individuals will submit relevant information regarding their past participation in MICCAI and similar conferences, as well as other biographical details.
Individuals will be chosen from the database, the received applications and other sources, and invited to join the MICCAI 2024 Program Committee, as ACs or Reviewers. When they apply, ACs and Reviewers will need to update their Toronto Paper Matching System entries, create/update their CMT profiles and associate themselves as being either MIC, CAI or MICCAI with additional information about their expertise in the Clinical Translation of Methodology and Health Equity. We encourage ACs and Reviewers to use their institutional email address. The MICCAI PCs will select the Program Committee using the following guidelines:
A webinar for ACs will be held to explain the review process, the AC role, responsibilities, tasks, and important dates.
A webinar for Reviewers will be held to explain the review process, what is expected from reviewers, and how to provide high-quality and constructive reviews.
The list of potential Reviewers is primarily based on the list of active reviewers of good standing from MICCAI 2019 to 2023.
In addition, all members of the MICCAI Society and all authors of MICCAI 2024 will be invited to apply to become a reviewer. Volunteering reviewers will be vetted by the PCs. Student reviewers must be at least enrolled at Ph.D. level or equivalent and have two or more published articles as first authors in a related field to qualify as reviewers. ACs will contribute to updating and expanding the list of reviewers.
Selected Reviewers will be invited to serve for MICCAI 2024 by accepting the following:
Authors will provide an intention to submit 14 days prior to the paper submission deadline. Authors must provide: i) a list of all co-authors and their affiliations and email addresses, ii) Google Scholar, Semantic Scholar and DBLP IDs if available and publication email addresses, as requested in the CMT User Profile iii) conflict of interest domains for all authors, iv) the title of the paper, and v) the abstract of the paper.
At least one co-author of the paper must review for MICCAI 2024. Authors will be asked to provide the name and email address of the co-author who will participate as a reviewer. Note that student reviewers are to be at least enrolled at a Ph.D. level.
All co-authors will be asked to fill out their CMT User Profile including Google Scholar, Semantic Scholar and DBLP IDs if available and publication email addresses. This information will be used to start building the assignment of papers to ACs and potential reviewers. Authors and co-authors should use their institutional email addresses whenever possible.
Authors will submit papers in CMT. The MICCAI Conference review process will be double-blind, i.e., the names of the authors will be hidden from the ACs and Reviewers, and the names of Reviewers and ACs will not be revealed to the authors. Papers must be fully anonymized before submission and a paper non-compliant with the anonymization or other paper submission rules (e.g., paper length) will be desk rejected. ACs and Reviewers will bring concerns about potential breaches to the attention of the Program Chairs.
Each paper must be submitted with Primary and Secondary areas selected from the CMT system. Authors also must identify to which stream the paper belongs, i.e., either MIC, CAI, MICCAI, Clinical Translation of Methodology, or MICCAI for Health Equity. These areas, the stream, and the paper itself, will be used to generate suggested reviewers using the automated TPMS paper matching system embedded in the CMT system (see Stages 6 and 7).
The goal of this step is to ascertain the breadth and depth of expertise required within the Program Committee and to increase the number of ACs when needed in response to the initial submissions from authors. This step will allow for the adjustment of the Program Committee composition based on the paper domains most represented at any conference edition.
Each paper will be assigned to an AC automatically by the CMT system, based on the TPMS and CMT subject areas, while balancing the load across ACs. Note that the AC remains blind to the paper authorship. The PCs will check the assignments to make sure all papers received an adequate assignment.
The essential role of the AC is to move the paper through the review process, up until the decision by the Program Committee for rebuttal papers. ACs will use their knowledge of the topic and of the appropriate reviewers to ensure the high quality, informative, and constructive reviews.
For MICCAI 2024, 20-25 papers will be allocated to each AC. Thus, for approximately 100 ACs, MICCAI 2024 is expecting to handle between 1600 and 2000 submitted papers. The number of ACs will be adjusted based on the number of submissions.
The goal of the paper allocation is to find the most appropriate reviewers in terms of expertise for a given paper. This step is achieved in two phases using the CMT system:
In phase 1, the CMT system will automatically provide a list of potential reviewers for each paper to the ACs. This ordered list will be generated based on keywords and TPMS scores. Using their expertise and judgement, the ACs will create a ranked list of 10-15 suggested reviewers for each paper, avoiding reviewers who already have more than 25 papers pre-assigned. In phase 2, reviewers will bid for papers by categorising them into "Eager to review”, "Willing to review”, "In a pinch”, "Not willing”. Reviewers will be asked to bid on all the papers for which the ACs suggested them as possible a reviewer.
The CMT system will then optimize matching of reviewers to papers based on the ranked list provided by the ACs, reviewer bids, TPMS scores, and keywords, while load balancing across all papers, reviewers and ACs. Three reviewers are assigned for each paper.
Note that the reviewers and ACs remain blind to the paper authorship at all times. The PCs will ensure ACs have suggested reviewers on time. The PCs and Platform Manager will check reviewer assignments and manually adjust where needed.
The goal of the review step is to provide high quality, informative, and constructive feedback for each submitted paper.
The reviewers will:
MICCAI 2024 will use a 6 point reviewing scale, associated with specific semantic meaning, as follows:
| Score | Semantic meaning |
|---|---|
| 6. strong accept | must be accepted due to excellence |
| 5. accept | should be accepted, independent of rebuttal |
| 4. weak accept | could be accepted, dependent on rebuttal |
| 3. weak reject | could be rejected, dependent on rebuttal |
| 2. reject | should be rejected, independent of rebuttal |
| 1. strong reject | must be rejected due major flaws |
The ACs will log in frequently to check review progress and monitor the quality of submitted reviews. ACs may ask Reviewers for a more detailed assessment, if needed. Should the ACs be unsatisfied with the quality of a review, and failing to get further feedback from the reviewer, then the ACs can ask for additional reviewer(s) to assess the paper, beyond the original three reviewers. Any reviewer who does not provide a quality review will be identified and logged in the reviewer database.
Continuing practice started in 2021, reviews and rebuttals of accepted papers will be made publicly available (without disclosing the reviewers names) on the MICCAI website. By reviewing for MICCAI, you give permission to publish your review.
In 2024, decisions for early acceptance and rejections will be primarily based on an algorithm taking into account the review scores, reviewers' expertise, and weighting. ACs will only perform a lightweight quality and sanity check, but are otherwise not involved in the early decisions. No additional feedback beyond the original reviews will be provided to authors.
We anticipate that after the initial review in this stage, 15-20% will be accepted outright; 40-50% will be rejected, and the remaining papers will be sent to the rebuttal stage.
Reviews and early decisions of the papers are sent to the authors. Authors of rebuttal papers will have one week to submit their response.
The goal of the rebuttal stage is to provide authors the opportunity to highlight possible misinterpretations or inaccuracies in the reviewers' assessments, and inform the ACs final recommendation. Rebuttals will be published along with the reviewer comments if the paper is accepted. Rebuttals should not include any new results, experiments, or data beyond what was provided at paper submission. Non-compliant rebuttals will lead to the desk rejection of the paper.
The original reviewers will be asked to read the rebuttal and reflect on their initial review. Reviewers should consider updating their reviews and scores in light of the authors' response.
All rebuttal papers will be assigned to two ACs (the original AC who handled the paper, and one secondary AC) based on TPMS matching and CMT subject areas, balancing load over ACs. Secondary AC assignment will be checked, and if necessary adjusted, by the PCs. After reviewers have considered the rebuttal, and potentially updated their reviews and scores, the two ACs read the reviews and the rebuttal, and, if necessary, the paper, and each independently make a recommendation for accept or reject. They may optionally write justifications for ‘accepts', but are expected to write a brief justification for ‘rejects'. The ACs will also rank their rebuttal papers, and make recommendations for orals, awards, session type for papers where they recommended acceptance.
The final decision for rebuttal papers is determined by the AC recommendations. If the two ACs agree in their recommendation, the paper is recommended for acceptance or rejection. If the two ACs disagree, a third independent (senior) AC reads the reviews, rebuttals, and the two AC justifications (where available) and makes a final recommendation.
Taking into account possible restrictions on the number of papers that can be accepted to the conference, in consultation with the Organizing Committee and the MICCAI Board, the PCs prepare the final list of accepted papers.
We will hold a set of dedicated teleconferences (in different timezones) between the MICCAI 2024 PCs and the ACs to inform about the overall results and statistics of the review process, to discuss and resolve any arising issues, and to collect feedback and recommendations for any further improvements of the process.
The goal of this step is to select the most suitable papers for oral presentation.
The number and format of the oral sessions will be determined by the Organizing Committee, in consultation with the MICCAI Board, and will reflect the overarching objectives of the conference and the themes of the submitted papers. The recommendations of the ACs will be taken into consideration when deciding which papers to select for full oral presentation. There will be an emphasis on providing Young Investigators with the opportunity to present and there will minimally be 1 CAI, 1 Clinical Translation, and 1 Health Equity oral session if the conference receives adequate submissions under each category.
Final decisions of oral versus poster presentations may be made and communicated up to two months after acceptance decision.
The goal of this step is to inform authors of rebuttal papers about the outcome of the review process.
The PCs will issue the following, via email, to all authors:
A complete list of acceptance and orals will be drafted for dissemination as a program. The PCs will record each complaint and follow up.