Reproducibility guidelines

From its very beginnings in 2018, MIDL has always been committed to open science and transparency. The reproducibility of research---especially the reproducibility of code, models and implemented experiments---is a crucial part of that mission.

However, the extra steps needed to ensure reproducible research remain unclear to most, and there are typically few incentives to perform this extra works. Until now, MIDL has not been an exception, and according to researchers from Umeå University in Sweden, only 22% of all MIDL papers are really reproducible.

With the aim to improve the reproducibility of our field, we are publishing those guidelines for authors, whom are kindly asked to fill this checklist when submitting a paper to MIDL. While not mandatory---not following those guidelines will not impact reviews---, we hope that this first step will already have an measurable impact, while at the same time start a conversation on the topic.

A free online academy was organized on the topic, on November 30 2022. Recording of the presentation is available here.

Is the reproducibility of the work addressed? [Yes, No]

Detail in the main body of the paper the publicly available materials from the submission. Is there published code with the submission to make reproducing the results easier? Are there publicly available datasets to train a similar model? Is the trained model public?

Is the code publicly available? [Yes, No]

Do property rights allow the authors to make their code publicly available? If so, include a link to the repository that stores the code used for the project. For code sharing, the most common option is a public Git repository, for instance on Github. If storing large trained models is an issue, Zenodo, AWS, OneDrive, Google Drive, Dropbox, Hugging Face, etc. are popular choices. Details about how to access the code should be contained in the main body of the paper, and not only in the paper submission form. The repository should be available long-term.

Are public datasets used? [Yes, No]

Address here, if the dataset was collected for the project and is not made publicly available. If the training dataset is private, aim to evaluate on public datasets for comparability. Alternatively, mention if there are similar publicly available datasets for reference.

Repository: Are the required package dependencies listed? [Yes/No/NA]

The packages that have been used to achieve the reported results, and their version numbers. Without exact version numbers, the repositories become more difficult to use and therefore lose their value over time.

Repository: Is the code for model training available? [Yes/No/NA]

Code for building the model with the exact same hyper-parameters and loss functions as reported in the paper, together with the training method.

Repository: Is the code for model evaluation available? [Yes/No/NA]

Code for evaluating the trained model with the metrics presented in the paper.

Repository: Is the presented trained model available? [Yes/No/NA]

If the trained model can be made publicly available, include the trained weights in the repository. Common formats are .pt for PyTorch, .h5 for TensorFlow, .pb for TensorFlow frozen graphs, or .onnx as an open format.

Repository: Is there documentation for the available material? [Yes/No/NA]

A thorough and detailed description of the repository can be a major help for the interested reader to fully understand the code. Aim to describe the methods in a similar way as in the main body of the paper for coherence.

Repository: Is there licensing for the available material? [Yes/No/NA]

A detailed description on how the shared material can be used for research and commercial purposes.