Medical Out-of-Distribution Analysis Challenge

Register for the Challenge

Go to <synapse.org> and click on Join.

Get access to the data

Go to <synapse.org> and go to the Data tab. In the mood folder, you can access the data.

We suggest the following folder structure (to work with our github examples):

mood/
--- brain/
------ brain_train/
------ toy/
------ toy_label/
--- abdom/
------ abdom_train/
------ toy/
------ toy_label/

Quick-Start

Join the Challenge, download the Data and have a look at our ready-to-go examples or last year’s submissions http://medicalood.dkfz.de/web/2020/#leaderboard.

Load the scans

In python you can load and write the nifti files using nibabel:

• Install nibabel: pip install nibabel

• Load the image data and affine matrix:

1
2
3
4
5
  import nibabel as nib

  nifti = nib.load(source_file)
  data_array = nifti.get_fdata()
  affine_matrix = nifti.affine 

• Save a array as nifti file:

1
2
3
4
  import nibabel as nib

  new_nifti = nib.Nifti1Image(data_array, affine=affine_matrix)
  nib.save(new_nifti, target_file) 

Build a docker

1. Requirements

Please install and use docker for submission: https://www.docker.com/get-started

You can build and use any docker base/ image you like. There are already good base docker images to build on for:

• Pytorch: https://hub.docker.com/r/pytorch/pytorch/
• Tensorflow https://www.tensorflow.org/install/docker

For GPU support you may need to install the NVIDIA Container Toolkit: https://github.com/NVIDIA/nvidia-docker

2. Docker Setup

For the different tasks the docker needs the following scripts (which accept parameter input_folder and output_folder):

• Sample-level:
  /workspace/run_sample_brain.sh input_folder output_folder (for the brain dataset)
  /workspace/run_sample_abdom.sh input_folder output_folder (for the abdominal dataset)

• Pixel-level:
  /workspace/run_pixel_brain.sh input_folder output_folder (for the brain dataset)
  /workspace/run_pixel_abdom.sh input_folder output_folder (for the abdominal dataset)

The docker has to allow mounting the input folder to /mnt/data and the output folder to /mnt/pred. We will mount the input and output folder and pass them to the run scripts. You will only have write access to /mnt/pred so please also use it for temporary results if needed.

During testing, the docker image will be run with the following commands:

• Sample-level:

  docker run --gpus all -v "input_dir/:/mnt/data" -v "output_dir:/mnt/pred" --read-only docker-image-name /workspace/run_sample_brain.sh /mnt/data /mnt/pred
  docker run --gpus all -v "input_dir/:/mnt/data" -v "output_dir:/mnt/pred" --read-only docker-image-name /workspace/run_sample_abdom.sh /mnt/data /mnt/pred
  

• Pixel-level:

  docker run --gpus all -v "input_dir/:/mnt/data" -v "output_dir:/mnt/pred" --read-only docker-image-name /workspace/run_pixel_brain.sh /mnt/data /mnt/pred
  docker run --gpus all -v "input_dir/:/mnt/data" -v "output_dir:/mnt/pred" --read-only docker-image-name /workspace/run_pixel_abdom.sh /mnt/data /mnt/pred
  

Your model should process all scans and write the outputs to the given output directory.

Please be aware of the different output formats depending on your task:

• Sample-level: For each input file (e.g. input1.nii.gz) create a txt file with the same name and an appended “.txt” file-ending (e.g. input1.nii.gz.txt) in the output directory and write out a single (float) score.
• Pixel-level: For each input file (e.g. input1.nii.gz) create a nifti file with the same dimensions and save it under the same (e.g. input1.nii.gz) in the output directory.

For more information have a look at our github example: https://github.com/MIC-DKFZ/mood

Test a docker locally

You can test you docker locally using the toy cases. After submitting your docker, we will also report the toy-test scores on the toy examples back to you, so you can check if your submission was successful and the scores match.

To run the docker locally you can:

1. Either run the docker manually and use the evalresults.py file (not recommended)
2. Or use the test_docker.py script (recommended):

Clone the github repo: git clone https://github.com/MIC-DKFZ/mood.git

Install the requirements: pip install -r requirements.txt

Run the test_docker.py script:

python test_docker.py -d docker_image_name -i /path/to/mood -t sample

where with -d you can pass your docker image name, with -i you pass the path to your base input directory which must contain a brain and an abdom folder (see above -> ‘Get access to the data’), and with -t you can define the task, either sample or pixel.

Test a docker on our submission system

You can test you docker on our submission system only for the toy cases. After submitting your docker, we will report the toy-test scores on the toy examples back to you, so you can check if your docker runs successful and the scores match with your local system.

To test your docker please submit to the tasks Toy Examples - Pixel-Level or Toy Examples - Sample-Level.

In the next Section on “Submit a docker” you can see how to submit a docker but submit to the tasks Toy Examples - Pixel-Level or Toy Examples - Sample-Level only.

Please not that this will not count as a MOOD 2020 submission. Submitting to the Toy Examples tasks will not increase you submission count and will not include your submission in the challenge.

Submit a docker

0. Becoming a certified Synapse user:
   Important: In order to use all docker functionality you must be a certified user : https://www.synapse.org/#!Quiz:

1. Create a new project on synapse:

   • To submit a docker file, you first need to create a new project on the synapse platform (e.g: MOOD_submission_<task>_<Your team name>).

   • Note the Synapse Project ID (e.g. syn20482334).

   • The organizing team of the challenge (Medical Out of Distribution Analysis Challenge 2020 Organizers) must be given download permissions to your project (Under Project Settings -> Project Sharing Settings ),

2. Upload docker-image to synapse:

1
2
3
   docker login docker.synapse.org
   docker tag docker_imagename docker.synapse.org/synapse_project_ID/docker_imagename
   docker push docker.synapse.org/synapse_project_ID/docker_imagename:latest 

3. Submit the docker image:
   
   • Got to your Project Page → Docker and click on the docker file .
   • Click on Docker Repository Tools → Submit Docker Repository to Challenge .
   • Choose the ‘tag’ you want to submit.
   • Choose the task you want to submit to (either ‘Sample-level’ or ‘Pixel-level’) (if you plan to participate in multiple tasks of the challenge, you need to submit your corresponding docker to each queue individually) .
   • Specify if you are entering alone or as a team.

Common Errors

• Read-only files system:

  OSError: [Errno 30] Read-only file system: '/workspace/XXXX
  

  Please make sure only to write your (temp) data to /mnt/pred and perhaps set ENV TMPDIR=/mnt/pred.

• Python site packages permission error:

  File "/opt/venv/lib/python3.X/site-packages/pkg_resources/__init__.py", line XXXX, in get_metadata
      with io.open(self.path, encoding='utf-8') as f:
  PermissionError: [Errno 13] Permission denied
  

  Please make sure you use virtual environments and use the virtualenv path as the first place to check for packages.

• Relative Paths:

  FileNotFoundError: [Errno 2] No such file or directory: './XXX'
  

  Please only use absolute paths, i.e. /workspace/XXX.

• Any other Errors ? Feel free to contact us =).

Submit the abstract

To offically take part in the challenge, the participants are required to provide a short abstract describing their submission. You can find the abstract templates on Synapse or Overleaf. Please fill out the provided template and send it as pdf to d.zimmerer@dkfz.de before the 07 September 2022.

Cite the Challenge

Please be sure to include the following citations in your work if you use the challenge data:

@article{zimmerer2022mood,
title={MOOD 2020: A public Benchmark for Out-of-Distribution Detection and Localization on medical Images},
author={Zimmerer, David and Full, Peter M and Isensee, Fabian and J{\"a}ger, Paul and Adler, Tim and Petersen, Jens and K{\"o}hler, Gregor and Ross, Tobias and Reinke, Annika and Kascenas, Antanas and others},
journal={IEEE Transactions on Medical Imaging},
volume={41},
number={10},
pages={2728--2738},
year={2022},
publisher={IEEE}
} 

For more information see https://zenodo.org/record/6362313.