hra-workflows

HRA workflows for annotating h5ad using different tools.

Table of Content

• Requirements
• Building docker images
• Running the annotation tools
  • Downloading models
  • Creating jobs
  • Running jobs
• Adding custom annotation tools
  • Folder structure
  • Updating top level pipeline

Requirements

• cwl-runner
• Docker

Building the docker images locally

Docker images can be build locally by running ./scripts/build-containers.sh. By default the script will build all containers when run. To build individual images or a set of images provide the container names as arguments to the build script, ex. ./scripts/build-containers.sh azimuth gene-expression.

Running the annotation tools

1. Download annotation tool models

Download model data by running cwl-runner download-models.cwl.

2. Create a job file

The first step is to create a job file that will specify inputs to the pipeline. The file can be written as either a json or yaml file.

Example job.yml running Azimuth

matrix:
  class: File
  path: path/to/data.h5ad
organ: UBERON:0002048 # Uberon id for lung
algorithms:
  # Algorithm specific options are documented in the container's options.yml
  - azimuth:
      referenceDataDir:
        class: Directory
        path: path/to/models/directory

3. Running the job

After creating a job file running the annotation tools is as simple as running cwl-runner pipeline.cwl my-job.yml (replace my-job.yml with your job file).

Running on Mac

Since container images are built for linux/amd64 only, Mac users must:

1. Ensure Docker Desktop is running with sufficient resources (8GB+ memory, 50GB+ disk)
2. Set the Docker platform environment variable:

   export DOCKER_DEFAULT_PLATFORM=linux/amd64

## Adding a new annotation tool

### 1. Tool folder structure
An annotation tool generally has the following file structure:

containers/ my-annotation-tool/ Dockerfile options.yml pipeline.cwl download-data.cwl (optional) context/* code and assets...

Where each file should perform the following function:
- `Dockerfile`
  - Instructions for building a docker image.
- `options.yml`
    - Cwl definition of tool specific options.
- `pipeline.cwl`
  - Main cwl pipeline for running the tool.
  - 3 inputs: "matrix", "organ", and "options"
  - 3 outputs:  "annotations", "annotated_matrix", and "report".
- `download-data.cwl` (optional)
  - Download models and other data required for running the tool.
  - Implement this pipeline when the model data is to large to embed directly in the docker image.
- `context/*`
  - Directory containing the code and assets implementing the tool.

### 2. Updating top level steps
After implementing a new algorithm a few changes have to be made to enable the tool from the main pipeline. The files that have to be updated are: `pipeline.cwl`, `./steps/annotate.cwl`, and `./steps/run-one.cwl`. After adding the new tool to the top level pipeline it can be used by specifying the tool in a job file.

#### Example job for a new annotation tool
```yaml
matrix:
  class: File
  path: path/to/data.h5ad
organ: UBERON:0002048 # Uberon id for lung
algorithms:
  - my-annotation-tool:
      # Options specific to my-annotation-tool
      option1: value1
      option2: value2
      ...