Prototype workflow RO-Crate from snakemake workflow

[douglowe]

Work coming from the BGE hackathon in Leiden. Reporting of products made should go in the report here: https://docs.google.com/document/d/1if6ukMKN3xHQHAwGEQPhhgvp7iQcFnauj4W1ZtIs8wk/edit

Aim is to write a python tool which will create a workflow RO-Crate from the outputs and reports created from a snakemake workflow.

Snakemake workflow used: https://github.com/o-william-white/skim2mt.git
Working code and example inputs: https://github.com/UoMResearchIT/ro-crate_snakemake_tooling

Definition of done:

• Have something where we return an object (probably not complete), preferably with information about which steps were run
• Documentation of what is missing and how to use it

[douglowe]

Workflow Run RO-Crate (RO-Crate profiles to capture the provenance of workflow runs): https://www.researchobject.org/workflow-run-crate/

[douglowe]

Working issue on development repository: UoMResearchIT/ro-crate_snakemake_tooling#1

[douglowe]

Change of plan - probably better to create test rocrate within snakemake source repo, and follow TTD principles to integrate this into snakemake.

See issue: eScienceLab/snakemake#1

[douglowe]

@douglowe and @alexhambley to work on definition of contents of the RO-Crates required from this tool

[stain]

@douglowe to discuss with Tim Booth. keep up to date with Felix. @douglowe keeping more of an oversight in sprint 3.
May be relevant for @alexhambley to include for Snakemake aspects in HDR UK.

[douglowe]

Following discussion with Tim Booth - we are intending to open a discussion with Snakemake developers in a Discussions channel in their GH organisation. Waiting for lead developer to return to do this (week starting 12th August).

[ianhinder]

Doug has been added to the discord chat. This is now waiting for people interested in RO-Crate in snakemake to decide what they want.

[stain]

Good progress in https://github.com/uomresearchit/ro-crate_snakemake_tooling and suggestions from Simone and Felix on how to progress.

[douglowe]

I've merged Tim Booth's changes (possibly there's more than we need here, as he notes in his 'not a PR' PR, but for the moment I want to add these in - we can trim back down later if needed), and added functions for creating the user record and workflow run properties. The latest changes are in the process-rocrate branch. This work isn't complete - but Oliver is working on Galaxy RO-Crate tooling, which has a lot of functionality overlap with this work, so I will pause this until he has had chance to progress that, and then we will compare the two toolsets to see what code can be reused.

Work will continue in issue #99

[prquinlan]

Hey, this is cool if this was created - I dont think we reported this, can you update back the main HDR board on progress - how far did this get?

[elichad]

@prquinlan unfortunately progress has stalled since the last updates above.

@OliverWoolland experimented with adding general Workflow Run Crate features into the runcrate library, with the intent to provide shared functions which could be utilised by Snakemake, Galaxy, and other workflow engines to build Workflow Run Crates. ResearchObject/runcrate#89

However, we ended up deprioritizing both that work and the direct Snakemake integration (https://github.com/UoMResearchIT/ro-crate_snakemake_tooling/tree/feat/process-rocrate) in favour of other development. We could revive it if someone in the Snakemake community becomes interested in driving it forward, or if it becomes a focus in Fed-A or other projects going forward.