Pipeline Creation Instructions

nf-core instructions

This page is heavily inspired by the nf-core page Adding a new pipeline.

Create the pipeline

All pipelines must use the nf-core template. This is done by using the nf-core pipelines create command - see the docs for detailed instructions. This tool does lots of things for you: it gives you the correct file structure and boiler plate code and also sets up the required git infrastructure for you to keep your pipeline in sync in the future.

The command has a user interface that will ask how you want to customise the template. Answer these:

• Choose pipeline type: choose "custom"
• GitHub organisation: enter sanger-tol
• Template features: disable:
  • Use reference genomes. This is only relevant when dealing exclusively with model organisms.
  • Use multiqc. If you're unsure whether you need it, or don't want to consider it a requirement of your pipeline (if you leave the option enabled, every template upgrade will try to update it).
  • Use fastqc. Same reasoning as multiqc above.

There are other options such as the Microsoft Teams notifications that we don't use ourselves but are harmless to keep (and our users may find those options useful).

Push to GitHub

Create an empty repository on GitHub for your new pipeline under https://github.com/sanger-tol. Do this by:

1. going to the GitHub website,
2. clicking + then New Repository,
3. selecting "sanger-tol" as the Owner.

Make sure not to initialise it with any file, README or LICENSE: you just want an empty repository. You already have these files generated from the nf-core template.

Leave the repository as "Public". We don't want to hide our pipelines, even when they're in progress.

Once created, copy the git URL and add this as a remote to your local git repository. The nf-core create command will have initialised a git repository for you, so all you need to do is add the remote:

## Add a remote called 'origin' - this is the default name for a primary remote
git remote add origin https://github.com/sanger-tol/PIPELINE_NAME.git
## Or the following if you have SSH keys configured on GitHub
git remote add origin git@github.com:sanger-tol/PIPELINE_NAME.git

The create command also generated the three standard nf-core branches (master, dev and TEMPLATE), together with an initial commit which is shared between them. This git structure is required for automatic template synchronisation in the future.

You first need to rename the master branch:

git branch -m master main

Then, you can push these new branches to the remote GitHub repository:

git push --all origin

You should now see the vanilla nf-core template and branches in the github.com web interface.

GitHub configuration

Head up to your repository on GitHub and do the following.

In the About section on the right, click on the cog wheel and:

1. Set the URL to https://pipelines.tol.sanger.ac.uk/$PIPELINE_NAME.
2. Add the topics pipeline and nextflow. This is required to enable it on the pipelines website.
   • Most pipelines also have workflow and genomics.
3. Enter a description.

Then, ask @muffato or @mcshane to:

1. Add the repository to the "nextflow_all" team with the "write" permission.
2. Add the repository to the "nextflow_admin" team with the "admin" permission.
3. Double-check that you're part of the "nextflow_all" team (and add you otherwise !).
4. Remove your individual access to the repository.
5. Allow your repository to access all the secrets from https://github.com/organizations/sanger-tol/settings/secrets/actions.

Finally, ask @gq1 or @muffato to:

1. Refresh the list of pipelines by clicking the link at the bottom of the pipelines page.
2. Set up the repository branch settings by selecting the pipeline and clicking "Fix data" on the pipeline health page.

Other bits

We're almost done. We now need to push some changes to the main branch to customise our pipeline a little further.

Copyright

We licence our pipelines with the MIT license. The MIT licence should already be in your repository, coming from the nf-core template, but we need to update the copyright statement to:

Copyright (c) 2025 Genome Research Ltd.

(or whichever year we're in !).

main vs master branch

Support for main is gradually coming in nf-core but we still need to change a few things:

• In .github/workflows/ci.yml, replace the occurrence of master with main.
• In .github/workflows/linting.yml, replace the two occurrences of master with main.

Logo

To add the sanger-tol logo to your pipeline, edit subworkflows/local/utils_nfcore_${PIPELINE_NAME}_pipeline/main.nf.

Add these two variables

     //
     // Validate parameters and generate parameter summary to stdout
     //
+
+    before_text = """
+-\033[2m----------------------------------------------------\033[0m-
+\033[0;34m   _____                               \033[0;32m _______   \033[0;31m _\033[0m
+\033[0;34m  / ____|                              \033[0;32m|__   __|  \033[0;31m| |\033[0m
+\033[0;34m | (___   __ _ _ __   __ _  ___ _ __ \033[0m ___ \033[0;32m| |\033[0;33m ___ \033[0;31m| |\033[0m
+\033[0;34m  \\___ \\ / _` | '_ \\ / _` |/ _ \\ '__|\033[0m|___|\033[0;32m| |\033[0;33m/ _ \\\033[0;31m| |\033[0m
+\033[0;34m  ____) | (_| | | | | (_| |  __/ |        \033[0;32m| |\033[0;33m (_) \033[0;31m| |____\033[0m
+\033[0;34m |_____/ \\__,_|_| |_|\\__, |\\___|_|        \033[0;32m|_|\033[0;33m\\___/\033[0;31m|______|\033[0m
+\033[0;34m                      __/ |\033[0m
+\033[0;34m                     |___/\033[0m
+\033[0;35m  ${workflow.manifest.name} ${workflow.manifest.version}\033[0m
+-\033[2m----------------------------------------------------\033[0m-
+        """
+    after_text = """${workflow.manifest.doi ? "\n* The pipeline\n" : ""}${workflow.manifest.doi.tokenize(",").collect { doi -> "    https://doi.org/${doi.trim().replace('https://doi.org/', '')}" }.j>
+* The nf-core framework
+    https://doi.org/10.1038/s41587-020-0439-x
+
+* Software dependencies
+    https://github.com/sanger-tol/readmapping/blob/main/CITATIONS.md
+"""
     command = "nextflow run ${workflow.manifest.name} -profile <docker/singularity/.../institute> --input samplesheet.csv --outdir <OUTDIR>"

     UTILS_NFSCHEMA_PLUGIN (

of course replacing readmapping with your actual pipeline name, and add this to use them

         help,
         help_full,
         show_hidden,
-        "",
-        "",
+        before_text,
+        after_text,
         command
     )

You should then see this in your terminal when running the pipeline:

Shared modules and sub-workflows

You will most likely need to include modules or sub-workflows from our shared modules repository alongside components from nf-core. Do the following changes to ensure your pipeline works seamlessly with both.

1. In .pre-commit-config.yaml, add extra lines to ignore sanger-tol modules and sub-workflows the same way nf-core ones are ignored:

   --- a/.pre-commit-config.yaml
   +++ b/.pre-commit-config.yaml
   @@ -15,6 +15,8 @@ repos:
                  .*ro-crate-metadata.json$|
                  modules/nf-core/.*|
                  subworkflows/nf-core/.*|
   +              modules/sanger-tol/.*|
   +              subworkflows/sanger-tol/.*|
                  .*\.snap$
            )$
         - id: end-of-file-fixer
   @@ -23,5 +25,7 @@ repos:
                  .*ro-crate-metadata.json$|
                  modules/nf-core/.*|
                  subworkflows/nf-core/.*|
   +              modules/sanger-tol/.*|
   +              subworkflows/sanger-tol/.*|
                  .*\.snap$
            )$

2. nf-test.config needs similar rules:

   --- a/nf-test.config
   +++ b/nf-test.config
   @@ -9,7 +9,7 @@ config {
      configFile "tests/nextflow.config"
   
      // ignore tests coming from the nf-core/modules repo
   -    ignore 'modules/nf-core/**/*', 'subworkflows/nf-core/**/*'
   +    ignore 'modules/nf-core/**/*', 'subworkflows/nf-core/**/*', 'modules/sanger-tol/**/*', 'subworkflows/sanger-tol/**/*'
   
      // run all test with defined profile(s) from the main nextflow.config
      profile "test"

   It also needs to refers to the version 0.0.7 or later of the nft-utils plugin (load "nft-utils@ line).

3. So does .gitattributes (to automatically collapse files changed in the sanger-tol components)

   --- a/.gitattributes
   +++ b/.gitattributes
   @@ -2,3 +2,5 @@
   *.nf.test linguist-language=nextflow
   modules/nf-core/** linguist-generated
   subworkflows/nf-core/** linguist-generated
   +modules/sanger-tol/** linguist-generated
   +subworkflows/sanger-tol/** linguist-generated

Tests

Finally, the nf-core template provides a way of defining the root path of the test data via the pipelines_testdata_base_path parameter. This defaults to https://raw.githubusercontent.com/nf-core/test-datasets/ for nf-core pipeline but since we have our own server space for test data, we can configure the Sanger base location instead.

Go to nextflow.config and tests/nextflow.config and change the value of pipelines_testdata_base_path from https://raw.githubusercontent.com/nf-core/test-datasets/ to https://tolit.cog.sanger.ac.uk/test-data/.

Zenodo

The repository needs to be integrated with Zenodo before making the first release. Better to do it now before anyone forgets ! Ask @muffato to enable the Zenodo integration.