Skip to contents

NOAA SEDNA Setup

Source: vignettes/NOAA-SEDNA.Rmd
NOAA-SEDNA.Rmd

How to use MitoPilot on the NOAA NMFS SEDNA computing cluster

You will need an account to access the SEDNA computing cluster. Detailed instructions can be found here. Or contact Krista Nichols () for more information.

First time setup

For first time setup, you will need to create a Mamba environment for the MitoPilot dependencies Nextflow and Singularity. We will also include a version of R in this environment; there are some issues installing the pipeline with the cluster’s R module.

Log in to the NOAA NMFS VPN, then log in to SEDNA. If you’ve never used mamba on SEDNA before, run the following.

/opt/bioinformatics/mambaforge/bin/mamba init

Let’s create the MitoPilot_deps mamba environment. This may take a while.

mamba create --name MitoPilot_deps 'bioconda::nextflow<26' conda-forge::singularity conda-forge::zlib 'conda-forge::r-base>=4.4' -y

Note: Please ensure your mamba environment is named MitoPilot_deps, case sensitive. Otherwise, RStudio server may not be able to access the required dependencies.

You can now call nextflow and singularity from anywhere on the cluster, as long as this MitoPilot_deps environment is activated.

Installing MitoPilot

Time to install MitoPilot! Activate your new mamba environment and launch R.

# activate the mamba environment you just created
mamba activate MitoPilot_deps

# launch R
R

In the new R session, run the following to install MitoPilot. It will take a while to install all of the necessary dependencies.

if (!requireNamespace("BiocManager", quietly = TRUE)) {
  install.packages("BiocManager")
}
BiocManager::install("Smithsonian/MitoPilot")

If install was successful, you can exit the R session using quit().

Updating MitoPilot

See updating, pinning, and rolling back. Note that some releases change the project database schema and require backwards_compatibility() before existing projects will open.

On SEDNA, activate the MitoPilot_deps environment first, and remember that a stale Singularity image will keep an updated package running old pipeline code.

Setting up RStudio server

Note: If you only need the MitoPilot GUI (not a full RStudio session), you can skip RStudio Server entirely and run the GUI headless over an SSH tunnel. See Headless GUI over an SSH tunnel in the Custom HPC vignette.

Next we need to set up RStudio server. The version of RStudio server must match the R version in your MitoPilot_deps mamba environment. You should be able to see the R version by running mamba activate MitoPilot_deps;mamba list.

In a SEDNA terminal, run the following. It may take a few minutes to download and set up the Singularity image file.

# Make a directory to host R-studio. 
mkdir -p ~/rstudio
cd ~/rstudio

# activate MitoPilot mamba environment
mamba activate MitoPilot_deps

# Pull R studio from singularity
# change version number if needed
singularity pull docker://rocker/rstudio:4.5.2

# additional setup
mkdir -p run var-lib-rstudio-server
printf 'provider=sqlite\ndirectory=/var/lib/rstudio-server\n' > database.conf

Now let’s make a helper script to launch RStudio server. Again, make sure you’re setting Rstudio_version correctly.

# set up bin directory if it doesn't exist
mkdir ~/bin 

# create helper script
cat > ~/bin/start-rstudio-server-MitoPilot <<'EOL'
#!/bin/bash
# script to start RStudio server 
# and print instructions on how to connect

# modify to match your RStudio server version
Rstudio_version="4.5.2"

cd ~/rstudio
source ~/.bashrc

mamba activate MitoPilot_deps

# Assign yourself a port
PORT=$(shuf -i 8000-9000 -n 1)

HOST=$( hostname )

GREEN='\033[0;32m' # green ANSI
RED='\033[0;31m' # red ANSI
NC='\033[0m' # no color ANSI

echo ""
echo -e "${GREEN}TO ACCESS RSTUDIO SERVER${NC}"
echo "In a terminal on your local system, run the following command:"
echo "ssh -N -L 8787:${HOST}:${PORT} ${USER}@sedna.nwfsc2.noaa.gov"
echo ""
echo "Enter your password when prompted"
echo "If successful, nothing will happen"
echo "Then open http://localhost:8787 on a local web browser"
echo "" 
echo -e "${RED}NOTE:${NC} This window and your local terminal session"
echo "must remain open in order to access the Rstudio server"
echo ""

# set Singularity temp dir, if needed
# if not set, Singularity will write to /tmp on the compute nodes
#mkdir -p "${HOME}/.singularity/temp"
#export SINGULARITY_TMPDIR="${HOME}/.singularity/temp"

singularity exec \
 --bind run:/run,var-lib-rstudio-server:/var/lib/rstudio-server,database.conf:/etc/rstudio/database.conf,${HOME}/.conda/envs/MitoPilot_deps/bin,${HOME}/.conda/envs/MitoPilot_deps/lib \
 --env APPEND_PATH="${HOME}/.conda/envs/MitoPilot_deps/bin:${HOME}/.conda/envs/MitoPilot_deps/lib" \
 rstudio_${Rstudio_version}.sif \
 rserver --www-address=0.0.0.0 --www-port=${PORT} --server-user=${USER}
EOL

# make script executable
chmod 755 ~/bin/start-rstudio-server-MitoPilot

Launching RStudio server

To launch RStudio server, first start an interactive session in SEDNA. You won’t need much computing resources, since MitoPilot uses Nextflow to distribute the analyses.

srun -c 2 --mem=16GB -p standard --pty /bin/bash

Then run the following.

mamba activate MitoPilot_deps
start-rstudio-server-MitoPilot

Follow the instructions to access your RStudio server session. They should look something like this.

TO ACCESS RSTUDIO SERVER
In a terminal on your local system, run the following command:
ssh -N -L 8787:node01.cluster:8377 dmacguigan@sedna.nwfsc2.noaa.gov

Enter your password when prompted
If successful, nothing will happen
Then open http://localhost:8787 on a local web browser

NOTE: This window and your local terminal session
must remain open in order to access the Rstudio server

Launching MitoPilot

Once you have opened the RStudio server session, run library(MitoPilot) to load the package. You should see a message about Nextflow if successful.

Want to learn how to use MitoPilot? Check out the MitoPilot workshop website.

Running MitoPilot Jobs

On SEDNA, run the pipeline as a batch job rather than in an interactive session. An interactive run needs you to hold the connection open for the whole workflow, which is fine for a handful of samples and risky for a few dozen or more.

MitoPilot builds the submission script for you. Click UPDATE, and the update window shows a ready-to-edit SLURM script pre-filled with #SBATCH directives. Add the SEDNA environment setup where the script indicates:

source ~/.bashrc
mamba activate MitoPilot_deps

Then use “Save Script Only” and submit the written .sh from a normal shell with sbatch. Do not use “Submit to Cluster” if you launched the app from a container, because sbatch is not available inside it. Your edits to the resource block are remembered per project, so the next run pre-fills them.

Monitor the job with squeue and the log files. When it finishes, relaunch the GUI to inspect the results. The same approach works for the annotate module. See Running the pipeline in the Custom HPC vignette for the full description of the submission window.