Rice University CRC Workshop

Best Practices for Using HPC

Scaling Up to Cluster-Native Workflows

Bridging the gap between personal laptops and large-scale, hardware-accelerated supercomputing.

Recep Erol, PhD • Center for Research Computing (CRC)

Presenter Profile

Recep Erol, PhD

Computational Scientist for AI

🏛️ Center for Research Computing (CRC) • Rice University

📧 re51@rice.edu

🔗 linkedin.com/in/erolrecep

💻 github.com/erolrecep

🌐 erolrecep.github.io

We Are Your Partners in Research

An HPC cluster is not just a larger laptop—it is a shared community ecosystem.

🔬

Cutting-Edge Science

You push domain boundaries in molecular dynamics, AI/LLMs, genomics, and simulations.

🔄

New Paradigms

Cluster computing introduces batch queues, parallel storage, and hardware accelerators.

🤝

Active Collaboration

CRC engineers co-design pipelines with you before runs happen, turning queue delays into speedups.

Guiding Philosophy: Everyone needs help scaling up. Our goal is empowering you to write cluster-native code that runs faster, costs less in wait time, and reproduces seamlessly.

The 5 Pillars of Cluster Efficiency

What we will cover today

Pillar 1 The Mindset Shift

Shared resource ethics, interactive vs. batch paradigms, and login node etiquette.

Pillar 2 Data Logistics & Storage

High-speed VAST scratch vs. archives; migrating multi-TB datasets via Globus & rsync.

Pillar 3 Reproducible Environments

Containerizing workloads with Apptainer (Singularity) and automating with Makefiles.

Pillar 4 Slurm & The 3 Traps

Memory over-allocation, sequential job loops, and single-GPU lockups on multi-GPU nodes.

Pillar 5 Observability & CRC Support

Auditing jobs with seff, monitoring live GPUs with nvtop, and collaborating during office hours.

Pillar 1: Architecture

Laptop vs. Supercomputer

Moving from synchronous execution to orchestrated batch scheduling

💻 Personal Laptop / Workstation

Dedicated: You own 100% of memory and CPU cores.
Interactive: Click "Run", view output, hit Ctrl+C.
Unbounded I/O: Writing temp files to desktop is harmless.
Failure: If laptop sleeps or restarts, process dies.

⚡ Shared HPC Cluster

Shared: Hundreds of researchers compete for compute nodes.
Scheduled: Slurm orchestrates jobs by fair-share priority.
Parallel I/O: Shared network filesystem (VAST / GPFS).
Resilient: Detached batch jobs continue 24/7 uninterrupted.

💡 Key Takeaway: HPC is not an interactive playground—it is a production compute factory. Develop locally or interactively, then dispatch autonomous batch workloads.

Accessing HPC: The Golden Rule

Never execute computational workloads on login nodes!

🚫 Login Nodes (Gateway Only)

Strict Limits

Allowed:

Navigating files, editing scripts with nano/vim.
Submitting jobs (sbatch, squeue, scancel).
Light Git cloning and checking status.

Forbidden: Remote VSCode Server / Copilot indexing, heavy compiling, running simulations directly.

✅ Compute Nodes (The Workhorses)

Full Power

Interactive Debugging:

salloc --cpus-per-task=4 --mem=16G --time=02:00:00
OpenOnDemand (OOD): Web-based Jupyter & MATLAB with dedicated compute allocation.

Production Batch Jobs:

sbatch my_job.slurm (unattended, queued, logged).

Pillar 2: Data Transfer

Moving Data at Scale

Choose the right vehicle for your data payload

📁 Small to Medium (< 100 GB) Terminal

Tool: rsync or scp

# Efficient incremental synchronization
rsync -avzP --exclude '*.tmp' ./my_dataset/ \
  re51@range.rice.edu:/scratch/re51/data/

Resumes interrupted transfers (-P).
Preserves timestamps and permissions (-a).
Ideal for code, configs, and preprocessed subsets.

🌐 Massive Payloads (Multi-TB) High Throughput

Tool: Globus Transfer

GridFTP Engine: Maxes out campus 100GbE science DMZ.
Auto-Healing: Checksums every chunk and retries on dropped connections.
Background Execution: Does not require your laptop to remain open.
Direct endpoint-to-endpoint between Rice, TACC, AWS, and external universities.

Pillar 2: Storage Tiers

Storage Hygiene: Scratch vs. Archive

HPC storage is partitioned for specific I/O profiles

🏠 Home (~/home)

~20-50 GB

Quota Limited

For dotfiles, SSH keys, virtual environment definitions, and Git repositories. Never write simulation checkpoints here.

⚡ Scratch (/scratch)

VAST NVMe

High-Speed Parallel I/O

For active compute runs, checkpointing, and uncompressed datasets. Purge policy applies—clean intermediate files!

📦 Project / Archive

Lab Shares

Long-Term Storage

For published benchmark data and final analysis results. Not for hundreds of thousands of tiny temp cache files.

⚠️ The Million Tiny Files Trap: Storing 1,000,000 loose 10KB images degrades Lustre/VAST metadata servers. Always archive or bundle datasets (e.g. tar, HDF5, WebDataset).

Pillar 3: Reproducibility

Eliminate "Works on My Machine"

Package once with Apptainer (Singularity) • Run anywhere identically

📦 Why Containers on HPC?

No Root Needed: Apptainer executes in unprivileged user space.
Direct Hardware Binding: --nv automatically passes NVIDIA driver & CUDA libraries into container.
Host Filesystem Mounting: Access /scratch and /home seamlessly inside.
Reproducibility: One .sif file encapsulates Python 3.13, PyTorch, C libraries, and CUDA.

🚀 Example Slurm Execution

# Run PyTorch GPU script inside container
apptainer exec --nv \
  --bind /scratch/re51:/scratch \
  /scratch/re51/containers/pytorch_2.4.sif \
  python3 /scratch/re51/code/train.py \
    --data-dir /scratch/re51/dataset

Pair with a project Makefile so that make train executes the identical recipe locally and on RANGE.

Slurm Anti-Pattern #1

Trap 1: The Memory Overallocation Trap

Requesting 128 GB "just to be safe" inflates queue wait times for days

❌ over_allocation_bad_example.sh

#!/bin/bash
#SBATCH --job-name=mem_hog
#SBATCH --mem=128G 
# Job stays pending in queue for days for 
# resources it doesn't actually need.

python data_process.py

Slurm scheduler must drain nodes to carve out 128GB RAM.
Actual script memory peak: only 12.4 GB.
Wasted: 115.6 GB RAM locked away from peers.

✅ enough_amount_allocation.sh

#!/bin/bash
#SBATCH --job-name=mem_smart
#SBATCH --mem=16G 
# Job starts quickly. Validated using `seff` 
# from a previous run.

python data_process.py

Slurm instantly fits this job into available node fragments.
Queue wait drops from 48 hours to 3 minutes.
Verified with: seff <jobid> (85% memory efficiency).

Slurm Anti-Pattern #2

Trap 2: The Sequential Loop Trap

Running 100 independent simulations sequentially inside a huge multi-core allocation

❌ single_core_job_full_allocate.sh

#!/bin/bash
#SBATCH --job-name=bad_seq
#SBATCH --nodes=1
#SBATCH --ntasks=64
#SBATCH --time=48:00:00

# Wastes 63 cores if the script is single-threaded
python my_single_thread_script.py param1
python my_single_thread_script.py param2

Allocates 64 cores for 48 hours, but python executes on 1 core. 63 CPU cores idle at 0% load!

✅ single_core_job_slurm_jobarray.sh

#!/bin/bash
#SBATCH --job-name=good_array
#SBATCH --array=1-100%10
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --time=02:00:00

# Runs 100 tasks concurrently (max 10 at once)
echo "Running task ID: $SLURM_ARRAY_TASK_ID"
python my_single_thread_script.py --id $SLURM_ARRAY_TASK_ID

Parallel throughput: Slurm launches 10 concurrent worker tasks, dropping wall-clock runtime by 10x!

Slurm Anti-Pattern #3

Trap 3: The Multi-GPU Hoarding Trap

Requesting 4 H100 GPUs for a script that only addresses `cuda:0`

❌ wasting_gpu_resource.sh

#!/bin/bash
#SBATCH --job-name=gpu_waste
#SBATCH --gres=gpu:4
#SBATCH --time=24:00:00

# The Python script is only coded to use cuda:0.
# 3 H100 GPUs sit idle while others wait in queue.
python train_model.py

Locks \$120,000 worth of GPUs. 3 GPUs sit at 0% utilization while lab mates are stuck in PENDING status.

✅ correct_gpu_reseource_allocation.sh

#!/bin/bash
#SBATCH --job-name=gpu_optimized
#SBATCH --gres=gpu:1
#SBATCH --time=24:00:00

# Loads Apptainer container with PyTorch
apptainer exec --nv my_pytorch.sif python train_model.py

# (If using 4 GPUs, script MUST use 
# DistributedDataParallel or similar).

High priority queueing, 100% GPU saturation. If scaling to 4 GPUs, verify with torch.distributed first!

Pillar 5: Observability

Know Your Hardware Utilization

Audit running and finished jobs with native diagnostics

📊 Post-Mortem Audit: `seff`

Inspect real CPU & RAM efficiency after job finishes:

$ seff 4819201
Job ID: 4819201
State: COMPLETED (exit code 0)
Cores: 8
CPU Utilized: 02:45:12
CPU Efficiency: 89.2% of 03:05:12 core-walltime
Memory Utilized: 14.21 GB
Memory Efficiency: 88.8% of 16.00 GB

✅ Golden Rule: If Memory Efficiency < 30%, reduce --mem in your next run!

🔍 Live Monitoring During `salloc`

nvtop / nvidia-smi: Real-time GPU compute %, memory %, and PCIe bandwidth.
htop: Inspect per-core CPU utilization and thread affinity.
sacct -j <jobid> --format=JobID,Elapsed,MaxRSS,State

Always run a 1-epoch or 5-minute interactive test on compute nodes before launching a 72-hour batch job.

Folder Structure & Slurm Hygiene

Pair submitted job scripts with their output logs

📁 Recommended Project Blueprint

my_project/
├── Makefile               # Automated build & test targets
├── requirements.txt       # Environment lockfile
├── container.sif          # Apptainer image
├── src/                   # Python / C source modules
├── sbatch/                # Parameterized Slurm scripts
│   ├── 01_preprocess.slurm
│   └── 02_train_array.slurm
└── results/               # Job history logs
    ├── 01_prep_48210.out
    └── 01_prep_48210.err

🎯 Best Practices for Slurm Directives

# Auto-route standard output & errors with JobID
#SBATCH --output=results/%x_%j.out
#SBATCH --error=results/%x_%j.err

# Email notifications on failure or completion
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=re51@rice.edu

Never dump stdout into your root home directory.
%x expands to Job Name, %j expands to Job ID.

Let's Collaborate: We Are Here to Help!

Engage with CRC engineers during experimental design, not just when jobs fail

🚪

Open Office Hours

Every Tuesday:

10:00 AM – 12:00 PM
1:00 PM – 3:00 PM

Drop in with code, Slurm scripts, or architecture questions.

🎫

CRC Support Tickets

Submit questions anytime:

helpdesk@rice.edu
or Service Now portal

Tag CRC for cluster, GPU, and software support.

🤝

1-on-1 Consultations

Scaling up a grant or pipeline?

Dedicated Engineering

We help optimize MPI, PyTorch DDP, and Slurm arrays.

Portal: researchcomputing.rice.edu • Cluster Docs: kb.rice.edu

Questions & Discussion

Thank You!

Let's build scalable, reproducible, cluster-native science together.

Recep Erol, PhD

Computational Scientist for AI • Rice CRC

📧 re51@rice.edu

🔗 linkedin.com/in/erolrecep | github.com/erolrecep

🌐 erolrecep.github.io | researchcomputing.rice.edu

Best Practices for Using HPC

Scaling Up to Cluster-Native Workflows

Recep Erol, PhD

We Are Your Partners in Research

An HPC cluster is not just a larger laptop—it is a shared community ecosystem.

Cutting-Edge Science

New Paradigms

Active Collaboration

The 5 Pillars of Cluster Efficiency

What we will cover today

Laptop vs. Supercomputer

Moving from synchronous execution to orchestrated batch scheduling

💻 Personal Laptop / Workstation

⚡ Shared HPC Cluster

Accessing HPC: The Golden Rule

Never execute computational workloads on login nodes!

🚫 Login Nodes (Gateway Only)

✅ Compute Nodes (The Workhorses)

Moving Data at Scale

Choose the right vehicle for your data payload

Storage Hygiene: Scratch vs. Archive

HPC storage is partitioned for specific I/O profiles

🏠 Home (~/home)

⚡ Scratch (/scratch)

📦 Project / Archive

Eliminate "Works on My Machine"

Package once with Apptainer (Singularity) • Run anywhere identically

📦 Why Containers on HPC?

🚀 Example Slurm Execution

Trap 1: The Memory Overallocation Trap

Requesting 128 GB "just to be safe" inflates queue wait times for days

Trap 2: The Sequential Loop Trap

Running 100 independent simulations sequentially inside a huge multi-core allocation

Trap 3: The Multi-GPU Hoarding Trap

Requesting 4 H100 GPUs for a script that only addresses cuda:0

Know Your Hardware Utilization

Audit running and finished jobs with native diagnostics

📊 Post-Mortem Audit: seff

🔍 Live Monitoring During salloc

Folder Structure & Slurm Hygiene

Pair submitted job scripts with their output logs

📁 Recommended Project Blueprint

🎯 Best Practices for Slurm Directives

Let's Collaborate: We Are Here to Help!

Engage with CRC engineers during experimental design, not just when jobs fail

Open Office Hours

CRC Support Tickets

1-on-1 Consultations

Thank You!

Let's build scalable, reproducible, cluster-native science together.

Recep Erol, PhD

Requesting 4 H100 GPUs for a script that only addresses `cuda:0`

📊 Post-Mortem Audit: `seff`

🔍 Live Monitoring During `salloc`