💻 Local Machine (macOS / Linux)

• Terminal with git and make installed.
• Python 3.9+ runtime available.
• Internet connection (for initial package downloads).
• Clone the repository:

  git clone \
    https://github.com/erolrecep/scientific_python_workflow_hpc.git
  cd scientific_python_workflow_hpc

☁️ Google Colab (Windows / No-Install)

• Zero local setup required! Ideal for Windows or Chromebook attendees.
• Free access to cloud Linux containers + NVIDIA Tesla T4 GPU.
• Open directly from your browser:
  
• Under Runtime ➔ Change runtime type ➔ Select T4 GPU.

💻 Local Laptop / Dev Node

• Interactive experimentation and REPL debugging.
• Small dataset slices (10 to 100 samples).
• Fail fast, iterate quickly, inspect intermediate tensors.
• Verify memory footprint before submitting jobs.

🏢 HPC Cluster / Compute Nodes

• Non-interactive batch execution via SLURM scheduler.
• Full dataset scale (10,000+ files, Terabytes of data).
• Zero GUI: everything driven by scripts and Makefiles.
• Golden Rule: NEVER run heavy computation on login nodes!

💥 Resolution Failures

Standard pip struggles to resolve complex compiled C/C++, CUDA, and Fortran extensions without compiler mismatches on CentOS/Rocky Linux nodes.

📁 Inode Quota Exhaustion

Installing full Anaconda/Miniforge extracts 30,000+ tiny files into $HOME. Shared Lustre/GPFS filesystems impose strict inode caps, locking user accounts.

🐌 Slow Dependency Solving

Traditional Conda SAT solvers take 10+ minutes to resolve dependency graphs, burning costly interactive allocation time.

⚡ Astral uv (Pure Python Speed)

• Single static binary written in Rust (~25 MB).
• 10-100x faster than standard pip/virtualenv.
• Installs in 1-2 seconds with zero system side effects.
• Global deduplicated hardlink cache.

uv venv .venv
uv pip install -r requirements.txt

🦎 micromamba (Compiled Binaries)

• Pure C++ standalone executable (~15 MB).
• Fast conda-forge resolution without heavy base environments.
• No shell pollution: does NOT hijack ~/.bashrc with conda init.
• Low inode footprint on cluster shared storage.

micromamba create -p ./.mamba-env \
  -c conda-forge python=3.10
./.mamba-env/bin/pip install -r requirements.txt

❌ The Trap: Shell Activation in Make

Every line in a Makefile runs inside an independent subshell. Sourced virtualenvs vanish instantly!

# ❌ THIS FAILS SILENTLY:
process:
	source .venv/bin/activate
	python src/process_images.py
# (python is now system python, not .venv!)

✅ The Solution: Direct Binary Invocation

Target the environment's python executable explicitly. Idempotent, robust, and cluster-proof:

PYTHON = .venv/bin/python

# ✅ THIS SUCCEEDS EVERY TIME:
process: env data
	$(PYTHON) src/process_images.py

1. Grayscale Conversion

Flattens 3-channel RGB image to 1-channel luminance:

Y = 0.299·R + 0.587·G + 0.114·B

Strips redundant color channels while preserving textural edges.

2. Spatial 2D Convolution

Custom edge-enhancing sharpening filter:

Emphasizes high-frequency gradients across flower petal boundaries.

3. Standardization (229x229)

Normalizes variable-sized images to fixed shape:

Target: (229, 229, 1) • uint8 [0, 255]

Strict input tensor dimension for downstream deep neural networks.

cores = int(os.environ.get(
    "SLURM_CPUS_PER_TASK",
    len(os.sched_getaffinity(0))
))

os.environ["OMP_NUM_THREADS"] = "1"
cv2.setNumThreads(1)
torch.set_num_threads(1)

# 1000 images / 16 cores
chunk_size = len(images) // workers

⚡ GPU Architecture Highlights

• Host-to-Device Streaming: Multi-threaded SIMD host decoding (cv2.imread) streams non-blocking tensors directly to GPU memory (MPS / CUDA).
• Batched 2D Convolutions: Sharpening filter computed simultaneously across all batch items via F.conv2d(batch, weight, padding=1).
• GPU Bilinear Resizing: Resizing to (229, 229) executes entirely on high-bandwidth VRAM via F.interpolate().

python -m memray run -o results/profile.bin script.py
python -m memray flamegraph results/profile.bin

scalene run --cpu-only -o results/profile.json script.py
scalene view results/profile.json

def test_opencv_processing():
    img = cv2.imread(SAMPLE_IMAGE)
    processed = process_opencv(img)
    assert processed.shape == (229, 229)
    assert processed.dtype == np.uint8

def test_multicore_consistency():
    seq = process_all_sequential(images[:10])
    par = process_all_parallel(images[:10], workers=4)
    assert np.allclose(seq, par)

$ make test
=== test session starts ===
collected 3 items

tests/test_pipeline.py::test_opencv_processing PASSED  [ 33%]
tests/test_pipeline.py::test_pillow_processing PASSED  [ 66%]
tests/test_pipeline.py::test_pytorch_processing PASSED [100%]

=== 3 passed in 0.42s ===

📚 Links & Repositories

• Workshop Repo:
  github.com/erolrecep/scientific_python_workflow_hpc
• Interactive Colab Notebook:
  scientific_python_workflow_colab.ipynb
• Technical Blog:
  erolrecep.github.io

Questions & Discussion

Open floor for questions on cluster scaling, profilers, or environment management.