Troubleshooting#

This page collects the most common setup and runtime issues seen during development.

First Checks#

Before debugging a workflow failure, run:

thesis info
thesis show-config default
thesis run -w hcp -p 114823 -c default --dry-run

These commands usually reveal missing tools, wrong paths, or invalid merged config.

Missing External Tools#

antsApplyTransforms not found#

  • Install ANTs (or conda install -c conda-forge ants) and make sure it is on PATH.

  • Verify with antsApplyTransforms --version.

  • Atlas ROI transformation in the HCP workflow depends on this binary.

probtrackx2 not found#

  • Install FSL and verify that the FSL binaries are on PATH.

  • Confirm with probtrackx2 --help.

  • GPU-enabled runs may also require a working probtrackx2_gpu or probtrackx2_gpu11.0 binary.

mri_synthseg not found#

  • mri_synthseg ships with FreeSurfer; it is not bundled with the Python package. The standalone SynthSeg distribution also provides it.

  • Use the synthseg workflow, tract_synthseg, or preprocess only after this binary is on PATH.

MRtrix3 binaries not found#

  • The mrtrix3 backend needs 5ttgen, dwi2response, dwi2fod, mtnormalise, tckgen, tcksift2, tckmap, and mrconvert from MRtrix3 on PATH.

  • thesis info only checks for FSL and ANTs, so it will not flag missing MRtrix3, FreeSurfer/SynthSeg, FSLeyes, or FireANTs — verify those manually.

Missing Input Files#

BedpostX samples are missing#

The HCP workflow expects merged thsamples, phsamples, and fsamples files.

Check:

  • hcp.bedpostx_dir

  • the subject input directory

  • whether the subject was preprocessed completely

Brain mask not found#

The default HCP path expects T1w/Diffusion/nodif_brain_mask.nii under the subject input directory. Override hcp.mask_path if your layout differs.

ROI file or label CSV not found#

  • Confirm the path relative to the project root or use an absolute path.

  • When label_name is used, label_file must exist and contain matching names.

  • When using integer labels directly, prefer label_values.

Config Problems#

Wrong patient-specific values are being used#

Remember the merge order:

  1. config/default.yaml

  2. config/hardware.yaml

  3. config/protocols/{protocol}.yaml

  4. config/patients/{patient_id}.yaml

  5. CLI overrides

Use thesis show-config default to inspect the merged output.

GPU scheduling behaves unexpectedly#

Check:

  • hardware.gpu_enabled

  • hardware.n_gpu_procs

  • hardware.n_gpus

  • nipype.plugin_args

When GPU mode is enabled, the executor injects scheduler hints so concurrent GPU node launches stay bounded.

Nipype Runtime Issues#

Workflow starts but produces little output#

  • Use -v for verbose mode.

  • Check the Nipype working directory from the run summary.

  • If progress bars are disabled automatically, confirm you are running in an interactive terminal.

A node fails and the CLI only shows a short message#

  • Re-run with -v.

  • Inspect the working directory or configured crash directory.

  • Reduce parallelism temporarily by lowering nipype.plugin_args.n_procs.

Batch runs fail inconsistently#

  • Reduce -j/--max-workers.

  • Lower Nipype n_procs or scheduler memory settings.

  • Check whether multiple subjects are competing for one GPU.

MRtrix3 Gotchas#

tckgen produces 0 streamlines with brainstem seeds#

Symptom: a mrtrix3 or tract_synthseg (mrtrix3 backend) run completes the FOD stages but tckgen reports zero streamlines, and tcksift2 then fails with a degenerate-input error.

Root cause: 5ttgen fsl returns a zero brainstem segmentation, so any seed dropped into the brainstem lands in unsegmented tissue and is rejected silently.

Fix: switch to a FreeSurfer-based 5TT.

tractography:
  fivett_algorithm: freesurfer    # or "hsvs"

SIFT2 weights are not cross-subject comparable without mu#

Symptom: summing raw SIFT2 weights across subjects produces wildly different totals for similar tracts.

Root cause: tcksift2 weights are only meaningful in combination with the per-subject mu scale factor (sum(weights) × mu / streamline_count defines Fibre Bundle Capacity).

The MRtrix3 workflow now captures mu automatically via tcksift2 -out_mu and writes mu × Σ weights into waytotal. If you ever re-process the output by hand, make sure to apply the same scaling, or use atlas.normalization_method: streamline_density which assumes the workflow-provided waytotal semantics.

dwi2mask legacy over-erodes deep WM#

Symptom: a mrtrix3 run produces a handful of streamlines (e.g. 6 out of 2,000,000 attempted seeds) because the brain mask contains holes in deep WM (cerebral peduncle, internal capsule).

Root cause: dwi2mask legacy erodes through low-FA regions.

Fix: reuse the FSL nodif_brain_mask.nii.gz (produced by preprocess) and dilate it once. This is the default in the shipped mrtrix3.yaml:

tractography:
  mask_source: fsl_nodif
  mask_dilate_voxels: 1

If you must use dwi2mask, prefer b02template over legacy and verify the mask visually (mrview dwi.mif -overlay.load mask.mif) before tracking.

Documentation and Developer Checks#

If you are changing the project during rapid development, verify docs and code health:

make check
make docs-strict

Workflow Preflight Errors#

Workflows declared with @requires(...) generate implicit existence checks that run before the Nipype graph is built. When thesis run aborts with a message like::

Required input 't1' not found at: data/...

the offending declaration is named in the error. Look in the workflow’s factory function for the matching @requires entry. Common fixes:

  • Override the path via the appropriate config key (each PatientFile lists its config_paths chain).

  • Add fallback_dirs=[...] if the file legitimately lives somewhere other than input_dir.

  • Pass optional=True when the workflow can run without that input.

Re-run with --dry-run to see all preflight errors at once without launching Nipype.

Developer / Extension Errors#

These failures show up when authoring or composing workflows rather than running a shipped one.

Connecting to a non-existent inputnode/outputnode field#

Symptom: a meta-workflow build raises a Nipype connection error naming a field that is not on the target’s inputnode / outputnode (e.g. 'roi_seeed' is not a valid input).

Root cause: meta-workflows wire contract-to-contract (upstream.outputnode.<field> -> downstream.inputnode.<field>). The field name must match a published contract field exactly.

Fix:

  • Look up the field tuples in thesis.core.contracts (e.g. ROI_OUTPUT_FIELDS) and use the exact spelling.

  • Confirm the sub-workflow actually publishes the field — find its attach_inputnode(...) / attach_outputnode(...) call (see src/thesis/workflows/full_pipeline/_core.py and the contracts section in docs/guides/nipype_integration.md).

  • Never connect to a sub-workflow’s internal node name; rewire through the contract node instead.

Workflow registered twice / name collision#

Symptom: a warning (or, under strict mode, a ValueError) that a workflow name is already registered.

Root cause: the module was imported twice, or two @workflow factories share a name.

Fix:

  • Keep one @workflow per file (the user-script loader enforces this — see docs/guides/custom_workflows.md).

  • Pick a unique name; user scripts may not shadow a built-in workflow name.

  • Set THESIS_STRICT_REGISTRY=1 to make duplicate registrations raise instead of warn, which surfaces accidental double-imports in tests immediately.

User-script config block rejected#

Symptom: config load fails with an unknown-key / validation error on your workflow’s YAML namespace.

Root cause: namespace blocks reject unknown keys, and the active schema does not declare the key you used.

Fix:

  • Check config_namespace / config_schema on the @workflow decorator. With no config_schema, only keys named in @requires config_paths are accepted (auto-derived as optional strings) — see the auto-derived schema section in docs/guides/custom_workflows.md.

  • For typed or extra keys, provide an explicit config_schema BaseConfig subclass that declares them.

Still Stuck?#

  • Inspect logs/ for loguru output.

  • Inspect configuration/index.md for config semantics and per-section references.

  • Inspect workflows.md for workflow expectations.

  • Run thesis info to confirm dependency detection.

  • Run thesis show-config <name> to inspect the merged config.

Documentation and Developer Checks#

If you are changing the project during rapid development, verify docs and code health:

make check
make docs-strict