API Reference

Sync Module

srunx.sync.rsync

Rsync-based file synchronization for remote SLURM servers.

logger module-attribute

logger = get_logger(__name__)

_ESCAPE_RUN_RE module-attribute

_ESCAPE_RUN_RE = compile('(?:\\\\#[0-7]{3})+')

_ESCAPE_GROUP_RE module-attribute

_ESCAPE_GROUP_RE = compile('\\\\#([0-7]{3})')

_MODE_RE module-attribute

_MODE_RE = compile('[0-7]{3}')

RsyncResult dataclass

RsyncResult(returncode: int, stdout: str, stderr: str)

Result of an rsync operation.

RsyncClient

RsyncClient(
    hostname: str,
    username: str,
    port: int = 22,
    key_filename: str | None = None,
    proxy_jump: str | None = None,
    ssh_config_path: str | None = None,
    exclude_patterns: Sequence[str] | None = None,
)

Rsync wrapper for syncing files to/from remote SLURM servers.

Handles SSH connection options (port, key, ProxyJump, ssh_config) and builds rsync commands with sensible defaults for development workflow synchronization.

Source code in src/srunx/sync/rsync.py

def __init__(
    self,
    hostname: str,
    username: str,
    port: int = 22,
    key_filename: str | None = None,
    proxy_jump: str | None = None,
    ssh_config_path: str | None = None,
    exclude_patterns: Sequence[str] | None = None,
) -> None:
    rsync_path = shutil.which("rsync")
    if rsync_path is None:
        raise RuntimeError(
            "rsync is not installed or not found in PATH. "
            "Please install rsync before using RsyncClient."
        )

    self.hostname = hostname
    self.username = username
    self.port = port
    self.key_filename = key_filename
    self.proxy_jump = proxy_jump
    self.ssh_config_path = ssh_config_path

    # Detect rsync capabilities
    self._supports_protect_args = False
    self._supports_mkpath = False
    self._detect_rsync_capabilities(rsync_path)

    # Merge caller-supplied excludes with defaults (no duplicates)
    self.exclude_patterns = list(self.DEFAULT_EXCLUDES)
    if exclude_patterns:
        seen = set(self.exclude_patterns)
        for pattern in exclude_patterns:
            if pattern not in seen:
                self.exclude_patterns.append(pattern)
                seen.add(pattern)

push

push(
    local_path: str | Path,
    remote_path: str | None = None,
    *,
    delete: bool = False,
    dry_run: bool = False,
    itemize: bool = False,
    verbose: bool = False,
    max_delete: int | None = None,
    exclude_patterns: Sequence[str] | None = None,
) -> RsyncResult

Sync a local directory/file to the remote server.

Parameters:

Name	Type	Description	Default
local_path	str | Path	Local file or directory to push.	required
remote_path	str | None	Destination path on the remote server. If None, uses get_default_remote_path().	None
delete	bool	Remove remote files not present locally. Defaults to False: a push that silently prunes remote-only files (training checkpoints, run logs) is a data-loss footgun, and every historical incident here came from a caller inheriting a mirror-by-default. Mirror semantics are opt-in — callers that genuinely want them pass delete=True explicitly.	False
max_delete	int | None	Blast-radius cap for a mirror — not an atomic refusal. rsync deletes up to this many entries, skips the remaining deletions, finishes transferring, and only then exits 25, so on a cap hit the destination has already changed (verified against openrsync). A caller that needs "refuse without touching anything" must count deletions in a separate dry run and decide before calling; the MCP :func:~srunx.mcp.tools.sync.sync_files tool does exactly that. Only meaningful with delete=True. Must be >= 1 — see :class:ValueError below.	None
dry_run	bool	Perform a trial run that changes nothing on the remote — no transfers, no deletions, and no directory creation. Note that rsync without --mkpath cannot evaluate a transfer against a missing destination parent, so a preview of a not-yet-created destination fails rather than reporting an empty diff.	False
itemize	bool	Add --itemize-changes so the result lists every file rsync would (or did) touch, with the standard YXcstpoguax flag prefix. Required for dry_run callers that want a human-readable preview.	False
verbose	bool	Stream rsync's per-file progress to stderr live instead of capturing it silently. Adds --info=progress2 so users with large mounts see progress instead of a frozen terminal.	False
exclude_patterns	Sequence[str] | None	Additional exclude patterns for this call only.	None

Returns:

Type	Description
RsyncResult	RsyncResult with returncode, stdout, and stderr.

Raises:

Type	Description
ValueError	If max_delete is 0 or negative. Zero cannot be forwarded safely (rsync 2.6.x / openrsync read --max-delete=0 as unlimited, inverting the strictest cap into no cap at all), and silently downgrading the call to delete=False would be worse: the caller asked for a mirror that refuses rather than one that quietly leaves extra destination files behind. To forbid deletion, pass delete=False explicitly.

Source code in src/srunx/sync/rsync.py

def push(
    self,
    local_path: str | Path,
    remote_path: str | None = None,
    *,
    delete: bool = False,
    dry_run: bool = False,
    itemize: bool = False,
    verbose: bool = False,
    max_delete: int | None = None,
    exclude_patterns: Sequence[str] | None = None,
) -> RsyncResult:
    """Sync a local directory/file to the remote server.

    Args:
        local_path: Local file or directory to push.
        remote_path: Destination path on the remote server.
            If None, uses ``get_default_remote_path()``.
        delete: Remove remote files not present locally. **Defaults to
            False**: a push that silently prunes remote-only files
            (training checkpoints, run logs) is a data-loss footgun, and
            every historical incident here came from a caller inheriting
            a mirror-by-default. Mirror semantics are opt-in — callers
            that genuinely want them pass ``delete=True`` explicitly.
        max_delete: Blast-radius cap for a mirror — **not** an atomic
            refusal. rsync deletes up to this many entries, skips the
            remaining deletions, finishes transferring, and only then exits
            25, so on a cap hit the destination *has already changed*
            (verified against openrsync). A caller that needs "refuse
            without touching anything" must count deletions in a separate
            dry run and decide before calling; the MCP
            :func:`~srunx.mcp.tools.sync.sync_files` tool does exactly
            that. Only meaningful with ``delete=True``. Must be >= 1 —
            see :class:`ValueError` below.
        dry_run: Perform a trial run that changes nothing on the remote —
            no transfers, no deletions, and no directory creation. Note
            that rsync without ``--mkpath`` cannot evaluate a transfer
            against a missing destination parent, so a preview of a
            not-yet-created destination fails rather than reporting an
            empty diff.
        itemize: Add ``--itemize-changes`` so the result lists every
            file rsync *would* (or did) touch, with the standard
            ``YXcstpoguax`` flag prefix. Required for ``dry_run``
            callers that want a human-readable preview.
        verbose: Stream rsync's per-file progress to stderr live
            instead of capturing it silently. Adds
            ``--info=progress2`` so users with large mounts see
            progress instead of a frozen terminal.
        exclude_patterns: Additional exclude patterns for this call only.

    Returns:
        RsyncResult with returncode, stdout, and stderr.

    Raises:
        ValueError: If ``max_delete`` is 0 or negative. Zero cannot be
            forwarded safely (rsync 2.6.x / openrsync read
            ``--max-delete=0`` as *unlimited*, inverting the strictest cap
            into no cap at all), and silently downgrading the call to
            ``delete=False`` would be worse: the caller asked for a mirror
            that refuses rather than one that quietly leaves extra
            destination files behind. To forbid deletion, pass
            ``delete=False`` explicitly.
    """
    if max_delete is not None and max_delete < 1:
        raise ValueError(
            f"max_delete must be >= 1, got {max_delete}. To forbid "
            "deletion entirely, pass delete=False — 0 cannot be forwarded "
            "to rsync safely (2.6.x / openrsync read --max-delete=0 as "
            "unlimited), and quietly dropping --delete instead would turn "
            "a mirror that should refuse into one that silently leaves "
            "extra destination files in place."
        )

    if remote_path is None:
        remote_path = self.get_default_remote_path(local_path)

    local = Path(local_path)
    src = str(local)
    # Trailing slash ensures rsync copies directory *contents*, not the
    # directory itself.
    if local.is_dir() and not src.endswith("/"):
        src += "/"

    dst = self._format_remote(remote_path)

    # Ensure remote directory exists when --mkpath is unavailable.
    #
    # Deliberately restricted to real runs. A dry run must not touch the
    # remote at all: otherwise "nothing was changed" stops being true, and
    # a preview whose destination is a *file* path would create a
    # directory exactly where that file belongs. The cost is that a first
    # mirror against a not-yet-existing destination fails its preflight —
    # a safe, explicit failure that ``sync_files`` explains how to work
    # around, which is a better trade than a preview with side effects.
    if not self._supports_mkpath and not dry_run:
        # A file destination needs its *parent* created. ``mkdir -p`` on
        # the file path itself would put a directory where the file goes,
        # after which rsync can never write it.
        if local.is_dir() or remote_path.endswith("/"):
            self._ensure_remote_dir(remote_path)
        else:
            parent = str(PurePosixPath(remote_path).parent)
            if parent not in (".", "/", ""):
                self._ensure_remote_dir(parent)

    excludes = self._merge_excludes(exclude_patterns)
    cmd = self._build_rsync_cmd(
        src,
        dst,
        delete=delete,
        dry_run=dry_run,
        itemize=itemize,
        verbose=verbose,
        max_delete=max_delete,
        excludes=excludes,
    )
    if verbose:
        return self._run_rsync_streaming(cmd)
    return self._run_rsync(cmd)

pull

pull(
    remote_path: str,
    local_path: str | Path,
    *,
    delete: bool = False,
    dry_run: bool = False,
    itemize: bool = False,
    exclude_patterns: Sequence[str] | None = None,
) -> RsyncResult

Sync a remote directory/file to the local machine.

Parameters:

Name	Type	Description	Default
remote_path	str	Source path on the remote server.	required
local_path	str | Path	Local destination path.	required
delete	bool	Remove local files not present on the remote (default False).	False
dry_run	bool	Perform a trial run with no changes made.	False
itemize	bool	Add --itemize-changes so the result enumerates every file rsync would (or did) touch.	False
exclude_patterns	Sequence[str] | None	Additional exclude patterns for this call only.	None

Returns:

Type	Description
RsyncResult	RsyncResult with returncode, stdout, and stderr.

Source code in src/srunx/sync/rsync.py

def pull(
    self,
    remote_path: str,
    local_path: str | Path,
    *,
    delete: bool = False,
    dry_run: bool = False,
    itemize: bool = False,
    exclude_patterns: Sequence[str] | None = None,
) -> RsyncResult:
    """Sync a remote directory/file to the local machine.

    Args:
        remote_path: Source path on the remote server.
        local_path: Local destination path.
        delete: Remove local files not present on the remote (default False).
        dry_run: Perform a trial run with no changes made.
        itemize: Add ``--itemize-changes`` so the result enumerates
            every file rsync *would* (or did) touch.
        exclude_patterns: Additional exclude patterns for this call only.

    Returns:
        RsyncResult with returncode, stdout, and stderr.
    """
    src = self._format_remote(remote_path)
    dst = str(local_path)

    excludes = self._merge_excludes(exclude_patterns)
    cmd = self._build_rsync_cmd(
        src,
        dst,
        delete=delete,
        dry_run=dry_run,
        itemize=itemize,
        excludes=excludes,
    )
    return self._run_rsync(cmd)

read_remote_file

read_remote_file(
    remote_path: str, *, require_owned: bool = False
) -> str | None

Return the remote file's contents, or None if it doesn't exist.

require_owned=True additionally refuses a file belonging to another account. Worth the extra check wherever the contents drive an action — the upload manifest decides which paths a user is told are safe to delete, so a file another account planted there must not be believed. The ownership marker leaves it off: it is advisory, and refusing to read a foreign one would turn a warning into a hard failure.

The check narrows that window rather than closing it: the ownership test and the read are two pathname lookups, so a peer able to write the directory could swap the file in between. Closing it needs the owner verified on the same descriptor that is read, which a shell cannot express — it would mean driving this over SFTP rather than ssh.

Used by the per-machine ownership marker (#137 part 4) to read .srunx-owner.json before each sync. The check needs to distinguish "file missing" (legitimate first sync, returns None) from "ssh / network failed" (raise so the caller knows the marker can't be trusted).

Implementation: ssh ... cat -- <path> with a per-file existence test wrapped in a single shell command — keeps the round-trip count to one per check.

Source code in src/srunx/sync/rsync.py

def read_remote_file(
    self, remote_path: str, *, require_owned: bool = False
) -> str | None:
    """Return the remote file's contents, or ``None`` if it doesn't exist.

    ``require_owned=True`` additionally refuses a file belonging to another
    account. Worth the extra check wherever the contents drive an action —
    the upload manifest decides which paths a user is told are safe to
    delete, so a file another account planted there must not be believed.
    The ownership marker leaves it off: it is advisory, and refusing to read
    a foreign one would turn a warning into a hard failure.

    The check narrows that window rather than closing it: the ownership test
    and the read are two pathname lookups, so a peer able to write the
    directory could swap the file in between. Closing it needs the owner
    verified on the same descriptor that is read, which a shell cannot
    express — it would mean driving this over SFTP rather than ssh.

    Used by the per-machine ownership marker (#137 part 4) to read
    ``.srunx-owner.json`` before each sync. The check needs to
    distinguish "file missing" (legitimate first sync, returns
    ``None``) from "ssh / network failed" (raise so the caller
    knows the marker can't be trusted).

    Implementation: ``ssh ... cat -- <path>`` with a per-file
    existence test wrapped in a single shell command — keeps the
    round-trip count to one per check.
    """
    # A dedicated exit code for "missing", so it is never confused with a
    # failed read. ``test -f X && cat X`` cannot do that: ``cat`` also exits
    # 1 when it cannot open the file, so an unreadable file (bad
    # permissions, an I/O error) looked exactly like an absent one. For the
    # ownership marker that only meant a lost warning, but the upload
    # manifest treats "missing" as a normal first run and rebuilds from
    # scratch — turning an unreadable record into a confident, wrong "clean"
    # report.
    quoted = shlex.quote(remote_path)
    owner_check = (
        # A peer able to write the mount root can also drop in a perfectly
        # ordinary file. Refusing symlinks is not enough when the contents
        # are acted on: a forged upload record becomes a list of paths the
        # user is told are safe to delete. They can only create files owned
        # by themselves, so a uid match rules it out.
        f"if [ \"$(ls -ldn {quoted} | awk 'NR==1{{print $3}}')\" "
        f'!= "$(id -u)" ]; then exit {self._READ_FOREIGN_EXIT}; fi; '
        if require_owned
        else ""
    )
    result = self._ssh_run(
        # Refuse a symlink here as the writer does. Without the check the
        # two halves disagree: writing to a planted link is rejected, but
        # reading through one is trusted, so a peer able to write the mount
        # root can serve arbitrary content as srunx's own control file.
        # ``test -f`` alone follows the link.
        f"if [ -h {quoted} ]; then exit {self._READ_SYMLINK_EXIT}; fi; "
        f"if [ ! -f {quoted} ]; then exit {self._READ_MISSING_EXIT}; fi; "
        f"{owner_check}"
        f"cat -- {quoted}"
    )
    if result.returncode == 0:
        return result.stdout
    if result.returncode == self._READ_MISSING_EXIT:
        return None
    if result.returncode == self._READ_SYMLINK_EXIT:
        raise RuntimeError(
            f"refusing to read {remote_path!r}: it is a symlink, so its "
            "contents are whatever it points at rather than srunx's own "
            "control file"
        )
    if result.returncode == self._READ_FOREIGN_EXIT:
        raise RuntimeError(
            f"refusing to read {remote_path!r}: it belongs to another "
            "account, so it is not a control file srunx wrote and its "
            "contents cannot be acted on"
        )
    raise RuntimeError(
        f"ssh read of {remote_path!r} failed (exit {result.returncode}): "
        f"{result.stderr.strip()}"
    )

write_remote_file

write_remote_file(
    remote_path: str, content: str, *, mode: str = "644"
) -> None

Write content to remote_path atomically (temp file + rename).

mode defaults to world-readable, which the ownership marker needs: its whole job is telling another account that this mount is in use, and a marker they cannot read is a marker that does not work. Anything only its own writer reads should pass 600 — the upload manifest does, since it enumerates a project's file names and the workstation that pushed them, and a mount root is often traversable even when the directories under it are not.

mv within one directory is a rename(2), so a concurrent reader sees either the previous file or the complete new one. Writing with tee straight at the target does not give that: tee truncates first, so a reader in that window sees an empty file. This function used to do exactly that while documenting the opposite.

The target is rejected when it is a symlink (tee / mv would act on whatever it points at, letting a planted link redirect the write outside the mount) or a directory (mv would move the new file inside it and exit 0, reporting success while the control file stays unreadable). A probe that cannot be completed is also an error, since its silence proves nothing.

Those checks are made twice — once up front and once immediately before the rename — and the result is verified afterwards. GNU coreutils can refuse a directory destination outright (mv -T) and that path is taken when available; elsewhere a swap in the instant before the rename is detected rather than prevented, and never reported as success.

Known limit: a peer could point our temp directory's name at a different directory we own, which passes the ownership check. That costs nothing (they cannot reach inside a directory of ours) beyond the publish possibly crossing filesystems. Closing it would need fd-relative operations, which a shell cannot express — it would mean driving this over SFTP instead of ssh.

The parent directory is assumed to exist (for the owner-marker case the rsync that just ran guarantees it).

Raises:

Type	Description
ValueError	If remote_path is login-relative. The write anchors its working directory inside a private temp directory, which would change how such a path resolves. Also if mode is not three octal digits — it reaches a remote shell unquoted.
RuntimeError	If the target is a symlink or a directory, if the target could not be probed, or if any write / publish step exits non-zero — so the caller surfaces the failure instead of silently leaving a stale file behind.

Source code in src/srunx/sync/rsync.py

def write_remote_file(
    self, remote_path: str, content: str, *, mode: str = "644"
) -> None:
    """Write *content* to *remote_path* atomically (temp file + rename).

    ``mode`` defaults to world-readable, which the ownership marker needs:
    its whole job is telling *another* account that this mount is in use,
    and a marker they cannot read is a marker that does not work. Anything
    only its own writer reads should pass ``600`` — the upload manifest
    does, since it enumerates a project's file names and the workstation
    that pushed them, and a mount root is often traversable even when the
    directories under it are not.

    ``mv`` within one directory is a ``rename(2)``, so a concurrent reader
    sees either the previous file or the complete new one. Writing with
    ``tee`` straight at the target does **not** give that: ``tee``
    truncates first, so a reader in that window sees an empty file. This
    function used to do exactly that while documenting the opposite.

    The target is rejected when it is a symlink (``tee`` / ``mv`` would act
    on whatever it points at, letting a planted link redirect the write
    outside the mount) or a directory (``mv`` would move the new file
    *inside* it and exit 0, reporting success while the control file stays
    unreadable). A probe that cannot be completed is also an error, since
    its silence proves nothing.

    Those checks are made twice — once up front and once immediately before
    the rename — and the result is verified afterwards. GNU coreutils can
    refuse a directory destination outright (``mv -T``) and that path is
    taken when available; elsewhere a swap in the instant before the rename
    is detected rather than prevented, and never reported as success.

    Known limit: a peer could point our temp directory's name at a *different
    directory we own*, which passes the ownership check. That costs nothing
    (they cannot reach inside a directory of ours) beyond the publish
    possibly crossing filesystems. Closing it would need fd-relative
    operations, which a shell cannot express — it would mean driving this
    over SFTP instead of ssh.

    The parent directory is assumed to exist (for the owner-marker case
    the rsync that just ran guarantees it).

    Raises:
        ValueError: If *remote_path* is login-relative. The write anchors
            its working directory inside a private temp directory, which
            would change how such a path resolves. Also if *mode* is not
            three octal digits — it reaches a remote shell unquoted.
        RuntimeError: If the target is a symlink or a directory, if the
            target could not be probed, or if any write / publish step
            exits non-zero — so the caller surfaces the failure instead of
            silently leaving a stale file behind.
    """
    if not _MODE_RE.fullmatch(mode):
        # Interpolated into the remote shell command below, so it is
        # constrained here rather than quoted — a mode is three octal
        # digits and nothing else, and anything else is a caller bug.
        raise ValueError(f"mode must be three octal digits, got {mode!r}")

    if not remote_path.startswith(("/", "~")):
        # The write anchors its working directory inside a private temp
        # directory (so the directory entry cannot be swapped out from under
        # it), which changes how a login-relative path would resolve: the
        # rename would land inside the temp directory and the verification
        # would look somewhere else entirely. Rejecting is better than
        # writing the file somewhere the caller did not ask for.
        raise ValueError(
            f"remote_path must be absolute or ~-relative, got "
            f"{remote_path!r} — a login-relative path cannot be resolved "
            "safely by this writer"
        )

    quoted = shlex.quote(remote_path)
    prefix = shlex.quote(f"{PurePosixPath(remote_path).parent}/.srunx-write.")

    # The whole guarded sequence runs in ONE ssh invocation. Split across
    # several it cost that many connections — that many key exchanges, and
    # that many hardware-key touches — on *every* synced submission, since
    # the owner marker is rewritten each time. Sharing one shell also shrinks
    # the window between checking the target and renaming over it to near
    # zero, and lets the checks repeat immediately before the rename.
    #
    # The temp lives inside a directory we create with ``mkdir -m 700``, and
    # that is what makes writing it safe. Two earlier attempts were not:
    #
    # 1. ``mktemp`` then ``chmod`` then ``cat`` — mktemp closes the file, so
    #    the later steps reopened it **by name**. Another account with write
    #    access to the directory could unlink it and leave a symlink in that
    #    gap. Verified locally: the sequence truncated an unrelated file
    #    through the planted link — arbitrary overwrite, not just a broken
    #    marker.
    # 2. A single ``set -C`` redirection — noclobber only refuses an existing
    #    *regular* file (POSIX XCU 2.7.2). Verified locally: with a FIFO
    #    planted at the predictable name, the redirection did not fail, it
    #    **hung** in open(). An attacker could stall every sync, or read the
    #    marker content, and then have ``mv`` publish their object.
    #
    # ``mkdir`` is the exclusive-create primitive that covers every kind of
    # inode: verified to fail with "File exists" against both a planted FIFO
    # and a planted symlink. Mode 700 means nobody else can enter the
    # directory afterwards, so the file created inside it cannot be swapped —
    # no name is reopened in an untrusted directory at any point.
    #
    # ``umask 022`` fixes the marker's mode at creation, since it must stay
    # readable to other accounts on a shared mount: an unreadable marker
    # reads as "no owner" and disables the guard for them.
    #
    # The directory sits beside the target so publishing is a rename within
    # one filesystem — across filesystems ``mv`` becomes copy+unlink and
    # stops being atomic.
    #
    # Cleanup removes the known filename and then ``rmdir``s, rather than
    # ``rm -rf``, so an unexpected entry is never deleted recursively.
    #
    # Exit codes are distinct so the failure can be reported precisely
    # rather than as one opaque "ssh failed".
    script = (
        f"set -u; "
        f"if [ -h {quoted} ]; then exit 3; fi; "
        f"if [ -d {quoted} ]; then exit 4; fi; "
        f"d={prefix}$$; "
        f'mkdir -m 700 -- "$d" || exit 5; '
        # ``cd`` into the directory we just created and work in relative
        # paths from here on. Mode 700 stops others entering it, but it does
        # NOT protect the directory *entry*: write permission on the parent
        # lets a watcher rename our directory away and recreate one under the
        # same name, after which a path like "$d/file" would resolve into
        # theirs. A shell's working directory follows the inode, not the
        # name, so nothing below reopens a path an attacker controls.
        f'cd -- "$d" || {{ rmdir -- "$d"; exit 5; }}; '
        # ``mkdir`` then ``cd`` is itself a TOCTOU: write permission on the
        # parent lets a peer rename our entry away and put their own
        # directory (or a symlink) under the same name before we open it. We
        # would then create, chmod and write inside *their* directory, where
        # they can swap the file for a symlink — back to the arbitrary
        # overwrite this whole sequence exists to prevent. So verify what we
        # actually entered: a peer can only create directories owned by
        # themselves, so a uid match rules that out. Compared via
        # ``ls -ldn`` + ``id -u`` because ``test -O`` is a bash/ksh
        # extension, absent from POSIX test and from dash — the /bin/sh on
        # most Linux clusters. No cleanup here: if this fails we are standing
        # in someone else's directory and must not delete anything in it.
        f'[ "$(ls -ldn . | awk \'NR==1{{print $3}}\')" = "$(id -u)" ] '
        f"|| exit 5; "
        # Random basename via mktemp — safe to use here precisely because
        # this directory is ours and unreachable by others, which was not
        # true of the mount root. It matters because on a non-GNU ``mv``
        # (BSD, BusyBox) a target swapped for a symlink-to-directory makes
        # the publish land at ``<referent>/<basename>``: a predictable name
        # like the PID could be guessed and made to collide with a file the
        # attacker wants destroyed. ``$$`` was no good — it is already
        # exposed in the directory name.
        f'f=$(mktemp -- ./w.XXXXXX) || {{ cd /; rmdir -- "$d"; exit 5; }}; '
        # mktemp creates 0600, which is right for a private control file and
        # wrong for the ownership marker: an unreadable marker reads as "no
        # owner" to the next account and disables the guard for them. Hence
        # the caller's mode. Reopening by name is fine *inside* this
        # directory — the risk it carried at the mount root does not exist
        # where no one else can reach.
        # No ``--`` here: BSD chmod does not accept it and treats it as a
        # filename ("chmod: --: No such file or directory"), which failed
        # every write on macOS-family remotes. Safe to omit because mktemp's
        # template starts with "./", so the name can never look like a flag.
        f'chmod {mode} "$f" || {{ rm -f -- "$f"; cd /; rmdir -- "$d"; exit 9; }}; '
        f'cat > "$f" || {{ rm -f -- "$f"; cd /; rmdir -- "$d"; exit 6; }}; '
        # Re-check right before the rename: a target swapped in after the
        # first check would otherwise be followed by ``mv``.
        f"if [ -h {quoted} ] || [ -d {quoted} ]; then "
        f'rm -f -- "$f"; cd /; rmdir -- "$d"; exit 3; fi; '
        # GNU coreutils can refuse a directory destination outright with
        # ``-T``, which closes the swap race instead of only detecting it.
        # ``mv --version`` is the reliable probe: it succeeds on GNU and
        # fails on BSD, where ``-T`` does not exist (verified — BSD reports
        # "illegal option -- T"). Blindly retrying without ``-T`` on any
        # error would be wrong, since "no such option" and "target is a
        # directory" would be indistinguishable.
        f"if mv --version >/dev/null 2>&1; then "
        f'mv -fT -- "$f" {quoted} '
        f'|| {{ rm -f -- "$f"; cd /; rmdir -- "$d"; exit 7; }}; '
        f"else "
        f'mv -f -- "$f" {quoted} '
        f'|| {{ rm -f -- "$f"; cd /; rmdir -- "$d"; exit 7; }}; '
        f"fi; "
        # Leave the directory before removing it, or the rmdir can fail with
        # the working directory still inside it.
        f'cd /; rmdir -- "$d" 2>/dev/null; '
        # Verify what the rename actually produced. If the target was
        # swapped for a directory (or a symlink to one) in the instant
        # between the recheck above and this rename, ``mv`` moved the temp
        # *inside* it and exited 0. Preventing that portably is not
        # possible — ``mv -T`` is a GNU extension, and its failure cannot
        # be told apart from "target is a directory", so falling back on
        # error would silently drop the guard on BSD. Detecting it is
        # portable, and refusing to call that success is what matters:
        # otherwise the caller believes the marker was updated when the
        # content landed somewhere else entirely.
        # Nothing is deleted on this path, on purpose. An earlier version
        # removed ``<target>/marker`` to avoid leaving the relocated file
        # behind, but if the target was swapped for a symlink to some other
        # directory, that path resolves to an unrelated pre-existing file and
        # the cleanup deletes it — under the syncing user's credentials, and
        # possibly outside the mount. A leftover file costs disk space; a
        # wrong ``rm`` costs data. The error tells the operator what to
        # inspect instead.
        f"if [ ! -f {quoted} ] || [ -h {quoted} ]; then exit 8; fi"
    )
    result = self._ssh_run(script, stdin=content)
    if result.returncode == 0:
        return

    detail = result.stderr.strip()
    reasons = {
        3: (
            f"refusing to write {remote_path!r}: it is a symlink or a "
            "directory. Following a symlink could write outside the mount, "
            "and renaming onto a directory would move the file inside it "
            "while reporting success"
        ),
        4: (
            f"refusing to write {remote_path!r}: it exists as a directory, "
            "so publishing would move the new file inside it instead of "
            "replacing it"
        ),
        5: (
            f"could not set up the private temp directory beside "
            f"{remote_path!r} — anything already occupying that name "
            "(file, symlink, FIFO) lands here, since the directory is "
            "created exclusively, and so does a directory that turned out "
            "not to be owned by this account (which means another user "
            "swapped it in between creating and entering it)"
        ),
        6: f"could not write the temp file beside {remote_path!r}",
        9: (
            f"could not make the replacement for {remote_path!r} readable "
            "(chmod failed); publishing it owner-only would hide the marker "
            "from other accounts sharing the mount, which reads as "
            '"no owner" and disables the guard for them'
        ),
        7: f"could not publish {remote_path!r} (rename failed)",
        8: (
            f"published {remote_path!r} but it is not a regular file "
            "afterwards — something replaced the target mid-write, so the "
            "content may have landed elsewhere and this file was NOT "
            "updated. Nothing was deleted in response, since the swapped-in "
            "path could point anywhere; inspect it on the cluster and remove "
            "any stray 'marker' file yourself before retrying"
        ),
    }
    reason = reasons.get(
        result.returncode,
        f"ssh write to {remote_path!r} failed",
    )
    raise RuntimeError(
        f"{reason} (exit {result.returncode})" + (f": {detail}" if detail else "")
    )

remote_sha256

remote_sha256(remote_path: str) -> str | None

Return the SHA-256 hex digest of remote_path, or None.

Used by :func:srunx.sync.hash_verify.verify_paths_match (#137 part 5) to detect the silent-rsync-failure case where rsync exits 0 but the specific file we're about to sbatch never reached the cluster (excluded by a stray rule, lost to a path-translation bug, …). A None return defers to the caller, which decides whether "missing" or "no tool" should block submission.

Returns:

Type	Description
str | None	The 64-char lowercase hex digest on success.
str | None	None when the file does not exist on the remote.
str | None	None when neither sha256sum nor shasum -a 256
str | None	is available on the remote PATH (logged at debug — the
str | None	rsync that just succeeded is the user's main signal).

Raises:

Type	Description
RuntimeError	For any other ssh / network failure (connection refused, host key mismatch, host unreachable, …). Callers that want "best effort" can catch and downgrade; the marker-read code in :func:check_owner is the prior art for that pattern.

Source code in src/srunx/sync/rsync.py

def remote_sha256(self, remote_path: str) -> str | None:
    """Return the SHA-256 hex digest of *remote_path*, or ``None``.

    Used by :func:`srunx.sync.hash_verify.verify_paths_match`
    (#137 part 5) to detect the silent-rsync-failure case where
    rsync exits 0 but the specific file we're about to ``sbatch``
    never reached the cluster (excluded by a stray rule, lost to
    a path-translation bug, …). A None return defers to the
    caller, which decides whether "missing" or "no tool" should
    block submission.

    Returns:
        The 64-char lowercase hex digest on success.
        ``None`` when the file does not exist on the remote.
        ``None`` when neither ``sha256sum`` nor ``shasum -a 256``
        is available on the remote PATH (logged at debug — the
        rsync that just succeeded is the user's main signal).

    Raises:
        RuntimeError: For any other ssh / network failure (connection
            refused, host key mismatch, host unreachable, …). Callers
            that want "best effort" can catch and downgrade; the
            marker-read code in :func:`check_owner` is the prior art
            for that pattern.
    """
    quoted = shlex.quote(remote_path)
    # Single round-trip: existence check, then prefer sha256sum
    # (Linux), fall back to shasum -a 256 (macOS / BSD). Custom
    # exit codes disambiguate "file missing" and "no tool" from
    # genuine failures so the Python side doesn't have to grep
    # stderr to make that distinction.
    # Written on ONE line. A multi-line script breaks on csh/tcsh login
    # shells even inside ``sh -c '...'``: csh cannot carry a newline through
    # single quotes, so it splits the text and parses the fragments itself.
    # Verified against tcsh — the newline form produced ``Unmatched '``,
    # ``Ambiguous output redirect`` and ``else: endif not found``, and ran
    # part of the script as separate commands.
    script = (
        f"test -f {quoted} || exit {self._SHA256_REMOTE_MISSING_EXIT}; "
        f"if command -v sha256sum >/dev/null 2>&1; then "
        f"sha256sum -- {quoted}; "
        f"elif command -v shasum >/dev/null 2>&1; then "
        f"shasum -a 256 -- {quoted}; "
        f"else exit {self._SHA256_REMOTE_NO_TOOL_EXIT}; fi"
    )
    result = self._ssh_run(script)
    if result.returncode == 0:
        match = self._SHA256_HEX_RE.match(result.stdout.strip())
        if match is None:
            # Unparseable output is a genuine failure — sha256sum
            # / shasum surfaced something we don't understand,
            # better to fail loud than silently fall through to
            # "no hash".
            raise RuntimeError(
                f"could not parse sha256 output for {remote_path!r}: "
                f"{result.stdout.strip()!r}"
            )
        return match.group(1).lower()
    if result.returncode == self._SHA256_REMOTE_MISSING_EXIT:
        return None
    if result.returncode == self._SHA256_REMOTE_NO_TOOL_EXIT:
        logger.debug(
            "Remote sha256 verification skipped for {}: "
            "neither sha256sum nor shasum available on remote PATH",
            remote_path,
        )
        return None
    raise RuntimeError(
        f"ssh sha256 of {remote_path!r} failed "
        f"(exit {result.returncode}): {result.stderr.strip()}"
    )

list_local_files

list_local_files(
    local_path: str | Path,
    exclude_patterns: Sequence[str] | None = None,
) -> list[str]

List the files a push of local_path would transfer, as relative paths.

Runs rsync with this client's binary and merged filter, so the answer cannot disagree with what a real push would send. Re-implementing the matching in Python would drift from rsync's actual semantics for anchored patterns, directory rules and **.

Paths come back in rsync's escaped form, the same one a deletion preview prints, and that is the point: the two are compared to each other. :func:unescape_rsync_path converts back for display.

The inventory is an itemize run against a throwaway empty directory — every file is "new" against it — rather than --list-only, whose output cannot be parsed safely. That listing prints a newline inside a filename literally, so one file becomes two lines; a name crafted to look like a listing line then yields two paths that do not exist. A directory named victim\n-rw-r--r-- 1 2026 was enough to record both victim and a fabricated output.py — and had a job written anything by either name, the comparison would have offered live job output for deletion. Itemize escapes the newline (\#012) and the backslash (\#134), so one line is always exactly one file.

Directories are excluded from the result — callers record files.

Raises:

Type	Description
RuntimeError	If rsync exits non-zero, so a partial listing is never mistaken for a complete one.

Source code in src/srunx/sync/rsync.py

def list_local_files(
    self,
    local_path: str | Path,
    exclude_patterns: Sequence[str] | None = None,
) -> list[str]:
    """List the files a push of *local_path* would transfer, as relative paths.

    Runs rsync with this client's binary and merged filter, so the answer
    cannot disagree with what a real push would send. Re-implementing the
    matching in Python would drift from rsync's actual semantics for
    anchored patterns, directory rules and ``**``.

    Paths come back in rsync's **escaped** form, the same one a deletion
    preview prints, and that is the point: the two are compared to each
    other. :func:`unescape_rsync_path` converts back for display.

    The inventory is an itemize run against a throwaway empty directory —
    every file is "new" against it — rather than ``--list-only``, whose
    output cannot be parsed safely. That listing prints a newline inside a
    filename literally, so one file becomes two lines; a name crafted to
    look like a listing line then yields *two* paths that do not exist. A
    directory named ``victim\\n-rw-r--r--   1 2026`` was enough to record
    both ``victim`` and a fabricated ``output.py`` — and had a job written
    anything by either name, the comparison would have offered live job
    output for deletion. Itemize escapes the newline (``\\#012``) and the
    backslash (``\\#134``), so one line is always exactly one file.

    Directories are excluded from the result — callers record files.

    Raises:
        RuntimeError: If rsync exits non-zero, so a partial listing is never
            mistaken for a complete one.
    """
    local = Path(local_path)
    src = str(local)
    if local.is_dir() and not src.endswith("/"):
        src += "/"

    # No ``--protect-args``: it exists to stop a *remote* shell re-splitting
    # arguments, and this listing is entirely local. openrsync does not have
    # the flag at all, so adding it would fail there for no benefit.
    #
    # ``-n`` keeps the destination untouched; the directory only has to be
    # empty so that nothing is filtered out as already up to date.
    with tempfile.TemporaryDirectory(prefix="srunx-inventory-") as empty:
        cmd: list[str] = ["rsync", "-a", "-n", "-i"]
        for pattern in self._merge_excludes(exclude_patterns):
            cmd.extend(["--exclude", pattern])
        cmd.extend(["--", src, empty + "/"])

        logger.debug("Listing local files: {}", shlex.join(cmd))
        result = subprocess.run(  # noqa: S603
            cmd,
            capture_output=True,
            text=True,
            # Escaped output is pure ASCII, so this never has to substitute
            # anything; it is here so an undecodable byte on *stderr* from
            # a remote can't raise in place of the real error.
            errors="surrogateescape",
            env=_C_LOCALE_ENV(),
        )
    if result.returncode != 0:
        raise RuntimeError(
            f"rsync inventory failed (exit {result.returncode}): "
            f"{result.stderr.strip()}"
        )
    return self._parse_inventory(result.stdout)

effective_excludes

effective_excludes(
    exclude_patterns: Sequence[str] | None = None,
) -> list[str]

Return the patterns a call passing exclude_patterns would filter on.

push / pull merge per-call patterns on top of the instance's for that invocation only, without storing them, so :attr:exclude_patterns alone under-reports what a given call actually filtered on — it omits exactly the mount-level patterns a user configured.

That matters wherever the filter is reported back to a user: an excluded path is invisible to an inspection and protected from a mirror's deletions, so a missing pattern makes the report read as "in sync" when it really means "never looked at".

Source code in src/srunx/sync/rsync.py

def effective_excludes(
    self, exclude_patterns: Sequence[str] | None = None
) -> list[str]:
    """Return the patterns a call passing *exclude_patterns* would filter on.

    ``push`` / ``pull`` merge per-call patterns on top of the instance's for
    that invocation only, without storing them, so
    :attr:`exclude_patterns` alone under-reports what a given call actually
    filtered on — it omits exactly the mount-level patterns a user
    configured.

    That matters wherever the filter is reported back to a user: an excluded
    path is invisible to an inspection *and* protected from a mirror's
    deletions, so a missing pattern makes the report read as "in sync" when
    it really means "never looked at".
    """
    return list(self._merge_excludes(exclude_patterns))

get_default_remote_path staticmethod

get_default_remote_path(
    local_path: str | Path | None = None,
) -> str

Derive a default remote workspace path from the git repo or cwd.

Parameters:

Name	Type	Description	Default
local_path	str | Path | None	Optional local directory to derive the project name from. If None, uses the current working directory.	None

Returns:

Type	Description
str	A path like ~/.config/srunx/workspace/<project_name>/.

Source code in src/srunx/sync/rsync.py

@staticmethod
def get_default_remote_path(local_path: str | Path | None = None) -> str:
    """Derive a default remote workspace path from the git repo or cwd.

    Args:
        local_path: Optional local directory to derive the project name
            from. If None, uses the current working directory.

    Returns:
        A path like ``~/.config/srunx/workspace/<project_name>/``.
    """
    cwd = str(Path(local_path)) if local_path else None
    try:
        result = subprocess.run(  # noqa: S603, S607
            ["git", "rev-parse", "--show-toplevel"],
            capture_output=True,
            text=True,
            cwd=cwd,
        )
        if result.returncode == 0:
            basename = Path(result.stdout.strip()).name
        else:
            basename = Path(cwd).name if cwd else Path.cwd().name
    except FileNotFoundError:
        # git not installed
        basename = Path(cwd).name if cwd else Path.cwd().name

    return f"~/.config/srunx/workspace/{basename}/"

get_logger

get_logger(name: str) -> Logger

Get a logger instance for a module.

Parameters:

Name	Type	Description	Default
name	str	Module name (usually name).	required

Returns:

Type	Description
Logger	Logger instance.

Source code in src/srunx/common/logging.py

def get_logger(name: str) -> Logger:
    """Get a logger instance for a module.

    Args:
        name: Module name (usually __name__).

    Returns:
        Logger instance.
    """
    return logger.bind(name=name)  # type: ignore

_C_LOCALE_ENV

_C_LOCALE_ENV() -> dict[str, str]

Environment forcing rsync's output into one deterministic encoding.

Which bytes rsync escapes depends on what the locale calls printable, and the two outputs this compares must agree. Verified on openrsync: under a UTF-8 locale データ.csv comes back as a mix of raw and escaped bytes (\ufffd\#203\#207\ufffd...) while LC_ALL=C gives the fully escaped \#343\#203\#207.... Run the inventory one way and the deletion preview the other, and a genuinely stale non-ASCII file matches nothing — it is dropped from the report and the mount reads as clean.

Only output formatting is affected. rsync passes filenames through as bytes unless --iconv is given, which srunx never does.

Source code in src/srunx/sync/rsync.py

def _C_LOCALE_ENV() -> dict[str, str]:  # noqa: N802 — reads as a constant
    """Environment forcing rsync's output into one deterministic encoding.

    Which bytes rsync escapes depends on what the locale calls printable, and
    the two outputs this compares must agree. Verified on openrsync: under a
    UTF-8 locale ``データ.csv`` comes back as a *mix* of raw and escaped bytes
    (``\\ufffd\\#203\\#207\\ufffd...``) while ``LC_ALL=C`` gives the fully escaped
    ``\\#343\\#203\\#207...``. Run the inventory one way and the deletion preview
    the other, and a genuinely stale non-ASCII file matches nothing — it is
    dropped from the report and the mount reads as clean.

    Only output formatting is affected. rsync passes filenames through as bytes
    unless ``--iconv`` is given, which srunx never does.
    """
    return {**os.environ, "LC_ALL": "C"}

unescape_rsync_path

unescape_rsync_path(path: str) -> str

Turn rsync's escaped output back into the filename it stands for.

Paths are compared in their escaped form — that is what makes an inventory entry and a deletion candidate the same string — but a user reading a report should see データ.csv, not \#343\#203\#207....

Only printable characters are restored. rsync escapes control bytes because they are not safe to print, and this output goes to a terminal: a file named innocent\#033c.py would otherwise emit ESC-c and reset the terminal, and \#012 would forge an extra line in a sync preview. Since a cluster job can choose the names it writes, that is an injection an attacker controls. Anything unprintable — control bytes, bidi overrides, undecodable bytes left as surrogates — stays in the escaped form, which is still perfectly readable and names the file unambiguously.

A run that decodes to a mix of printable and not is kept escaped whole. It is the conservative direction and such names do not occur by accident.

Source code in src/srunx/sync/rsync.py

def unescape_rsync_path(path: str) -> str:
    """Turn rsync's escaped output back into the filename it stands for.

    Paths are compared in their escaped form — that is what makes an inventory
    entry and a deletion candidate the same string — but a user reading a
    report should see ``データ.csv``, not ``\\#343\\#203\\#207...``.

    **Only printable characters are restored.** rsync escapes control bytes
    because they are not safe to print, and this output goes to a terminal:
    a file named ``innocent\\#033c.py`` would otherwise emit ESC-c and reset the
    terminal, and ``\\#012`` would forge an extra line in a sync preview. Since
    a cluster job can choose the names it writes, that is an injection an
    attacker controls. Anything unprintable — control bytes, bidi overrides,
    undecodable bytes left as surrogates — stays in the escaped form, which is
    still perfectly readable and names the file unambiguously.

    A run that decodes to a mix of printable and not is kept escaped whole. It
    is the conservative direction and such names do not occur by accident.
    """
    if "\\#" not in path:
        return path
    return _ESCAPE_RUN_RE.sub(_restore_run, path)

_restore_run

_restore_run(match: Match[str]) -> str

Source code in src/srunx/sync/rsync.py

def _restore_run(match: re.Match[str]) -> str:
    run = match.group(0)
    octets = bytes(int(g, 8) for g in _ESCAPE_GROUP_RE.findall(run))
    text = octets.decode("utf-8", "surrogateescape")
    return text if text.isprintable() else run

Sweep orchestration

srunx.runtime.sweep.expand

Pure functions for matrix expansion, sweep-spec merging, and CLI flag parsing.

All validation routes through WorkflowValidationError so CLI, Web API, and MCP paths surface a consistent error category.

expand_matrix

expand_matrix(
    matrix: dict[str, list[Any]], base_args: dict[str, Any]
) -> list[dict[str, Any]]

Cross-product of matrix axes merged into base_args.

Axis iteration order follows insertion order of matrix. Matrix values override any identically-keyed entry in base_args (matrix wins at the args level; the deps.<parent>.<key> channel remains a separate space).

Raises:

Type	Description
WorkflowValidationError	empty matrix (R2.10), empty axis list (R2.4), non-scalar axis value (R2.5), axis named deps (R2.3), or cell_count > 1000 (R2.8).

merge_sweep_specs

merge_sweep_specs(
    yaml_sweep: SweepSpec | None,
    cli_sweep_axes: dict[str, list[ScalarValue]],
    cli_arg_overrides: dict[str, str],
    cli_fail_fast: bool | None,
    cli_max_parallel: int | None,
) -> SweepSpec | None

Merge YAML sweep: block with CLI flags at axis granularity.

• If neither YAML nor CLI provides any matrix axis, returns None (caller runs the non-sweep path).
• CLI axes replace same-named YAML axes; CLI-only axes are added.
• --arg KEY colliding with --sweep KEY is rejected (R3.6).
• The final max_parallel must be set and >= 1 (R2.6).

Raises:

Type	Description
WorkflowValidationError	on --arg/--sweep key collision or missing/invalid final max_parallel.

parse_arg_flags

parse_arg_flags(raw: list[str]) -> dict[str, str]

Tokenize --arg KEY=VALUE occurrences.

Rules: - Split on the FIRST = (later = characters stay in the value). - Duplicate keys: last occurrence wins (R1.2). - Missing = raises WorkflowValidationError (R3.8). - Values are always strings; no int/float/bool auto-cast (R3.10).

parse_sweep_flags

parse_sweep_flags(raw: list[str]) -> dict[str, list[str]]

Tokenize --sweep KEY=v1,v2,v3 occurrences.

• Split axis at the first = (axis names cannot contain =).
• Values are split on , with no escape handling (Phase 1, R3.5).
• Empty elements (a,,b) are preserved as empty strings (R3.9).
• Missing = raises WorkflowValidationError (R3.8).
• Duplicate axis: last occurrence wins (consistent with parse_arg_flags).

srunx.runtime.sweep.orchestrator

SweepOrchestrator: materialize matrix cells and drive them under a semaphore.

See .claude/specs/workflow-parameter-sweep/design.md § SweepOrchestrator and tasks 17-20.

SweepOrchestrator

SweepOrchestrator(
    *,
    workflow_yaml_path: Path | None,
    workflow_data: dict[str, Any],
    args_override: dict[str, Any] | None,
    sweep_spec: SweepSpec,
    submission_source: Literal["cli", "web", "mcp"],
    callbacks: Sequence[Callback] | None = None,
    endpoint_id: int | None = None,
    preset: str = "terminal",
    executor_factory: WorkflowJobExecutorFactory
    | None = None,
    submission_context: SubmissionRenderContext
    | None = None,
)

Drive sweep execution: materialize cells, run them, aggregate status.

run

run() -> SweepRun

Execute the sweep synchronously and return the final SweepRun.

materialize

materialize() -> int

Expand + materialize cells synchronously; return sweep_run_id.

Separated from :meth:arun so HTTP callers can materialize inside the request (to obtain sweep_run_id) and then spawn :meth:arun_from_materialized as a background task.

arun_from_materialized async

arun_from_materialized(sweep_run_id: int) -> SweepRun

Run the execution loop for an already-materialized sweep.

Assumes :meth:materialize (or equivalent) populated self._cells and self._sweep_run_id. Used by both :meth:arun (materialize + run in the same call) and the Web dispatcher (materialize synchronously, then spawn this as a background task).

arun async

arun() -> SweepRun

Execute the sweep with bounded concurrency.

Steps: expand → materialize → spawn N cells behind an anyio.Semaphore(min(max_parallel, cell_count)) → return the final SweepRun row.

resume_from_db async

resume_from_db(
    sweep_run_id: int, pending_cells: list[CellSpec]
) -> SweepRun

Resume a sweep whose cells were already materialized.

Used by :class:srunx.runtime.sweep.reconciler.SweepReconciler to spawn orchestrator tasks after a crash. Skips expand + materialize; the caller provides the already-materialized pending cells.

request_cancel

request_cancel() -> None

Mark the sweep as cancel-requested and drain pending cells.

Idempotent: a second call is a no-op because _cancelled is set and SweepRunRepository.request_cancel guards on cancel_requested_at IS NULL.

get_active_orchestrator

get_active_orchestrator(
    sweep_run_id: int,
) -> SweepOrchestrator | None

Return the live orchestrator for sweep_run_id or None.

drain_sweep_pending_cells

drain_sweep_pending_cells(sweep_run_id: int) -> int

Cancel every still-pending cell for sweep_run_id and sync counters.

Runs in a single IMMEDIATE TX on a fresh connection. After the drain it triggers the aggregator so the sweep can transition to its final status if every in-flight cell is already done. Returns the number of cells moved from pending to cancelled (0 when nothing was pending).

This is the out-of-process drain used by the cancel endpoint when no in-process orchestrator is registered (crash-recovery path) and by :class:SweepOrchestrator itself via :meth:SweepOrchestrator._drain.

Security helpers

srunx.runtime.security.python_args

Reject python: prefix in user-supplied args / matrix values.

Runs on Web API submission (YAML + JSON) and MCP tool calls. The python: prefix in args is a server-side evaluation escape hatch reserved for CLI-local use; exposing it over transport boundaries is a security concern (remote code execution via workflow mutation). See :func:srunx.runtime.workflow.loader._has_python_prefix for the CLI-side parser that actually evaluates these values — the check here mirrors its matching rules (prefix match, leading-whitespace-tolerant, case-insensitive).

PythonPrefixViolation dataclass

PythonPrefixViolation(source: str, path: str, value: str)

Structured violation record. Caller converts to transport-specific error.

source instance-attribute

source: str

Logical origin of the payload (e.g. "args", "sweep.matrix").

path instance-attribute

path: str

Dotted / indexed path to the offending value (e.g. "x[2]", "lr").

value instance-attribute

value: str

The offending value, reproduced for error messaging.

find_python_prefix

find_python_prefix(
    payload: Any, *, source: str, _path: str = ""
) -> PythonPrefixViolation | None

Recursively scan a dict / list / scalar payload; return the first violation.

Traverses nested dict -> list -> str. Non-string scalars (int / float / bool / None) are ignored. Returns None when no violation is found.

srunx.runtime.security.mount_paths

Mount-root guard for ShellJob script paths.

Both the Web API and the MCP tool surface the same attack shape: a workflow YAML can declare template: shell with an arbitrary script_path that render_shell_job_script then reads verbatim. Without a guard, a caller could exfiltrate or inject arbitrary host files via e.g. script_path: ../../../etc/passwd.

The helper here returns a structured ShellJobScriptViolation so each transport caller can raise the right exception type (Web → HTTPException(403), MCP → ValueError) while the actual directory-check logic lives in one place.

ShellJobScriptViolation dataclass

ShellJobScriptViolation(job_name: str, script_path: str)

First ShellJob whose script path escapes the allowed mount roots.

find_shell_script_violation

find_shell_script_violation(
    workflow: Workflow, mount_roots: Iterable[Path]
) -> ShellJobScriptViolation | None

Return the first ShellJob pointing outside every mount_roots entry.

mount_roots must already be resolved absolute paths (see :meth:pathlib.Path.resolve). Non-ShellJob entries are ignored. Returns None when every ShellJob's script_path is contained in at least one root.

SLURM protocol constants

srunx.slurm.states

SLURM protocol-level state constants.

Separate from :class:srunx.domain.JobStatus (the domain-level enum) because SLURM's raw state vocabulary is wider than what srunx models: SLURM emits NODE_FAIL / PREEMPTED / OUT_OF_MEMORY for terminal failures that srunx currently collapses into FAILED at the domain boundary. Keeping these strings in a single module lets every caller that speaks SLURM-native states (notification preset filter, active-watch poller, web SSH adapter) agree on the set without risking drift when a new terminal state is added.