# Scheduler Lock v2 — Developer Operations Guide **Status:** current implementation, as of 2026-08-05 **Audience:** developers who maintain, operate, test and troubleshoot the Capture Scheduler **Scope:** how to *run* Scheduler Lock v2 — not how it was designed and not why it was ratified | Question | Where the answer lives | |---|---| | How do I operate, test and troubleshoot it? | **this document** | | Why was it built this way, what race did it close, what did the stress campaign measure? | [Scheduler Lock v2 — architecture](v2/architecture/v2_scheduler_lock_v2_2026-08-05.md) — **frozen** | | How is the scheduler as a whole deployed, scheduled and monitored? | [The Capture Scheduler](v2/operations/capture_scheduler.md) | | What do I do about a `LOSS` or a `BREACH` the health step reported? | ROM-01 §9 (removed 2026-08-31 — see [the retirement record](v2/architecture/v2_research_infrastructure_retirement_2026-08-30.md)) | > **Warning — the implementation is frozen.** > `backend/capture_scheduler/lock.py` is not modified incrementally. A defect or a > new requirement opens a **Scheduler Lock v3 design**, it does not open an edit. > Everything in this guide describes behaviour you can rely on; none of it > describes behaviour you should change. --- ## 1. Overview ### 1.1 What it is Scheduler Lock v2 is the **run lock** that guarantees at most one Research Scheduler job — the daily contract registration or the weekly capture — is executing against the platform's broker session and database at any moment. It is a **lease**: a single-line record on disk, whose every read-decide-write happens while the deciding process holds `flock(LOCK_EX)` on that same file. ``` logs/research/scheduler.lock version=2 pid=48213 at=2026-08-05T19:00:00+05:30 token=9f3c1ab27de40518 ``` ### 1.2 Why it exists The scheduler is invoked by cron, by a systemd timer, by a GitHub workflow, or by a developer typing `run weekly` — often on the same host, sometimes at the same minute. Two runs overlapping means: - two concurrent captures against **one** broker session, - duplicated broker calls counted against `MAX_CALLS`, - two writers into the same contract-registry rows. A lock file alone is not enough to prevent that, which is what v1 discovered. ### 1.3 What problems it solved | Problem in v1 | What went wrong | How v2 closes it | |---|---|---| | **Breaking a stale lock was a read-modify-write with no mutual exclusion** | `stat` → judge → `unlink` → `open(O_EXCL)` are four syscalls. Two runs judging the same stale lock in the same instant both unlinked and both created. Two owners, one window. | The judgement *and* the break happen inside one `flock(LOCK_EX)` critical section. The second contender cannot read the record until the first has finished rewriting it. | | **A displaced owner deleted its successor's lock** | v1's `release` unlinked unconditionally. A run whose lock had been broken as stale deleted the *new* holder's lock when it finished. | Each lease carries a **fencing token**. `release` removes the file only if the token on disk is still the one it wrote. | | **Two implementations could not be told apart** | A v1 lock file and a v2 lock file look similar and cannot serialise against each other. | Every record opens with `version=2`. A lease of an unsupported version is **never judged, never broken, never rewritten** — it is reported and the run is skipped. | Measured, on the frozen stress campaign (800 trials per implementation): **v1 produced 13 duplicate acquisitions and 28 overlap events; v2 produced none.** ### 1.4 What guarantees it provides | Guarantee | Rests on | |---|---| | **Exactly one owner.** N processes deciding simultaneously reach one verdict between them, not one each. | Kernel serialisation per open file description. A slower judgement widens no window, because there is no window. | | **The mutex cannot deadlock.** | It is held for two file reads and one write — never across a job, never across a subprocess — and the kernel drops it if the holder dies mid-section. | | **Ownership survives the process.** A crashed run leaves a lease that is respected while young and broken once stale. | The lease is the *record*, not the open descriptor. This is also why a **hung** holder can be recovered from at all. | | **A displaced owner cannot delete its successor's lock.** | The fencing token, checked on release. | | **A missed window is never caused by a crash.** A stale lease is broken rather than respected. | Staleness = `2 × STEP_TIMEOUT_SECONDS`, measured on the run's own clock. | | **Mixed versions fail loudly, not silently.** | The `version=` gate; a `VersionConflict` report on five alert surfaces. | | **Fail-closed on a platform with no file mutex.** | The one operation that needs serialisation to be safe — breaking somebody else's lock — is refused rather than performed unsafely. | > **Note.** "Exactly one owner" is a property of the lock, not of the host. > Two hosts sharing a checkout over a network filesystem is out of scope — see > §13.4. --- ## 2. Architecture ### 2.1 The lock file A single file, in the scheduler's log directory, present **only while a run is in flight**: ``` $LOG_DIR/scheduler.lock # default: /logs/research/scheduler.lock ``` Defined by `runner.LOCK_NAME = "scheduler.lock"` and located by `runner.lock_path(config) == config.log_dir / LOCK_NAME`. ### 2.2 The lease The file's body is the lease — one line, whitespace-separated `key=value` pairs, every field found **by prefix and never by position**. Full field reference in §5. A lease is *held* until either its owner releases it or it ages past the stale threshold. Age is measured on **one clock**: `now − at`, both readings of the run's own clock, which is what makes the rule deterministic under an injected test clock. ### 2.3 Versioning `version=` is written **first**, so `head -c 9` identifies a lock file. | Constant | Value | Meaning | |---|---|---| | `lock.SUPPORTED_VERSION` | `2` | The only format this build writes, and the only one it will judge | | `lock.LEGACY_VERSION` | `1` | What a lease carrying no `version=` field is — v1 wrote `pid=` and `at=` and nothing else | | `runner.SUPPORTED_LOCK_VERSION` | `2` | Re-export, so `runner.` callers need not import `lock` | A lease of any other version is refused — see §11. ### 2.4 The fencing token `token=` is 16 hex characters (`secrets.token_hex(8)`), **fresh on every acquisition**. It is the only thing `release` trusts: - token on disk matches the handle's → remove the file; - token differs, or there is no token → **leave it alone**. This run was displaced as stale while it worked; its successor owns the window. > **Tip.** The token is for the program, not for you. You never need to read, > compare or preserve it by hand. ### 2.5 `flock()` — the mutex | Property | Why it was chosen | |---|---| | Held per **open file description** | Two independent `open()` calls contend *even inside one process*, so the mutex is real in a test that forks nothing. | | POSIX `fcntl(F_SETLK)` was **rejected** | Its locks are per `(process, inode)`: a second acquisition inside one process silently succeeds and the mutex evaporates exactly where the test suite would be looking at it. | | The kernel drops it when the holder dies | The one failure mode a hand-rolled mutex has is the one `flock` structurally does not. | `lock.MUTEX_BACKEND` reports which backend is live: `"flock"` on POSIX, `"msvcrt"` on Windows, `"none"` if neither exists (then breaking a stale lock is refused — fail-closed). ### 2.6 Scheduler interaction The scheduler owns **which file** and **what stale means**; the lock module owns the mechanism. Nothing else crosses the boundary. | Layer | Symbol | Responsibility | |---|---|---| | `capture_scheduler/config.py` | `STEP_TIMEOUT_SECONDS`, `LOG_DIR` | The units | | `capture_scheduler/runner.py` | `lock_path`, `acquire_lock`, `release_lock`, `STALE_AFTER_STEP_TIMEOUTS = 2` | Config → primitive binding | | `capture_scheduler/lock.py` | `acquire`, `release`, `LockHandle`, `VersionConflict` | The primitive. Imported by `runner` only | Stale threshold = `STEP_TIMEOUT_SECONDS × 2` = **7200 s (2 h)** by default. Rationale: a run that hit the ceiling on *both* of its steps is still judged live; anything beyond that is a run that is no longer running. ### 2.7 Lifecycle — Acquire → Run → Release ``` python -m capture_scheduler run │ ▼ ┌────────────────────────────────────┐ │ ENABLE_AUTOMATIC_CAPTURE == false? │ └────────────────────────────────────┘ yes │ │ no log + exit 0 │ ▼ (lock NEVER touched) ╔═════════════════════════╗ ║ ACQUIRE ║ ╚═════════════════════════╝ │ ┌────────────────────────────┴─────────────────────────┐ ▼ ▼ open(O_RDWR|O_CREAT|O_EXCL) FileExistsError → created by us → open(O_RDWR) │ │ └──────────────────────┬───────────────────────────────┘ ▼ ┌───────────────────────────────┐ │ flock(fd, LOCK_EX) ← BLOCKS │ └───────────────────────────────┘ ▼ ┌───────────────────────────────┐ │ same inode still at the path? │──no──▶ close, retry └───────────────────────────────┘ (≤ 64 times) │ yes ╔═══════════════════ CRITICAL SECTION ═══════════════════╗ ║ read body ║ ║ state=free ......................... claim it ║ ║ version ≠ 2 ........................ REFUSE ────────╫──▶ VersionConflict ║ age ≤ 2 × step_timeout ............. REFUSE ────────╫──▶ "lock held" ║ no mutex available ................. REFUSE ────────╫──▶ fail-closed ║ otherwise: ftruncate + write + fsync ║ ║ version=2 pid=… at=… token= ║ ╚════════════════════════════════════════════════════════╝ ▼ flock(fd, LOCK_UN); close(fd) on_conflict(...) runs HERE — outside the mutex ▼ LockHandle(path, token, acquired_at) │ ╔════════════▼════════════╗ ║ RUN ║ lock NOT held — ╠═════════════════════════╣ only the lease is ║ 1. capture step ║ ║ daily → --register-only ║ weekly → --days N --max-calls M ║ retried, exponential backoff ║ 2. health step ║ ║ python -m research_platform health ║ never retried ║ ║ 3. logs, state, alerts ║ ╚════════════▼════════════╝ │ (finally: always) ╔════════════▼════════════╗ ║ RELEASE ║ ╚═════════════════════════╝ ▼ open(O_RDWR) → flock → verify inode ▼ ┌───────────────────────────────┐ │ token on disk == my token? │ └───────────────────────────────┘ yes │ │ no unlink()│ │ leave it — a successor owns it ▼ ▼ file gone file untouched ``` > **Note.** During **Run** the `flock` is *not* held. The mutex protects only the > decision; the lease protects the window. That is deliberate: a `flock` held for > the whole job would make a *hung* run's lock unbreakable, which is the failure > that actually shows up at 03:00. --- ## 3. File Structure ### 3.1 Implementation | File | Purpose | When it is used | When to modify | |---|---|---|---| | [`backend/capture_scheduler/lock.py`](../backend/capture_scheduler/lock.py) | **The primitive.** Mutex backend, record format, version gate, `acquire`, `release`, `LockHandle`, `VersionConflict` | Every job run, both ends | **Never — frozen.** A defect opens a Scheduler Lock v3 design | | [`backend/capture_scheduler/runner.py`](../backend/capture_scheduler/runner.py) | Job execution; the scheduler's half of the lock contract (`lock_path`, `acquire_lock`, `release_lock`, `STALE_AFTER_STEP_TIMEOUTS`) | Every job run | When job *behaviour* changes — steps, retries, alerts. Not for lock mechanics | | [`backend/capture_scheduler/config.py`](../backend/capture_scheduler/config.py) | Schedule and run policy; `STEP_TIMEOUT_SECONDS` (the stale threshold's unit) and `LOG_DIR` (the lock's home) | Every invocation | When a new configuration variable is genuinely needed | | [`backend/capture_scheduler/cli.py`](../backend/capture_scheduler/cli.py) | `run` / `status` / `config` / `render`; exit codes | Every invocation | When a command or an exit-code contract changes | | [`backend/capture_scheduler/state.py`](../backend/capture_scheduler/state.py) | The run ledger — `scheduler-state.json` | Every step, every alert | When the recorded position changes shape (bump `STATE_VERSION`) | | [`backend/capture_scheduler/journal.py`](../backend/capture_scheduler/journal.py) | The three dated log streams | Every step | When a log stream is added or its format changes | | [`backend/capture_scheduler/deploy.py`](../backend/capture_scheduler/deploy.py) | Renders cron / systemd / GitHub / env from one config | `render` only | When a deployment mode changes. **The lock is not part of any artefact** | | [`backend/capture_scheduler/__main__.py`](../backend/capture_scheduler/__main__.py) | Puts `backend/` on `sys.path`, then calls `cli.main` | Every `python -m` invocation | Effectively never | ### 3.2 Tests | File | Purpose | When it is used | When to modify | |---|---|---|---| | [`backend/tests/test_capture_scheduler_lock.py`](../backend/tests/test_capture_scheduler_lock.py) | **56 tests.** The primitive under real concurrency — forking, crash recovery, fencing, the version gate | CI gate 1, and after any change near the lock | Only alongside a v3 design. Never weaken an assertion to make a change pass | | [`backend/tests/test_capture_scheduler.py`](../backend/tests/test_capture_scheduler.py) | **82 tests.** Scheduler behaviour: config, schedule, retries, alerts, logs, state, deployment artefacts, boundaries — plus `TestTheLock`, the v1-era lock behaviour that v2 left intact | CI gate 1 | When job behaviour changes | ### 3.3 Tooling and artefacts | File | Purpose | When it is used | When to modify | |---|---|---|---| | [`backend/scripts/stress/scheduler_lock_stress.py`](../backend/scripts/stress/scheduler_lock_stress.py) | Contention harness at 50 and 100 processes, plus the **v1 exhibit** — the old algorithm, transcribed, imported by nothing | Manually, before a release touching the scheduler | Only to add a scenario. The v1 transcription is evidence — do not "improve" it | | [`backend/scripts/ci/run_all.sh`](../backend/scripts/ci/run_all.sh) | The thirteen-gate CI entry point | Before every commit | When a gate is added | | [`backend/scripts/capture_weekly.py`](../backend/scripts/capture_weekly.py) | The capture step's subprocess | Inside a held lease | Outside this guide's scope | | [`deploy/capture-scheduler/`](../deploy/capture-scheduler/) | Committed renders of cron / systemd / GitHub / env | On install | Regenerate, never hand-edit — a test compares them against the renderer | ### 3.4 Runtime files (git-ignored) | Path | Purpose | |---|---| | `logs/research/scheduler.lock` | **The lock.** Present only while a run is in flight | | `logs/research/scheduler-state.json` | The run ledger — last attempt, last success, consecutive failures, last 20 alerts | | `logs/research/scheduler_YYYY_MM_DD.log` | The envelope: job start, each step's outcome, retries, alerts, verdict, `SCHEDULER LOCK — VERSION CONFLICT` blocks | | `logs/research/capture_YYYY_MM_DD.log` | Every capture step, with what it wrote | | `logs/research/health_YYYY_MM_DD.log` | Every health check, with its verdict | | `logs/research/alerts.log` | Append-only, uncapped | | `logs/research/cron.out` | Whatever the rendered cron line redirects | --- ## 4. Daily Developer Commands Every command below is copy-paste ready. Set `REPO` once per shell. ```bash export REPO=/Users/macbook/Documents/pgProject/pg3/ns/AI_Trade ``` > **Warning — where you run from matters.** > `python -m capture_scheduler` resolves only when `backend/` is importable. > Two forms work; **anything else raises `ModuleNotFoundError: capture_scheduler`.** > > ```bash > cd "$REPO/backend" && ./venv/bin/python -m capture_scheduler status # form A > cd "$REPO" && PYTHONPATH=backend backend/venv/bin/python -m capture_scheduler status # form B > ``` > > This guide uses **form A**. The rendered cron and systemd artefacts run from > the repository root and depend on the deployment supplying the path — see §10. ### 4.1 Environment ```bash # Activate the virtualenv (optional — every command below names the interpreter) source "$REPO/backend/venv/bin/activate" # Which interpreter the scheduler will use, and where every value came from cd "$REPO/backend" && ./venv/bin/python -m capture_scheduler config ``` ### 4.2 Running jobs ```bash # Daily job — register contracts only. No broker candle download. cd "$REPO/backend" && ./venv/bin/python -m capture_scheduler run daily # Weekly job — register contracts AND backfill CAPTURE_DAYS of candles. cd "$REPO/backend" && ./venv/bin/python -m capture_scheduler run weekly ``` | Exit code | Meaning | |---|---| | `0` | Every step succeeded — **or** `ENABLE_AUTOMATIC_CAPTURE=false`, which is a deliberate state, not a failure | | `1` | A step failed, the health check reported a `LOSS`/`BREACH`, the lock was held, or the lock version conflicted. An alert has already been raised | | `2` | A configuration value could not be honoured. **Nothing ran** | > **Tip.** To exercise the lock without touching the broker, run the daily job > twice concurrently — see §7.1. ### 4.3 Status ```bash # Scheduler status — exit 0 when the schedule is being met, 1 when it is not cd "$REPO/backend" && ./venv/bin/python -m capture_scheduler status # The programme's screen — the same block, nested, alongside campaigns and α cd "$REPO/backend" && ./venv/bin/python -m research_platform status ``` ### 4.4 Verification ```bash # Research Platform reconciliation rules (RC-1…RC-4) — CI's entry point cd "$REPO/backend" && ./venv/bin/python -m research_platform verify # Programme health — the same command the scheduler's second step runs cd "$REPO/backend" && ./venv/bin/python -m research_platform health # Dashboard and metrics cd "$REPO/backend" && ./venv/bin/python -m research_platform dashboard cd "$REPO/backend" && ./venv/bin/python -m research_platform metrics ``` > **Note.** There is no `capture_scheduler verify` subcommand. The scheduler's > verification surface is `status` (exit code) and `config` (provenance); the > Research Platform owns `verify`. ### 4.5 Rendering deployment artefacts ```bash cd "$REPO/backend" # cron — the default deployment mode ./venv/bin/python -m capture_scheduler render cron # systemd — for a host that is not always on (Persistent=true) ./venv/bin/python -m capture_scheduler render systemd # GitHub Actions — rendered to stdout, never installed by this package ./venv/bin/python -m capture_scheduler render github # The env file an operator edits, with every default stated ./venv/bin/python -m capture_scheduler render env # Render for a host you are not on ./venv/bin/python -m capture_scheduler render cron --repo-root /srv/ai_trade ``` > **Warning.** None of the four artefacts mentions the lock. Changing lock > behaviour never requires a re-render; hand-editing a render always breaks the > test that compares `deploy/capture-scheduler/` against the renderer. ### 4.6 Tests ```bash cd "$REPO/backend" # The lock primitive — 56 tests, forking, ~1 s ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py -q # Scheduler behaviour — 82 tests ./venv/bin/python -m pytest tests/test_capture_scheduler.py -q # Both — 138 tests ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py tests/test_capture_scheduler.py -q # Only the concurrency tests ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py -q \ -k "SimultaneousAcquisition or CrashRecovery" # Only the versioning tests ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py -q \ -k "TheFormatVersion or MixedVersions" # Verbose, with the test names ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py -v # Coverage of the primitive ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py -q \ --cov=capture_scheduler.lock --cov-report=term-missing ``` ### 4.7 Stress tests ```bash cd "$REPO" # Both implementations, default sizes 2 8 24 50 100, plus a latency table backend/venv/bin/python backend/scripts/stress/scheduler_lock_stress.py # High contention, repeated — a race is a probability, not an event backend/venv/bin/python backend/scripts/stress/scheduler_lock_stress.py \ --repeat 40 --sizes 24 50 100 # v2 only, machine-readable backend/venv/bin/python backend/scripts/stress/scheduler_lock_stress.py \ --impls v2 --json /tmp/stress.json # Fast smoke run backend/venv/bin/python backend/scripts/stress/scheduler_lock_stress.py \ --sizes 8 --rounds 5 --skip-latency ``` ### 4.8 CI ```bash # The gate. Thirteen checks; every one runs even after an earlier one fails. bash "$REPO/backend/scripts/ci/run_all.sh" # Local iteration only — the GitHub workflow never sets CI_SKIP CI_SKIP="v2-coverage operations-coverage" bash "$REPO/backend/scripts/ci/run_all.sh" # A specific interpreter PYTHON=/usr/bin/python3.11 bash "$REPO/backend/scripts/ci/run_all.sh" ``` ### 4.9 Logs and state ```bash export LOGS="$REPO/logs/research" # The envelope — read this first tail -f "$LOGS/scheduler_$(date +%Y_%m_%d).log" # Capture and health tail -50 "$LOGS/capture_$(date +%Y_%m_%d).log" tail -50 "$LOGS/health_$(date +%Y_%m_%d).log" # Alerts — append-only, uncapped tail -20 "$LOGS/alerts.log" # Every lock event across every day grep -h "LOCK\|lock held\|lock version conflict" "$LOGS"/scheduler_*.log # A schedule history across all days grep -h '^20' "$LOGS"/scheduler_*.log | head -100 # The machine-readable position cat "$LOGS/scheduler-state.json" python3 -m json.tool "$LOGS/scheduler-state.json" | head -40 ``` ### 4.10 Inspecting the lock ```bash export LOGS="$REPO/logs/research" # Is a lock present at all? ls -l "$LOGS/scheduler.lock" 2>/dev/null || echo "no lock — nothing is running" # The lease cat "$LOGS/scheduler.lock" # Just the format version — the first field, deliberately head -c 9 "$LOGS/scheduler.lock"; echo # Is the recorded pid still alive? ps -p "$(tr ' ' '\n' < "$LOGS/scheduler.lock" | sed -n 's/^pid=//p')" || echo "owner is gone" # Who holds it open right now (macOS / Linux) lsof "$LOGS/scheduler.lock" 2>/dev/null || echo "nobody has it open" # Any scheduler process at all pgrep -fl capture_scheduler || echo "no scheduler process" ``` --- ## 5. Scheduler Lock File ### 5.1 Where it lives ``` $LOG_DIR/scheduler.lock ``` `LOG_DIR` defaults to `/logs/research` and is resolved against the **repository**, not the process's working directory — a relative `LOG_DIR` in `backend/.env` still lands in the same place whether cron, systemd or a developer started the run. Confirm the effective path: ```bash cd "$REPO/backend" && ./venv/bin/python -m capture_scheduler config | grep LOG_DIR ``` ### 5.2 Format One line. Whitespace-separated `key=value` pairs. **Every field is found by prefix, never by position** — which is why prepending `version=` broke no reader, including v1's. ``` version=2 pid=48213 at=2026-08-05T19:00:00+05:30 token=9f3c1ab27de40518 ``` ### 5.3 Every field | Field | Constant | Example | Meaning | Load-bearing? | |---|---|---|---|---| | `version=` | `LOCK_VERSION` | `version=2` | The format declaration. Written **first**, so `head -c 9` identifies the file. A reader that does not recognise the value must refuse the lease rather than interpret the rest of the line | **Yes** — the version gate | | `pid=` | `LOCK_PID` | `pid=48213` | The process that took the lease. Written for a human; load-bearing for exactly one machine purpose — a body carrying it is a lease *somebody wrote*, which separates a v1 record from a file created and not yet written | Partly | | `at=` | `LOCK_STAMP` | `at=2026-08-05T19:00:00+05:30` | The moment the lease was taken, from the acquiring run's clock, ISO-8601 to seconds | **Yes** — staleness | | `token=` | `LOCK_TOKEN` | `token=9f3c1ab27de40518` | The fencing token, 16 hex chars, fresh per acquisition. The only thing `release` trusts | **Yes** — fencing | | `state=` | `LOCK_STATE` | `state=free` | Written **in place of unlinking** only when the platform refuses to remove a file another process holds open (Windows). A body carrying it is *not a lease* and is claimable immediately | Yes, on Windows | ### 5.4 Examples ```bash # A live v2 lease version=2 pid=48213 at=2026-08-05T19:00:00+05:30 token=9f3c1ab27de40518 # A released lock on a platform that could not unlink (Windows) version=2 state=free # A Scheduler Lock v1 lease — no version field. REFUSED at any age. pid=48213 at=2026-08-05T19:00:00+05:30 # A future build's lease. REFUSED. version=3 pid=48213 at=2026-08-05T19:00:00+05:30 token=aabbccdd # An empty file — created by O_EXCL, body not yet written. NOT a version # conflict: it is this build's own lock, microseconds old. ``` ### 5.5 How to read it | What you see | What it means | What to do | |---|---|---| | No file | Nothing is running | Nothing | | `version=2 …`, `at=` within 2 h | A run owns the window | Wait, or read the day's scheduler log | | `version=2 …`, `at=` older than 2 h, no process | A crashed run | Nothing — the next run breaks it. Delete it to catch up immediately | | `version=2 state=free` | Released, on a platform that could not unlink | Nothing — claimable at once | | No `version=` field | **Scheduler Lock v1.** Every subsequent run is refused | §11.4 / §12 | | `version=` anything but 2 | A build this one does not know | §11.4 / §12 | | Empty file | A lease being written right now | Nothing. Respected while young, broken once aged | > **Warning.** Never edit a lock file. Every safe operation on it is either > *read it* or *delete it*. There is no field a human is expected to change, and > an edited `at=` or `token=` will be read as a lie by the next contender. --- ## 6. Operational Workflow A normal weekly run, end to end. ### Step 1 — the scheduler starts `cron` / `systemd` / a developer invokes `python -m capture_scheduler run weekly`. `cli.main` loads the configuration; a bad value exits **2** with nothing run. `runner.run_job` stamps `started = config.now()` and opens the journal. **The kill switch is checked first.** If `ENABLE_AUTOMATIC_CAPTURE=false`, the run logs a line, returns `ok=True, skipped=True, exit_code=0` and **never touches the lock file**. ### Step 2 — acquire the lease `runner.acquire_lock(config, now=started, on_conflict=conflicts.append)` calls `lock.acquire` with `stale_after_seconds = STEP_TIMEOUT_SECONDS × 2`. Three outcomes: | Outcome | Log | Alert subject | Exit | |---|---|---|---| | **Acquired** | `job=weekly start — timezone=… python=…` | — | proceeds | | **Held by a live run** | `job=weekly not run: another run holds …` | `run skipped — lock held` | `1` | | **Version conflict** | `SCHEDULER LOCK — VERSION CONFLICT` block, quoting the raw body | `run skipped — scheduler lock version conflict` | `1` | Both refusals raise the alert on **five surfaces**: stderr, the scheduler log, `alerts.log`, the state file, and `ALERT_COMMAND` if one is configured. ### Step 3 — run capture ``` python backend/scripts/capture_weekly.py --days 7 --max-calls 1500 # weekly python backend/scripts/capture_weekly.py --register-only # daily ``` A subprocess, deliberately — importing `capture_weekly` would pull the broker SDK onto the scheduler's import path, and a subprocess yields an **exit code** without this module deciding what an exception means. - Retried up to `RETRY_ATTEMPTS` (4) with exponential backoff from `RETRY_BACKOFF_SECONDS` (30 s), capped at `RETRY_BACKOFF_MAX_SECONDS` (900 s). - **Exit 2 is never retried** — argparse's usage error is deterministic. - Timeout at `STEP_TIMEOUT_SECONDS` yields exit `124`; a missing command yields `127`. - The JSON report is *summarised*, not copied: expiries registered, contracts, index candles written, option sessions/contracts/candles. ### Step 4 — run health ``` python -m research_platform health ``` **Runs whatever the capture did** — a failed capture is exactly when the state of the data most needs measuring. **Never retried**: a non-zero exit means a `LOSS` or a `BREACH` is open, and re-running the query is noise on top of a finding. The severity counts are **read back from the artefact** (`docs/v2/governance/research/programme/dataset-health.json`), never recomputed. If the report's `as_of` does not match this run's date, the figures are shown, labelled, with the discrepancy as the first warning. ### Step 5 — write logs, state and the verdict - `capture_YYYY_MM_DD.log` — the capture block. - `health_YYYY_MM_DD.log` — the health block. - `scheduler_YYYY_MM_DD.log` — `JOB WEEKLY — OK` or `— UNSUCCESSFUL`. - `scheduler-state.json` — last attempt, last success/failure, consecutive failures, per step and per job. - On failure: an alert on all five surfaces. `exit_code` is `0` **only when every step succeeded.** A capture that worked followed by a health check reporting a `BREACH` is not a successful run. ### Step 6 — release the lease In a `finally:` — so it happens on success, on failure, and on an exception. ``` open(O_RDWR) → flock → verify inode → token on disk == mine? yes → unlink() the file is gone no → leave it alone a successor owns the window ``` > **Note.** A run that was displaced as stale releases **nothing**, and that is > correct behaviour, not a leak. The successor's lock stays; the successor > removes it. --- ## 7. Validation Procedures Each procedure states its commands and its expected output. All are safe to run on a development checkout; none require a broker. ### 7.1 Normal validation ```bash cd "$REPO/backend" ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py tests/test_capture_scheduler.py -q ``` **Expected:** ``` 138 passed in ...s ``` Then confirm the primitive is bound to the scheduler as expected: ```bash cd "$REPO/backend" && ./venv/bin/python - <<'PY' from capture_scheduler import lock, runner print("mutex backend :", lock.MUTEX_BACKEND) print("supported version :", lock.SUPPORTED_VERSION, runner.SUPPORTED_LOCK_VERSION) print("legacy version :", lock.LEGACY_VERSION) print("stale = timeouts ×", runner.STALE_AFTER_STEP_TIMEOUTS) print("lock file name :", runner.LOCK_NAME) PY ``` **Expected:** ``` mutex backend : flock supported version : 2 2 legacy version : 1 stale = timeouts × 2 lock file name : scheduler.lock ``` > **Warning.** `mutex backend : none` means the platform offers no file mutex. > v2 then degrades fail-closed — a stale lock is *respected*, not broken, and > clearing it by hand becomes the only recovery. Every lock test would be > measuring something else. ### 7.2 Manual execution Exercise acquire → hold → release without the broker: ```bash cd "$REPO/backend" && ./venv/bin/python - <<'PY' import tempfile from datetime import datetime, timedelta, timezone from pathlib import Path from capture_scheduler import lock IST = timezone(timedelta(hours=5, minutes=30)) now = datetime(2026, 8, 5, 19, 0, tzinfo=IST) path = Path(tempfile.mkdtemp()) / "scheduler.lock" held = lock.acquire(path, now=now, stale_after_seconds=7200) print("acquired :", held.token) print("on disk :", path.read_text().strip()) second = lock.acquire(path, now=now + timedelta(minutes=1), stale_after_seconds=7200) print("second :", second, "(None means the window is owned)") broke = lock.acquire(path, now=now + timedelta(hours=3), stale_after_seconds=7200) print("stale :", "broken, new token " + broke.token if broke else "NOT broken") lock.release(held) print("displaced release left the file:", path.exists()) lock.release(broke) print("successor release removed it :", not path.exists()) PY ``` **Expected:** ``` acquired : <16 hex chars> on disk : version=2 pid= at=2026-08-05T19:00:00+05:30 token= second : None (None means the window is owned) stale : broken, new token displaced release left the file: True successor release removed it : False ``` A real end-to-end run, with the kill switch off so nothing contacts the broker: ```bash cd "$REPO/backend" && ENABLE_AUTOMATIC_CAPTURE=false \ ./venv/bin/python -m capture_scheduler run daily; echo "exit=$?" ``` **Expected:** `daily: not run — ENABLE_AUTOMATIC_CAPTURE=false` and `exit=0`, with **no** `scheduler.lock` created. ### 7.3 Crash recovery validation ```bash cd "$REPO/backend" ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py::TestCrashRecovery -v ``` **Expected:** 3 passed — | Test | Asserts | |---|---| | `test_a_crashed_run_leaves_a_lease_that_is_respected_then_broken` | The lease outlives the process; young → respected; past the threshold → broken | | `test_a_crash_inside_the_critical_section_releases_the_mutex` | A child dying while holding the `flock` does not block the next acquirer (< 5 s) | | `test_a_restart_after_a_crash_takes_the_lock_and_runs` | Restart acquires, then releases cleanly | Manual equivalent — leave a lease behind on purpose: ```bash export LOGS="$REPO/logs/research" mkdir -p "$LOGS" printf 'version=2 pid=999999 at=%s token=deadbeefdeadbeef\n' \ "$(date -u -v-3H +%Y-%m-%dT%H:%M:%S+00:00 2>/dev/null || date -u -d '3 hours ago' +%Y-%m-%dT%H:%M:%S+00:00)" \ > "$LOGS/scheduler.lock" cat "$LOGS/scheduler.lock" cd "$REPO/backend" && ENABLE_AUTOMATIC_CAPTURE=false \ ./venv/bin/python -m capture_scheduler run daily ``` **Expected:** exit 0 and the lease untouched — the kill switch is checked before the lock, so this proves the ordering. Remove the file afterwards: ```bash rm -f "$REPO/logs/research/scheduler.lock" ``` ### 7.4 Restart validation ```bash cd "$REPO/backend" && ./venv/bin/python -m capture_scheduler status; echo "exit=$?" ``` **Expected on a fresh checkout** (exit 1 is correct here — nothing has run yet): ``` Capture Scheduler — ENABLED, Asia/Kolkata schedule : daily 18:30; weekly SAT 19:00 last register : never run last capture : never run last health : never run next run : daily 2026-08-05T18:30+05:30 (in 36m) OVERDUE : daily has never run OVERDUE : weekly has never run recent alerts : none logs : /…/logs/research exit=1 ``` **Expected on an installed, healthy host** (exit 0): ``` last register : 2026-08-05T18:30 OK 2h ago last capture : 2026-08-01T19:00 OK 4d ago last health : 2026-08-05T18:31 OK 2h ago next run : weekly 2026-08-08T19:00 (in 3d) recent alerts : none ``` Also confirm no lock survived the restart: ```bash ls -l "$REPO/logs/research/scheduler.lock" 2>/dev/null || echo "clean" ``` ### 7.5 Health validation ```bash cd "$REPO/backend" && ./venv/bin/python -m research_platform health; echo "exit=$?" ``` **Expected:** exit 0 with no open `LOSS`/`BREACH`; exit non-zero when one is open — which is a **finding**, not a scheduler fault. Remedies live in ROM-01 §9 (removed 2026-08-31 — see [the retirement record](v2/architecture/v2_research_infrastructure_retirement_2026-08-30.md)). The health step rewrites `docs/v2/governance/research/programme/dataset-health.{md,json}` — a dirty git tree after a run is expected. ### 7.6 Research Platform validation ```bash cd "$REPO/backend" ./venv/bin/python -m research_platform verify # RC-1…RC-4 ./venv/bin/python -m research_platform status # includes the scheduler block ``` **Expected:** `verify` exits 0; `status` prints the same scheduler figures as `capture_scheduler status`, because both call `cli.status_lines` — one renderer, two callers. > **Note.** The scheduler's state never changes `research_platform status`'s exit > code. A stale capture is already a `BREACH` from the health monitor, which does > gate it. One morning, one reason to fail. --- ## 8. Test Suite 138 tests across two files. Both run in CI gate 1 (`tests`) on every push. ```bash cd "$REPO/backend" ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py tests/test_capture_scheduler.py -q # → 138 passed ``` ### 8.1 Functional tests — the record and its fields `tests/test_capture_scheduler_lock.py::TestTheRecord` ```bash ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py::TestTheRecord -v ``` | Test | Validates | Why it exists | Expected | |---|---|---|---| | `test_a_v2_record_is_readable_as_a_v1_one` | `pid=` and `at=` present, in v1's order, and parse | The on-disk format is a superset of v1's — a rollback stays readable | pass | | `test_a_v1_record_is_readable_by_v2` | Every v1 field parses; no token; age computed by the same rule | *Readable* is what lets `lease_version` identify and `VersionConflict` quote a foreign lease | pass | | `test_a_v2_record_declares_its_format_first` | Field order is exactly `version, pid, at, token` | `head -c 9` must identify the file | pass | | `test_every_acquisition_gets_a_fresh_token` | Two acquisitions never share a token | Fencing is worthless with a reused token | pass | | `test_the_handle_is_path_like` | `str()`, `Path()`, `os.fspath()` all give the lock path | v1 returned a `Path`; every existing caller keeps working | pass | | `test_an_unparseable_stamp_falls_back_to_the_filesystem_clock` | Empty body, missing `at=`, malformed `at=` → filesystem clock | Both cases that produce it need real elapsed time to separate them | pass | | `test_a_stamp_from_the_future_ages_negative` | A future stamp yields a negative age and is respected | Clock skew is a report, not a licence to break someone's lock | pass | | `test_a_mixed_naive_and_aware_comparison_degrades_rather_than_raises` | Naive-vs-aware degrades to the filesystem clock | A `TypeError` must never escape a lock check | pass | ### 8.2 Concurrency tests — real forked processes `TestSimultaneousAcquisition`, `TestCrashRecovery`, `TestTheMutex`. Skipped automatically where `os.fork` is unavailable. ```bash ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py -v \ -k "SimultaneousAcquisition or CrashRecovery or TheMutex" ``` | Test | Validates | Why it exists | Expected | |---|---|---|---| | `test_exactly_one_of_n_takes_a_free_lock[2/8/24]` | N contenders on a free lock → exactly 1 winner, 0 overlaps | The baseline single-valuedness claim | pass ×3 | | `test_exactly_one_of_n_recovers_a_stale_lock[2/8/24]` | N contenders **all entitled to break** a stale lease → exactly 1 winner | **This is the v1 defect.** Under v1 this yields 2+ winners | pass ×3 | | `test_no_two_holders_overlap_under_sustained_churn` | 8 processes × 25 rounds, 0 witness collisions | One decision being single-valued does not prove the *sequence* is | pass | | `test_every_contender_terminates` | 24 processes all report inside the deadline | No deadlock, no starvation. Deliberately no wall-clock assertion — a shared runner's scheduling is not a property of this lock | pass | | `test_a_real_file_mutex_is_available_here` | `MUTEX_BACKEND in ("flock", "msvcrt")` | If this fails, every other test in the file is measuring something else | pass | | `test_two_descriptors_in_one_process_contend` | Two `open()`s in one process contend under `flock` | The reason `flock` beat `fcntl(F_SETLK)` | pass | | `test_without_a_mutex_a_stale_lock_is_respected_not_broken` | With `_enter_mutex` patched to `False`, a 10-year-old lease is **not** broken | Pins the fail-closed degradation | pass | | `test_a_replaced_file_is_reopened_rather_than_locked_blind` | Identity revalidated **after** the lock; retry on a mismatch | The one way a mutex taken on a *path* can be undone | pass | | `test_a_crashed_run_leaves_a_lease_that_is_respected_then_broken` | Crash → lease survives; young respected, stale broken | v1's semantics, deliberately preserved | pass | | `test_a_crash_inside_the_critical_section_releases_the_mutex` | A child dying holding the `flock` does not block the next acquirer | Why the mutex is the kernel's and not a file of our own | pass | | `test_a_restart_after_a_crash_takes_the_lock_and_runs` | Restart acquires, then releases | The recovery path end to end | pass | > **Note — how an overlap is observed.** A process that believes it holds the > lock creates a `witness` file with `O_CREAT|O_EXCL` and drops it on release. > Two owners overlapping is therefore a `FileExistsError` **raised by the kernel > at the instant it happens**, not a count reconciled afterwards. Contenders are > released against a shared wall-clock deadline they busy-wait to — anything that > sleeps them apart measures a schedule rather than a race. ### 8.3 Regression tests — fencing, and the behaviour v2 preserved `TestFencing` (lock file) and `TestTheLock` (scheduler file). ```bash ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py::TestFencing -v ./venv/bin/python -m pytest tests/test_capture_scheduler.py::TestTheLock -v ``` | Test | Validates | Why it exists | Expected | |---|---|---|---| | `TestFencing::test_a_broken_owner_does_not_release_the_new_holder` | A displaced run's `release` leaves the successor's lock | **v1's second race**, closed | pass | | `TestFencing::test_release_never_removes_a_record_it_cannot_show_is_its_own` | An untokened record is not deleted either | The rule stated absolutely | pass | | `TestFencing::test_a_vanished_path_is_never_the_file_we_locked` | `_same_file` is False once the path is unlinked | The unlinked-inode case, directly | pass | | `TestFencing::test_release_reopens_when_the_file_is_replaced_under_it` | Release runs the same reopen loop as acquire | It must not delete a file it never confirmed | pass | | `TestFencing::test_a_bare_path_still_releases_unconditionally` | `release(Path)` always removes | v1's contract, and what an operator clearing by hand means | pass | | `TestFencing::test_releasing_nothing_is_not_an_error` | `release(None)` and a nonexistent path are no-ops | Release is called from a `finally:` | pass | | `TestFencing::test_the_free_marker_is_claimable_at_once` | `state=free` is claimed immediately | It must say exactly what an absent file says | pass | | `TestTheLock::test_a_second_run_does_not_start_while_the_first_holds_the_lock` | The second run executes no step | The scheduler-level contract | pass | | `TestTheLock::test_the_skipped_run_alerts_rather_than_passing_quietly` | A skip alerts | "Never silently ignore failures" | pass | | `TestTheLock::test_a_stale_lock_is_broken` | A crashed run does not wedge every capture | A missed window is the failure this package prevents | pass | | `TestTheLock::test_staleness_depends_on_the_interval_not_on_the_epoch` | Same verdict at any epoch | The rule is an interval, not a date | pass | | `TestTheLock::test_the_threshold_is_two_step_timeouts_exactly` | `≤ 2×` held, `+1 s` broken | The boundary, pinned | pass | | `TestTheLock::test_a_lock_stamped_in_the_future_is_never_broken` | Skew is respected | Same reason as §8.1 | pass | | `TestTheLock::test_the_lock_records_the_runs_clock` | The stamp is the run's injected clock | Both ends of the comparison are one clock | pass | | `TestTheLock::test_a_lock_being_written_right_now_is_respected` | An empty body is respected while young | The `O_EXCL` gap | pass | | `TestTheLock::test_a_lock_left_unstamped_by_a_crash_is_still_broken_eventually` | An unstamped lock ages out on the filesystem clock | The opposite case of the same body | pass | | `TestTheLock::test_the_lock_is_released_after_a_run` | No file survives a completed run | The `finally:` | pass | ### 8.4 Versioning tests `TestTheFormatVersion` and `TestMixedVersionsReachTheOperator`. ```bash ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py -v \ -k "TheFormatVersion or MixedVersions" ``` | Test | Validates | Why it exists | Expected | |---|---|---|---| | `test_a_v2_lease_is_judged_exactly_as_before` | Our own lease: live respected, stale broken | The compatibility claim | pass | | `test_the_free_marker_declares_its_version_too` | `version=2 state=free` claims with no conflict | `state=` is read before `version=` — a free marker is the absence of a lease | pass | | `test_an_unversioned_lease_is_scheduler_lock_v1` | `pid=` alone or `at=` alone → v1, `inferred=True` | The one inference this module makes | pass | | `test_a_declared_version_is_read_not_inferred` | `version=2` → supported; `version=7` → 7 | Declared beats inferred | pass | | `test_an_unreadable_version_is_unsupported_rather_than_v1` | `version=next` → unsupported, not v1 | v1 never wrote the field, so anything that did is not v1 | pass | | `test_a_file_created_and_not_yet_written_is_not_a_lease` | An empty body → not a lease, **no conflict** | Calling it v1 would wedge the scheduler on a race `age_seconds` already resolves | pass | | `test_an_unsupported_lease_is_refused_and_reported` (5 params) | v1, `version=1`, `version=3`, `version=999`, `version=next` all refused and reported | Forward *and* backward incompatibility | pass ×5 | | `test_a_v1_lease_is_refused_however_old_it_is` | Refused at 60 s, 2 h, 1 day, 1 year | The one deliberate departure from v1's staleness rule | pass | | `test_the_refused_lease_is_left_byte_for_byte_untouched` | Bytes identical before and after | Not broken, not upgraded, not rewritten — reported | pass | | `test_refusal_is_silent_without_a_callback` | No `on_conflict` → reads as an ordinary held lock | Every existing caller is unaffected | pass | | `test_the_callback_runs_outside_the_critical_section` | Re-entering `acquire` from `on_conflict` returns rather than hangs | The liveness argument depends on no user code inside the mutex — this would **deadlock** if it were violated | pass | | `test_the_diagnostic_names_what_was_found_and_what_to_do` | Summary names both versions, the file, "left untouched", and the remedy — on **one line** | It is an `alerts.log` field | pass | | `test_the_diagnostic_says_the_version_was_inferred` | "the file said v1" is distinguishable from "the file said nothing, so it is v1" | An inference must be labelled as one | pass | | `TestMixedVersions::test_a_v1_lock_stops_the_run_and_says_why` | `ok=False`, `skipped=True`, `exit_code=1`, `reason="lock version conflict"`; alert written | The conflict must reach a human | pass | | `TestMixedVersions::test_the_log_shows_both_versions_and_the_remedy` | The `SCHEDULER LOCK — VERSION CONFLICT` block | The operator's first surface | pass | | `TestMixedVersions::test_a_future_version_stops_the_run_the_same_way` | `version=3` behaves identically | Forward compatibility | pass | | `TestMixedVersions::test_the_lease_survives_the_refused_run` | The file is byte-identical after `run_job` | The file you inspect is the one that caused the alert | pass | | `TestMixedVersions::test_an_ordinary_held_lock_still_reads_as_a_held_lock` | A v2 lease held by a peer keeps v1's message and reason; the word "version" appears nowhere in the alert | The regression guard on the other branch | pass | ### 8.5 Compatibility tests The binding between config and primitive, and the boundaries the package must not cross. ```bash ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py::TestTheSchedulerBinding -v ./venv/bin/python -m pytest tests/test_capture_scheduler.py::TestBoundaries -v ``` | Test | Validates | Expected | |---|---|---| | `TestTheSchedulerBinding::test_stale_is_two_step_timeouts` | `STALE_AFTER_STEP_TIMEOUTS == 2`; boundary at `2×` and `2×+1` | pass | | `TestTheSchedulerBinding::test_the_lock_lives_beside_the_logs` | `lock_path == log_dir / "scheduler.lock"`; the handle is that path | pass | | `TestTheSchedulerBinding::test_a_run_that_was_displaced_releases_nothing` | Fencing, through the scheduler's own API | pass | | `TestTheSchedulerBinding::test_the_supported_version_is_re_exported` | `runner.SUPPORTED_LOCK_VERSION == lock.SUPPORTED_VERSION == 2` | pass | | `TestBoundaries::test_it_imports_no_broker_and_no_replay` | The scheduler's import path never reaches the Angel One SDK | pass | | `TestBoundaries::test_it_evaluates_nothing` | It spends no α and evaluates no endpoint | pass | | `TestBoundaries::test_it_fixes_nothing` | RR-1 — it reports and repairs nothing | pass | | `TestBoundaries::test_the_research_platform_is_still_a_leaf` | `research_platform` is not imported by production | pass | | `TestDeploymentArtefacts::test_the_committed_example_matches_the_renderer` | `deploy/capture-scheduler/` equals a fresh render | pass | ### 8.6 Stress tests Not part of CI — see §9. The CI suite proves single-valuedness at **2, 8 and 24** processes; the harness answers what happens at **50 and 100**, and what the same experiment does to **v1**. --- ## 9. Stress Testing ### 9.1 What it is [`backend/scripts/stress/scheduler_lock_stress.py`](../backend/scripts/stress/scheduler_lock_stress.py) forks *N* real processes, releases them against a shared wall-clock deadline they busy-wait to, and counts winners and overlaps. It runs the **identical experiment** against a transcription of the v1 algorithm, which lives in the harness, is imported by nothing, and is there as the **exhibit** rather than as an implementation. > **Tip.** A test that only ever shows green proves the new code passes. Running > the old algorithm through the same instrument is what shows there was something > to fix. ### 9.2 How to run it ```bash cd "$REPO" # Default: both implementations, sizes 2 8 24 50 100, 20 churn rounds, latency table backend/venv/bin/python backend/scripts/stress/scheduler_lock_stress.py ``` ### 9.3 Parameters | Flag | Default | What it does | Recommended | |---|---|---|---| | `--sizes` | `2 8 24 50 100` | Process counts per scenario | Keep the default for a release; `8` for a smoke run | | `--rounds` | `20` | Acquire/release cycles per process in the churn scenarios | `25` for a release campaign | | `--impls` | `v2 v1` | Which implementations to exercise | Both for evidence; `v2` alone for a quick check | | `--repeat` | `1` | Times to repeat every scenario | **`40` before a release.** A race is a probability, not an event — one clean pass is weak evidence | | `--json` | none | Write every record as JSON | Always, when producing evidence | | `--skip-latency` | off | Skip the 2 000-iteration latency table | Only when iterating | ### 9.4 Recommended invocations ```bash cd "$REPO" # Before a release touching the scheduler — the full campaign (minutes, not seconds) backend/venv/bin/python backend/scripts/stress/scheduler_lock_stress.py \ --repeat 40 --rounds 25 --json /tmp/scheduler-lock-stress.json # High contention only backend/venv/bin/python backend/scripts/stress/scheduler_lock_stress.py \ --repeat 40 --sizes 24 50 100 # Smoke run while iterating backend/venv/bin/python backend/scripts/stress/scheduler_lock_stress.py \ --impls v2 --sizes 8 --rounds 5 --skip-latency ``` ### 9.5 Expected output ``` ── v2 (flock) scenario procs rounds trials winners overlaps secs verdict free lock 2 1 1 1 0 0.05 PASS — one owner throughout stale recovery 2 1 1 1 0 0.05 PASS — one owner throughout sustained churn 2 20 1 47 0 0.11 PASS — one owner throughout stale + churn 2 20 1 45 0 0.11 PASS — one owner throughout ... ── v1 (no mutex) scenario procs rounds trials winners overlaps secs verdict free lock 100 1 1 1 0 0.42 PASS — one owner throughout stale recovery 100 1 1 2 1 0.44 RACE — 1 concurrent holder(s) ... ── uncontended latency impl median mean p99 v2 142.1µs 156.8µs 280.1µs v1 91.8µs 100.2µs 201.0µs ════ summary ════ v2: 0 race(s) observed v1: 3 race(s) observed (the defect this change removes; a zero here means the window did not open on this run, not that it is absent) ``` **Exit code is non-zero only if v2 raced.** A v1 race is reported, not failed — it is the finding. ### 9.6 The four scenarios | Scenario | Seed | Question it answers | |---|---|---| | `free lock` | none | Do *N* simultaneous contenders produce one owner? | | `stale recovery` | a lease stamped 24 h ago | Do *N* processes **all entitled to break it** produce one owner? **This is where v1 fails.** | | `sustained churn` | none, acquire/release ×R | Does the *sequence* stay single-valued? | | `stale + churn` | stale lease, acquire/release ×R | Recovery racing release, repeatedly | ### 9.7 How to interpret results | Reading | Meaning | Action | |---|---|---| | `v2: 0 race(s)` | The expected result on every run | None | | **`v2: ≥1 race(s)`** | **A defect in the primitive.** Overlaps are kernel-observed, not inferred | **Stop.** Capture the `--json`, do not patch `lock.py` — open a v3 design | | `v1: 0 race(s)` | The window did not open **on this run** | Not evidence of absence. Raise `--repeat` | | `v1: N race(s)` on the stale rows | Expected — the defect being demonstrated | None; this is the exhibit working | | `child_errors` > 0 | A forked child raised. The instrument is broken, not the lock | Read `raised` in the JSON | | `never_finished` > 0 | Processes killed on the deadline | Deadlock or a badly loaded host. Re-run on a quiet machine before concluding | | A `free lock` row racing under v1 | Would contradict the analysis — `O_EXCL` was never the defect | Investigate the harness | Reference results from the frozen campaign (darwin 25.4.0, APFS, Python 3.11, `flock`; 40 trials per cell, 25 rounds, ~100 000 acquisitions per implementation): | | v2 | v1 | |---|---|---| | Trials | 800 | 800 | | Trials that raced | **0** | **21** | | Duplicate acquisitions | **0** | **13** | | Overlap events | **0** | **28** | The v1 failures are confined to the two rows that require *breaking* a lock; the `free lock` row is clean for v1 too, because `open(O_CREAT|O_EXCL)` was never the defect. --- ## 10. Troubleshooting Guide | Problem | Possible cause | Diagnostic command | Resolution | |---|---|---|---| | **Scheduler won't start** — `ModuleNotFoundError: capture_scheduler` | `backend/` not importable: wrong cwd, wrong interpreter, or `PYTHONPATH` unset | `cd "$REPO/backend" && ./venv/bin/python -c "import capture_scheduler"` | Run from `backend/`, or set `PYTHONPATH=backend`. Set `PYTHON` explicitly if the venv is unusual | | **Scheduler won't start** — exits 2 with `FAIL: =…` | A configuration value cannot be honoured | `cd "$REPO/backend" && ./venv/bin/python -m capture_scheduler config` | Fix the named variable. Nothing ran — this is the intended fail-closed | | **Nothing runs and there is no log** | The process never started. cron runs with almost no environment | `crontab -l`; `journalctl -u research-capture-daily.service` | Re-render and reinstall the artefact; check the absolute interpreter path | | **Every run is skipped, `reason=lock version conflict`** | Two implementations of the run lock are deployed, or the file predates 2026-08-05 | `head -c 9 "$LOGS/scheduler.lock"; cat "$LOGS/scheduler.lock"` | §11.4. Stop every instance, finish the deployment, `rm` the lock, re-run. **Waiting never clears this** | | **`run skipped — lock held`, repeatedly** | A run really is in flight, or a lease is aging out | `cat "$LOGS/scheduler.lock"`; `ps -p `; `pgrep -fl capture_scheduler` | If the owner lives: wait. If not: nothing — the next run breaks it past 2 h. Delete to catch up now | | **Lock exists forever** | The lease is unsupported (refused at any age), or `MUTEX_BACKEND == "none"` so breaking is refused | `head -c 9 "$LOGS/scheduler.lock"`; `./venv/bin/python -c "from capture_scheduler import lock; print(lock.MUTEX_BACKEND)"` | Version conflict → §12. No mutex → clear by hand; that is the documented recovery | | **Unsupported version** — `version=3` or `version=next` | A newer build wrote it, or the file is corrupt | `cat "$LOGS/scheduler.lock"`; `grep "VERSION CONFLICT" "$LOGS"/scheduler_*.log` | Deploy one implementation everywhere, then `rm` the lock once nothing is in flight | | **Capture failed** — consecutive failures on `capture`/`register` | Broker or database unreachable; credentials expired; `MAX_CALLS` exhausted | `tail -80 "$LOGS/capture_$(date +%Y_%m_%d).log"` | Fix the cause, then re-run the job by hand. **Not a lock problem** — the lock released correctly in the `finally:` | | **Capture failed with exit 2** | A usage error — a malformed flag. Deliberately not retried | `grep "usage error" "$LOGS"/scheduler_*.log` | Fix the command in the config, re-render if it is in an artefact | | **Capture failed with exit 124** | The step hit `STEP_TIMEOUT_SECONDS` | `grep "timed out after" "$LOGS"/capture_*.log` | Investigate the hang. Raising the timeout also raises the stale threshold — it is `×2` | | **Health failed** — consecutive failures on `health` | A `LOSS` or a `BREACH` is open. **The scheduler is working** | `tail -60 "$LOGS/health_$(date +%Y_%m_%d).log"` | ROM-01 §9 (removed 2026-08-31 — see [the retirement record](v2/architecture/v2_research_infrastructure_retirement_2026-08-30.md)) owns every remedy. Never retried, by design | | **Health summary is yesterday's** | The health step could not write; the run reports the previous report's counts, labelled | `grep "the health report on disk is dated" "$LOGS"/*.log` | Treat the exit code as the only signal for that run. Fix the database connection | | **CI failure** — `tests` gate red on a scheduler test | Behaviour changed near the lock | `./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py -v` | Read the failing assertion. **Never weaken it to pass** — the assertions are the guarantees | | **CI failure** — `test_a_real_file_mutex_is_available_here` | The runner's filesystem or platform offers no `flock` | `./venv/bin/python -c "from capture_scheduler import lock; print(lock.MUTEX_BACKEND)"` | Do not skip it. Move the checkout to a local filesystem — every other lock test would be measuring something else | | **CI failure** — forking tests hang or time out | A shared runner under heavy load, or a genuine deadlock | Re-run alone: `pytest tests/test_capture_scheduler_lock.py -k SimultaneousAcquisition -v` | Reproduce on a quiet host before concluding. `_contend` kills at its deadline and fails on it | | **Lock contention** — two jobs at once | A cron line **and** a systemd timer both installed | `crontab -l`; `systemctl list-timers 'research-capture-*'` | Pick one deployment mode. The lock makes this safe, not correct | | **Crash** — lock present, no process | A run was killed | `cat "$LOGS/scheduler.lock"`; `ps -p ` | Nothing required — broken automatically past 2 h. `rm` to catch up immediately | | **Permission issue** — `PermissionError` on the lock path | `LOG_DIR` not writable by the scheduler's user | `ls -ld "$LOGS"`; `id` | `chown`/`chmod` the log directory. The lock is created `0o644` | | **Filesystem issue** — the lock lives on NFS or a network mount | `flock` semantics are not guaranteed on exotic servers | `df -T "$LOGS"` (Linux) / `mount \| grep research` | Move `LOG_DIR` to a local filesystem. `logs/` is local by default and should stay one | | **Every run leaves modified files in git** | Expected — the health step regenerates `programme/dataset-health.{md,json}` | `git status docs/v2/governance/research/programme/` | Nothing. Commit or discard per your workflow | | **`status` exits 1 but everything looks fine** | `OVERDUE`/`FAILING` lines | `./venv/bin/python -m capture_scheduler status` | On a fresh install both jobs read as never run; clears after the first success of each | --- ## 11. Version Compatibility ### 11.1 v1 — Scheduler Lock v1 (before 2026-08-05) | Property | Value | |---|---| | Record | `pid=48213 at=2026-08-05T19:00:00+05:30` | | Format field | **none** — the absence is the identification | | Mechanism | `open(O_CREAT\|O_EXCL)`; a stale lock broken by `stat` → judge → `unlink` → `open(O_EXCL)` | | Fencing | none — `release` unlinked unconditionally | | Known defects | Two runs could both break the same stale lock; a displaced owner deleted its successor's lock | | Status | **Superseded.** Recognised, quoted in diagnostics, never judged | ### 11.2 v2 — current | Property | Value | |---|---| | Record | `version=2 pid=48213 at=…+05:30 token=9f3c1ab27de40518` | | Mechanism | Every read-decide-write inside `flock(LOCK_EX)` on the same file | | Fencing | `token=`, checked on release | | Staleness | `2 × STEP_TIMEOUT_SECONDS`, on the run's own clock — **unchanged from v1** | | Public API | `acquire_lock(config, now=…, on_conflict=…)`, `release_lock(held)` — names, arguments and semantics preserved; `on_conflict` is the one new (optional) keyword | | Status | **Current and frozen** | ### 11.3 Unsupported versions Anything whose `version=` is not `2`: `version=1`, `version=3`, `version=999`, `version=next`, and an unversioned v1 record. **Rule:** a lease this build does not support is **never judged, never broken and never rewritten** — it is reported and the run is skipped. > **Warning — why refusal, and not "break it and move on"?** > Two implementations of the run lock cannot serialise against each other. v1 > breaks a stale lock *without* taking the mutex v2 takes, and releases it > *without* checking the fencing token v2 writes. A v2 run that broke a v1 lease > would be doing precisely the unsafe thing v2 exists to stop — against a peer > that can still delete the lock underneath it. A skipped window is recoverable; > two concurrent captures against one broker session are not. ### 11.4 Behaviour table | On disk | Classified as | `acquire` returns | `on_conflict` fires | Run outcome | |---|---|---|---|---| | `version=2 …`, age ≤ 2 h | this build's lease, live | `None` | no | skipped, `reason="lock held"`, exit 1 | | `version=2 …`, age > 2 h | this build's lease, stale | a fresh `LockHandle` | no | proceeds | | `version=2 state=free` | not a lease | a fresh `LockHandle` | no | proceeds | | `pid=… at=…` (no `version=`) | **v1, inferred** | `None` at **any** age | **yes** | skipped, `reason="lock version conflict"`, exit 1 | | `version=1 …` | v1, declared | `None` | **yes** | as above | | `version=3 …` | a future build | `None` | **yes** | as above | | `version=next …` | unreadable → unsupported (**not** v1) | `None` | **yes** | as above | | empty / whitespace | **not a lease** — this build's own file, mid-write | respected while young, broken once aged | **no** | ordinary staleness applies | ### 11.5 Upgrade — v1 → v2 **There is no migration step.** The change is source-only; the on-disk format is a superset of v1's. ```bash # 1. Deploy between windows — the scheduler runs twice a day. git pull && cd "$REPO/backend" && ./venv/bin/pip install -r requirements.txt # 2. Confirm no run is in flight, and clear any pre-v2 lease. pgrep -fl capture_scheduler || echo "nothing running" head -c 9 "$REPO/logs/research/scheduler.lock" 2>/dev/null # blank/absent → v1 or none rm -f "$REPO/logs/research/scheduler.lock" # 3. Verify. cd "$REPO/backend" && ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py -q ./venv/bin/python -m capture_scheduler status ``` No artefact re-render is needed: the cron line, the systemd unit and the GitHub workflow are unchanged. No new configuration variable exists. If you skip step 2, nothing breaks unsafely — the first v2 run refuses to start and tells you exactly this, on five surfaces. ### 11.6 Rollback — v2 → v1 ```bash git revert # or check out the prior revision rm -f "$REPO/logs/research/scheduler.lock" ``` A v2 lock file is readable by v1: `version=` and `token=` are ignored by v1's whitespace-token parser, which finds `at=` by prefix and not by position. Clearing the file anyway is cheaper than reasoning about it. > **Warning.** Rolling back reintroduces both v1 defects. It is an emergency > measure, not a supported operating position. ### 11.7 Running both concurrently **Do not** — but it is now *detected* rather than assumed against. A v2 run that finds a v1 lease leaves it alone, skips, and alerts with both versions and the remedy. Deploy between windows; if you do not, the second implementation refuses to start rather than racing. ### 11.8 When a version bump is warranted Bump `SUPPORTED_VERSION` when the **meaning** of a field changes — not when one is added. A reader that ignores an unknown field still judges the lease correctly; a bump costs a coordinated stop of every scheduler instance. --- ## 12. Emergency Recovery ### 12.1 When to remove a lock by hand | Situation | Remove? | Why | |---|---|---| | Version conflict (`version=` absent or ≠ 2) | **Yes**, after stopping every instance | Nothing else clears it. Waiting never will | | `MUTEX_BACKEND == "none"` and a stale lease is present | **Yes** | The documented recovery for the fail-closed path | | A crashed run and you need the window **now** rather than in 2 h | **Yes** | It only accelerates what would happen anyway | | A crashed run and the next window is acceptable | No | The next run breaks it automatically past `2 × STEP_TIMEOUT_SECONDS` | | A run is genuinely in flight | **No** — see below | Removing it lets a second run start alongside the first | | You cannot tell whether a run is in flight | **No** | Establish it first — §12.2 | | Because a test is failing | **No** | Tests use `tmp_path`; they never touch `logs/research/` | > **Warning.** Deleting the lock does not stop, signal or interrupt a running > job. It removes the *only* thing preventing a second job from starting > alongside it. That is why "is anything running?" is a step, not an assumption. ### 12.2 Verify no scheduler is running Run all four. If any says yes, do not delete. ```bash export LOGS="$REPO/logs/research" # 1. Any scheduler process on this host pgrep -fl capture_scheduler || echo "1 OK — no scheduler process" # 2. The pid recorded in the lease PID=$(tr ' ' '\n' < "$LOGS/scheduler.lock" 2>/dev/null | sed -n 's/^pid=//p') [ -n "$PID" ] && (ps -p "$PID" && echo "2 STOP — owner is alive" || echo "2 OK — owner is gone") # 3. Anyone holding the file open lsof "$LOGS/scheduler.lock" 2>/dev/null && echo "3 STOP — held open" || echo "3 OK" # 4. The scheduler's own view — the last log line should be a completed verdict tail -5 "$LOGS/scheduler_$(date +%Y_%m_%d).log" 2>/dev/null ``` On a multi-host deployment, repeat on **every** host, and stop the schedule first: ```bash crontab -l # cron — comment the lines out systemctl stop research-capture-daily.timer research-capture-weekly.timer systemctl list-timers 'research-capture-*' ``` ### 12.3 The recovery ```bash # 1. Read what you are about to delete — keep it for the incident record cat "$REPO/logs/research/scheduler.lock" # 2. Remove it rm -f "$REPO/logs/research/scheduler.lock" # 3. Confirm a clean acquire cd "$REPO/backend" && ./venv/bin/python -m capture_scheduler status # 4. Catch up — weekly registers AND backfills, so it subsumes a missed daily cd "$REPO/backend" && ./venv/bin/python -m capture_scheduler run weekly # 5. Re-enable the schedule if you stopped it systemctl start research-capture-daily.timer research-capture-weekly.timer ``` ### 12.4 Version-conflict recovery, in order The remedy the alert itself prints: 1. **Stop every scheduler instance on every host.** 2. **Complete the deployment** so that one implementation of the run lock remains. 3. **Delete `logs/research/scheduler.lock`** once no run is in flight. 4. **Re-run the job** — `python -m capture_scheduler run weekly`. ### 12.5 Risks | Action | Risk | Mitigation | |---|---|---| | Deleting a lock while a run is live | Two concurrent captures against one broker session; duplicated calls against `MAX_CALLS`; two writers into the registry | §12.2, all four checks, on every host | | Editing a lock file instead of deleting it | The next contender reads a lie. A hand-set `at=` can hold the window open indefinitely | Never edit. Delete or leave | | Deleting on one host in a multi-host deployment | The other host's run is unprotected | Stop the schedule everywhere first | | `rm -rf logs/research/` | Destroys the run ledger and every log — the evidence for the incident you are handling | Delete the lock file only, by name | | Recreating a lock "to be safe" | A hand-written lease has no valid token; the next release will refuse to remove it | Never create one | --- ## 13. Performance Notes ### 13.1 Uncontended acquisition latency 2 000 iterations of `acquire` + `release`, darwin 25.4.0 / APFS / Python 3.11: | | median | mean | p99 | |---|---|---|---| | v1 | 91.8 µs | 100.2 µs | 201.0 µs | | **v2** | **142.1 µs** | 156.8 µs | 280.1 µs | **+50 µs per acquisition.** The `flock`/`unlock` pair is a few microseconds; the rest is the `fsync` v2 performs on the record — v1 wrote through `fdopen` and never synced, so a power cut could lose a lease it had already granted. The scheduler acquires the lock **twice a day**. Fifty microseconds twice a day is 36 milliseconds a century. Reproduce: ```bash cd "$REPO" backend/venv/bin/python backend/scripts/stress/scheduler_lock_stress.py \ --sizes 2 --rounds 1 ``` ### 13.2 Contention behaviour Under contention the comparison **inverts**. At 100 processes × 25 rounds: | | successful acquisitions | wall clock | |---|---|---| | **v2** | **2 125** | 8.5 s | | v1 | 1 091 | 7.8 s | v2 did roughly twice the work in the same time. A v1 contender that loses spends a `stat`, a `read` and a second `stat` finding out; a v2 contender gets a definite answer from one critical section. **v1's apparent speed is contenders failing to make progress, not work being done.** ### 13.3 Stress behaviour - **No deadlock.** The mutex is held for two file reads and one write — never across a job, never across a subprocess. - **No starvation.** Every one of 24 contenders reports inside the deadline in CI; every one of 100 does in the harness. - **Retry bound.** `REOPEN_ATTEMPTS = 64` bounds re-opens when the file is replaced underneath a contender. Exhausting it returns "not acquired", which skips a run — the fail-closed direction. - Contention scales cleanly from 2 to 100 processes with zero overlaps across 800 trials. ### 13.4 Normal performance and filesystem expectations | Aspect | Expectation | |---|---| | Acquisitions per day | 2 (daily + weekly, once a week) | | Lock held (lease) | The whole job — minutes to an hour | | Mutex held | Microseconds, twice per acquire and once per release | | Lock file size | ~70 bytes | | Filesystem | **Local.** `flock` is required for the stale-recovery path and is present on every local filesystem on Linux and macOS | | NFS | Linux maps `flock` onto POSIX locks over NFSv3+ and it works; older or exotic servers may not. `logs/` is local by default and should stay one | --- ## 14. Developer Best Practices > **Warning — the module is frozen.** `capture_scheduler/lock.py` is not > modified incrementally. A defect or a new requirement opens a **Scheduler Lock > v3 design**, reviewed before implementation, not an edit to v2. 1. **Never edit a lock file by hand.** Read it or delete it. There is no field a human is expected to change, and an edited `at=` or `token=` is read as a lie by the next contender. 2. **Never create a lock file by hand.** A hand-written lease has no valid token; the next `release` will refuse to remove it, and you have manufactured a lock that outlives every run. 3. **Never bypass the lock.** Do not call `capture_weekly.py` directly on a host with an installed schedule when a window may be open, and never add a code path that runs a job without `acquire_lock`. 4. **Never mix Scheduler Lock versions.** One implementation per deployment. Deploy between windows. If you cannot, the mixed case now refuses to start — treat that alert as a deployment bug, not as an outage. 5. **Never break a foreign lease "to unblock things".** Refusal is the safe verdict. Stop the instances, finish the deployment, then clear the file. 6. **Always run the lock tests after any change near the lock** — including changes to `runner.py`, `config.py`'s `STEP_TIMEOUT_SECONDS`, or `LOG_DIR` resolution: ```bash cd "$REPO/backend" && ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py -q ``` 7. **Always run stress tests before a release that touches the scheduler:** ```bash backend/venv/bin/python backend/scripts/stress/scheduler_lock_stress.py --repeat 40 ``` Zero v2 races is the release criterion. 8. **Always run CI before committing:** ```bash bash "$REPO/backend/scripts/ci/run_all.sh" ``` Thirteen gates, all of them, reporting every failure rather than stopping at the first. 9. **Never weaken a test assertion to make a change pass.** The assertions in `test_capture_scheduler_lock.py` *are* the guarantees in §1.4. 10. **Never skip `test_a_real_file_mutex_is_available_here`.** If it fails, every other test in the file is measuring something else. 11. **Do not "improve" the v1 transcription** in the stress harness. It is the exhibit — evidence that there was something to fix — not a second implementation. 12. **Keep `LOG_DIR` on a local filesystem.** A scheduler whose lock lives on a network mount has an untested primitive underneath it. 13. **Remember the timeout coupling.** Raising `STEP_TIMEOUT_SECONDS` raises the stale threshold too — it is `×2`. A 6-hour step timeout means a crashed run's lock is respected for 12 hours. 14. **Preserve the fail-closed direction.** Every refusal in this design skips a run. A skipped window is recoverable; two concurrent captures against one broker session are not. 15. **`on_conflict` must never do real work.** It runs outside the mutex by construction, and a test proves it by re-entering `acquire` from inside it. 16. **The scheduler fixes nothing (RR-1).** Do not add a path that repairs what it finds — not a re-run, not a lock repair, not a widened window. --- ## 15. Quick Reference ```bash export REPO=/Users/macbook/Documents/pgProject/pg3/ns/AI_Trade export LOGS="$REPO/logs/research" ``` | Task | Command | |---|---| | **Status** | `cd "$REPO/backend" && ./venv/bin/python -m capture_scheduler status` | | **Config + provenance** | `cd "$REPO/backend" && ./venv/bin/python -m capture_scheduler config` | | **Health** | `cd "$REPO/backend" && ./venv/bin/python -m research_platform health` | | **Verify programme** | `cd "$REPO/backend" && ./venv/bin/python -m research_platform verify` | | **Run daily** | `cd "$REPO/backend" && ./venv/bin/python -m capture_scheduler run daily` | | **Run weekly** | `cd "$REPO/backend" && ./venv/bin/python -m capture_scheduler run weekly` | | **Lock tests** | `cd "$REPO/backend" && ./venv/bin/python -m pytest tests/test_capture_scheduler_lock.py -q` | | **All scheduler tests** | `cd "$REPO/backend" && ./venv/bin/python -m pytest tests/test_capture_scheduler{,_lock}.py -q` | | **Stress** | `backend/venv/bin/python backend/scripts/stress/scheduler_lock_stress.py --repeat 40` | | **CI** | `bash "$REPO/backend/scripts/ci/run_all.sh"` | | **Scheduler log** | `tail -f "$LOGS/scheduler_$(date +%Y_%m_%d).log"` | | **Capture log** | `tail -50 "$LOGS/capture_$(date +%Y_%m_%d).log"` | | **Health log** | `tail -50 "$LOGS/health_$(date +%Y_%m_%d).log"` | | **Alerts** | `tail -20 "$LOGS/alerts.log"` | | **State** | `python3 -m json.tool "$LOGS/scheduler-state.json"` | | **Inspect lock** | `cat "$LOGS/scheduler.lock"` | | **Lock version only** | `head -c 9 "$LOGS/scheduler.lock"; echo` | | **Who's running** | `pgrep -fl capture_scheduler` | | **Emergency clear** | `pgrep -fl capture_scheduler \|\| rm -f "$LOGS/scheduler.lock"` | | **Render cron** | `cd "$REPO/backend" && ./venv/bin/python -m capture_scheduler render cron` | **Exit codes:** `0` success or kill switch · `1` step failed / health finding / lock held / version conflict · `2` bad configuration, nothing ran. **Key numbers:** supported version `2` · stale after `2 × STEP_TIMEOUT_SECONDS` = **7200 s** · token 16 hex chars · `REOPEN_ATTEMPTS` 64 · 56 lock tests + 82 scheduler tests = **138**. --- ## 16. Appendix ### 16.1 Scheduler directory layout ``` backend/capture_scheduler/ __init__.py __main__.py puts backend/ on sys.path, then cli.main cli.py run | status | config | render; exit codes config.py SchedulerConfig, parsing, precedence, the schedule deploy.py cron / systemd / github / env renderers journal.py the three dated log streams lock.py Scheduler Lock v2 — the primitive (FROZEN) runner.py job execution; the config → primitive binding state.py scheduler-state.json backend/tests/ test_capture_scheduler.py 82 tests test_capture_scheduler_lock.py 56 tests backend/scripts/stress/ scheduler_lock_stress.py the harness and the v1 exhibit deploy/capture-scheduler/ README.md research-capture.crontab research-capture.systemd research-capture.github-workflow.yml capture-scheduler.env.example ``` ### 16.2 Generated files | File | Generated by | Hand-edit? | |---|---|---| | `deploy/capture-scheduler/research-capture.crontab` | `render cron --repo-root /srv/ai_trade` | **No** — a test compares it against the renderer | | `deploy/capture-scheduler/research-capture.systemd` | `render systemd --repo-root /srv/ai_trade` | **No** | | `deploy/capture-scheduler/research-capture.github-workflow.yml` | `render github --repo-root /srv/ai_trade` | **No.** Deliberately not installed into `.github/workflows/` | | `deploy/capture-scheduler/capture-scheduler.env.example` | `render env` | **No** | | `docs/v2/governance/research/programme/dataset-health.{md,json}` | `research_platform health` (the scheduler's second step) | **No** | ### 16.3 Log files | File | Contents | Rotation | |---|---|---| | `logs/research/scheduler_YYYY_MM_DD.log` | Job start, per-step outcomes, retries, alerts, verdict, lock blocks | Dated, never by size. Nothing deletes old logs | | `logs/research/capture_YYYY_MM_DD.log` | Every capture step and what it wrote | Dated | | `logs/research/health_YYYY_MM_DD.log` | Every health check and its verdict | Dated | | `logs/research/alerts.log` | ` ` | Append-only, uncapped | | `logs/research/cron.out` | Whatever the rendered cron line redirects | Host's business | | `logs/research/scheduler-state.json` | The machine-readable position; last 20 alerts | Overwritten atomically | | `logs/research/scheduler.lock` | The lease | Removed on release | `logs/` is git-ignored. Retention is the host's business — a scheduler that pruned its own evidence would be the wrong shape of program. ### 16.4 Configuration variables Every variable also resolves under a `RESEARCH_SCHEDULER_` prefix, **which wins**. Precedence: prefixed process env → bare process env → the same two in the env file → the default. The env file is `RESEARCH_SCHEDULER_ENV_FILE`, or `backend/.env` when that exists. | Variable | Default | Effect on the lock | |---|---|---| | `ENABLE_AUTOMATIC_CAPTURE` | `true` | `false` → the lock is **never touched** | | `STEP_TIMEOUT_SECONDS` | `3600` (min 60) | **Stale threshold = this × 2** | | `LOG_DIR` | `logs/research` (relative → repo root) | **Where the lock lives** | | `DAILY_REGISTER_TIME` | `18:30` | — | | `WEEKLY_CAPTURE_DAY` | `SAT` (name, or 0–6 with 0=MON) | — | | `WEEKLY_CAPTURE_TIME` | `19:00` | — | | `TIMEZONE` | `Asia/Kolkata` | The clock that stamps and judges the lease | | `CAPTURE_DAYS` | `7` (min 1) | — | | `MAX_CALLS` | `1500` (min 1) | — | | `RETRY_ATTEMPTS` | `4` (min 1) | Longer runs hold the lease longer | | `RETRY_BACKOFF_SECONDS` | `30.0` | as above | | `RETRY_BACKOFF_MAX_SECONDS` | `900.0` | as above | | `ALERT_COMMAND` | unset | The fifth alert surface; receives `RESEARCH_SCHEDULER_ALERT_{JOB,SUBJECT,DETAIL}` | | `PYTHON` | `backend/venv/bin/python` if it exists, else `sys.executable` | — | | `RESEARCH_SCHEDULER_ENV_FILE` | `backend/.env` | Where the above are read from | > **Note.** There is **no lock-specific configuration variable.** The lock's only > tunables are `STEP_TIMEOUT_SECONDS` (×2 = staleness) and `LOG_DIR` (location). ### 16.5 Environment variables at runtime | Variable | Set by | Purpose | |---|---|---| | `PYTHONPATH` | `runner.child_env` | Prepends `backend/` so a child can `python -m research_platform` | | `RESEARCH_SCHEDULER_ALERT_JOB` | `raise_alert` | `daily` / `weekly` | | `RESEARCH_SCHEDULER_ALERT_SUBJECT` | `raise_alert` | e.g. `run skipped — scheduler lock version conflict` | | `RESEARCH_SCHEDULER_ALERT_DETAIL` | `raise_alert` | The one-line diagnostic | | `CI_SKIP` | you, locally only | Space-separated CI gate names to skip. The GitHub workflow never sets it | | `PYTHON` | you | Interpreter for `run_all.sh` | ### 16.6 Module constants | Constant | Module | Value | |---|---|---| | `LOCK_VERSION` / `LOCK_PID` / `LOCK_STAMP` / `LOCK_TOKEN` / `LOCK_STATE` | `lock` | `"version="` / `"pid="` / `"at="` / `"token="` / `"state="` | | `SUPPORTED_VERSION` | `lock` | `2` | | `LEGACY_VERSION` | `lock` | `1` | | `STATE_FREE` | `lock` | `"free"` | | `REOPEN_ATTEMPTS` | `lock` | `64` | | `MUTEX_BACKEND` | `lock` | `"flock"` / `"msvcrt"` / `"none"` | | `LOCK_NAME` | `runner` | `"scheduler.lock"` | | `STALE_AFTER_STEP_TIMEOUTS` | `runner` | `2` | | `SUPPORTED_LOCK_VERSION` | `runner` | `2` (re-export) | | `USAGE_EXIT_CODE` | `runner` | `2` | | `STREAMS` | `journal` | `("capture", "health", "scheduler")` | | `STATE_VERSION` / `ALERT_HISTORY` | `state` | `1` / `20` | ### 16.7 Related CI gates `bash backend/scripts/ci/run_all.sh` runs thirteen gates. Every one runs even after an earlier one fails. | Gate | Relevance to Scheduler Lock v2 | |---|---| | `tests` | **Direct** — runs all 138 scheduler and lock tests | | `research-isolation` | Keeps the broker SDK off the scheduler's import path (`TestBoundaries`) | | `research-platform` | The health step's owner; the scheduler reads its artefact | | `dataset-programme` | Upstream of the platform the health step reports on | | `v2-isolation`, `v2-importlinter`, `v2-config-schema-sync`, `operations-isolation`, `research-engine`, `schema-evolution`, `research-archive-integrity`, `v2-coverage`, `operations-coverage` | Unrelated to the lock; must still be green to commit | > **Note.** Scheduler Lock v2 **added no CI gate and changed none.** It is > covered by gate 1. ### 16.8 Relevant documentation | Document | What it holds | |---|---| | [Scheduler Lock v2 — architecture, race analysis, stress report](v2/architecture/v2_scheduler_lock_v2_2026-08-05.md) | **Frozen 2026-08-05.** Root cause of v1, the design, the TOCTOU argument, the full stress campaign, migration notes | | [The Capture Scheduler](v2/operations/capture_scheduler.md) | The scheduler runbook — §6.4 overlapping runs, §6.5 mixed lock versions, §8 recovery, §9 troubleshooting | | ROM-01 — Research Operations Manual (removed 2026-08-31) | §9 owns every health-finding remedy | | [RG-04 §7 — Dataset Governance](v2/governance/research/RG-04_dataset_governance.md) | The standing capture obligation (P-5) this scheduler discharges | | [Platform Broker operations](v2/operations/platform_broker.md) | The identity a capture authenticates with | | [`deploy/capture-scheduler/README.md`](../deploy/capture-scheduler/README.md) | How the four artefacts were rendered | | [`backend/capture_scheduler/lock.py`](../backend/capture_scheduler/lock.py) | The module docstring is the canonical statement of *why* — read it before any v3 proposal | ### 16.9 Relevant reports | Report | Finding | |---|---| | [Scheduler Lock v2 §5 — stress test report](v2/architecture/v2_scheduler_lock_v2_2026-08-05.md#5-stress-test-report) | v1: 21 of 800 trials raced, 13 duplicate acquisitions, 28 overlaps. v2: 0 | | [Scheduler Lock v2 §6 — performance](v2/architecture/v2_scheduler_lock_v2_2026-08-05.md#6-performance) | +50 µs uncontended; ~2× throughput under contention | | [Capture Recovery, 2026-08-04](v2/governance/research/capture-recovery-report-2026-08-04.md) | The last time a person did the capture by hand | | Research Platform readiness review §5 (removed 2026-08-31) | N-1 — nothing scheduled the capture; closed by this package | --- *Scheduler Lock v2 is frozen. Operate it with this guide; change it with a v3 design.*