# AI_Trade — project instructions

## Where documents go

**[docs/docs-folder-structure.md](docs/docs-folder-structure.md) is the source of
truth.** Read it before creating **any** `.md` file. Never drop a new document at
the `docs/` root — only cross-cutting documents live there. Update that file when
you add a folder.

Links in `docs/` are **relative to the document's own folder** (`../spec/…`,
`../../../backend/…`). A repo-root-relative `backend/foo.py` will not resolve.

Three scripts have functional dependencies on doc paths — moving the files they
read breaks them:

| Script | Reads |
|---|---|
| `backend/scripts/ci/check_v2_config_schema_sync.sh` | `docs/v2/spec/v2_spec_05_config_schema.md` |
| `backend/scripts/ci/check_research_archive_integrity.sh` | every `*.manifest.sha256` under `docs/v2/governance/research/registry/`, and the paths listed inside them |
| `backend/tools/generate_developer_command_reference.py` | the whole repository; **writes** `docs/DEVELOPER_COMMAND_REFERENCE.md` |

`dataset_programme/paths.py` and `research_platform/paths.py` were the fourth
and fifth entries and are gone with their packages (2026-08-31). Nothing now
reads a heading out of a governance document.

**`docs/DEVELOPER_COMMAND_REFERENCE.md` is generated — never edit it.** It is
the answer to "how do I run this": every CLI, sub-command, script, test file,
CI gate, migration and deployment renderer, discovered by parsing the tree.
Regenerate with `python backend/tools/generate_developer_command_reference.py`
in the same change that adds or removes a command; CI gate 12 compares the
committed copy against a fresh generation and fails on any difference. Add a
command and the document grows a row on its own — there is no second list.

## CI

**`bash backend/scripts/ci/run_all.sh` is the gate.** Thirteen of them, all
run, every failure reported rather than stopping at the first:

| # | Gate | Holds |
|--:|---|---|
| 1 | `tests` | the whole suite, `backend/tests` — not just `tests/v2/` |
| 2 | `v2-isolation` | the V2 boundary |
| 3 | `v2-importlinter` | the import contracts |
| 4 | `v2-config-schema-sync` | `v2/config/schema.py` equal to Spec 05 |
| 5 | `operations-isolation` | the Operations boundary |
| 6 | `research-engine` | RG-10 — a formula in `experiments/` is declared and proved equal to production |
| 7 | `schema-evolution` | `db/migrations/` is the only place schema-mutating DDL lives |
| 8 | `ai-auto-scan-isolation` | the scanner hands trades to Paper Admission and lets go |
| 9 | `research-archive-integrity` | RG-07 §3 — closed campaigns byte-identical to their state at closure |
| 10 | `signal-validator` | RG-15 — the validator stays a leaf |
| 11 | `developer-command-reference` | the generated command reference matches the tree |
| 12 | `v2-coverage` | the V2 coverage floor |
| 13 | `operations-coverage` | the Operations coverage floor |

It was sixteen until 2026-08-31: `research-isolation`, `dataset-programme` and
`research-platform` left with the packages they policed. **A gate and its
subject leave together** — removing the package first turns the gate red,
removing the gate first leaves the package unpoliced.

`.github/workflows/ci.yml` calls this script on every push and pull request, so
local and CI cannot disagree about what green means. Add a check by adding it
there, not only to the workflow.

CI provisions a real MySQL and calls `backend/scripts/ci/bootstrap_ci_db.py`.
That is not optional: several suites assert against the live database and skip
when they cannot reach one, so a database-less pipeline goes green with ~97
tests skipped and 13 failing. The bootstrap calls
`db.bootstrap.initialise_database()` — **the same function `main.lifespan`
calls, with the same (absent) arguments** — so CI and production initialise the
database identically by construction rather than by two lists staying in step.

**A green `run_all.sh` proves the tests pass in the environment you have, not
that the environment rebuilds from `requirements.txt`.** Only the GitHub job
answers the second, because it is the only thing that installs from a clean
slate. A dependency present in a developer venv but undeclared reads as green
locally and fails collection on the runner — see
[the 2026-08-03 broker-SDK import boundary report](docs/v2/architecture/v2_ci_broker_sdk_import_boundary_2026-08-03.md).
`smartapi-python` does not declare `websocket-client` despite importing it; do
not "clean up" either that pin or `websockets`, which is a different package.

**The broker SDK must stay off the import path of everything that does not
trade.** `data/index_scanner.py`, `data/option_chain.py` and `sector/` open
with a broker import, so importing them costs the Angel One SDK. Import
`INDEX_META` from `data/index_metadata.py` (which imports nothing), and reach
for the scanner's fetchers **inside the function that needs them** — the
pattern `replay/`, `v2/data/gateway_impl.py` and `api/routes.py` already use.

**The rule outlived the gate that used to spell it out.** `check_research_isolation.py`
rule 10 failed the build on a module-level chain from `research/` to a broker,
and went with `backend/research/` on 2026-08-31. Nothing enforces the general
form now — `v2-isolation`, `operations-isolation` and `ai-auto-scan-isolation`
each police their own package — so this paragraph is the whole of it for
everything else. Import cost is invisible until a suite is slow or a headless
job needs credentials it should not.

`backend/platform_broker/` is bound by the same rule for a non-obvious reason:
the admin router imports it and `admin/__init__` imports the router, so **one
module-level `from brokers…` there puts the SDK on the import path of every
suite that touches `admin.dependencies`.** Importing any `brokers` submodule
runs `brokers/__init__.py`, which loads the SDK. `platform_broker/client.py` is
the single exception and is only ever imported inside a function. This was
found the hard way once, through `research/routes.py`; that importer is gone
and the trap is not.

## Schema evolution

**`backend/db/migrations/` is the only supported way to change the schema.**
`scripts/ci/check_schema_evolution.py` fails the build on `ALTER TABLE`,
`CREATE INDEX`, `DROP INDEX`, `DROP TABLE`, `RENAME TABLE` or `TRUNCATE`
anywhere else. A model module's `init_*_db()` creates tables from the models and
seeds rows; it does **not** carry a column sweep.

Three properties are load-bearing and easy to break:

- **Migrations run before `create_all`** (`db/bootstrap.py`), because `0005`
  renames `ai_strategy_config` → `strategy_registry` and a `create_all` that got
  there first would leave the register empty. **Every migration must no-op when
  its table does not exist.**
- **Guarded DDL *and* the `schema_migrations` ledger.** Not either. The live
  databases already have every legacy sweep applied and an empty ledger, so a
  ledger-only runner would re-apply and fail.
- **`downgrade=None` requires `irreversible_because`** — the dataclass refuses
  to be constructed without it.

`python backend/scripts/db_migrate.py status | upgrade [--dry-run] | downgrade <v>`.

**A module creates the tables it declares, and no others** —
`db.schema.create_owned(__name__)`, never a bare `Base.metadata.create_all()`.
`Base` is shared, so an unfiltered `create_all` creates *whatever the caller
imported*: `main.py` imports `replay.routes` and so created eight tables at boot
that `bootstrap_ci_db.py` did not, and the same `initialise_database()` produced
a 60-table schema in production and a 45-table one in CI. `tests/
test_bootstrap_determinism.py` fails the build on an unfiltered `create_all` and
on any divergence between entry points; the account is
[the 2026-08-05 schema-ownership report](docs/v2/architecture/v2_schema_ownership_2026-08-05.md).

**Two stores are not created at boot and readers must not assume them** —
`historical_candles` and the two option tables. They belong to Super Admin tools
that install them on first use (`db.schema.OPTIONAL_STORES`). Ask
`db.schema.candle_store_present()` / `option_store_present()` — the inspector,
never a probe query — and report the absence.

It was three. The twelve `research_*` tables were the third and were dropped by
migration `0026` on 2026-08-31 with the packages that owned them, along with
the two CI gates whose rule 7 kept the health monitor and the Dataset Programme
from installing one.

## Configuration

**Every ratified V2 threshold resolves through `v2/config/resolved.py`**, which
reads `v2/config/schema.py` — the artefact `check_v2_config_schema_sync.sh`
holds equal to Spec 05. Do not re-type a threshold in the filter or risk rule
that uses it; that was H-4.

It reads nothing from a database, deliberately: a retune requires a Spec 05
amendment **and** a component version bump (`m4_exhaustion.v2`), and an operator
switch would make both advisory. Eleven of the twenty-two keys are classified
`DECLARED_BUT_UNREAD` — **wiring one up changes behaviour** (`feature.v2_engine_active`
is `false` while V2 is the only engine), and a test greps `v2/` to keep them
unread.

**Removing a key is a dated Spec 05 amendment plus a tombstone in the code
fence** — the precedent is `strategy.multi_layer_ai.enabled` (2026-07-27), and
the three most recent are `risk.max_open_positions_count`,
`risk.max_positions_per_symbol_count` and `threshold.session_pnl_stop_pct`
(2026-08-20). Remove rather than leave declared-and-unread when the component
that named the key is gone: an unread key naming something that still exists is
a wiring job someone may do, and an unread key naming something that does not
is a false statement about the system.

## Capability vocabulary

**Markets, instrument types, timeframes and reward-to-risk ratios have exactly
one definition each, and no module may keep a second.**

| Concept | Owner |
|---|---|
| Markets | `backend/data/index_metadata.py` (`INDEX_META`) |
| Instrument types | `backend/data/instrument_metadata.py` (`INSTRUMENT_META`) |
| Timeframes, R:R, the two defaults | `capability_options` + `ai_config`, via `services/strategy_capability_service.py` |
| Step granularities | `backend/replay/runner.py` (`STEP_CHOICES`) |
| Position size / lot size | `backend/v2/data/contract_quantity.py` (`resolve_contract_quantity`) |
| What a *campaign validated* | `strategy_registry`, via `services/strategy_registry_service.py` |

**An option position is sized in lots, and the lot is the contract's.** Nothing
outside `contract_quantity.py` may write a quantity or a lot size: NIFTY's lot
is not BANKNIFTY's, and the exchange revises both. Four tiers, most specific
first — the contract's own `lot_size` (carried onto every chain row by
`data/option_chain.py` live and `replay/option_chain.py` historically), the
scrip master by trading symbol, the scrip master by underlying, and
`instrument_master.FALLBACK_LOT_SIZES` only when the master is unreachable.
**No tier has a default**; an unresolvable lot raises, and `PaperExecutor`
refuses the position rather than opening one at 1/75th of its size. The
index/equity path is separate and still sizes at one unit
(`entry_plan._DEFAULT_QUANTITY`) — applying *that* to options was the
2026-08-22 sizing defect, which understated investment, `pnl_amount`, ROI and
every Reports total by the lot size while leaving `pnl_pct` correct.

The two `data/` registers **import nothing**. That is load-bearing: it is what
lets `api/` and `replay/` read them without acquiring one another's
dependencies, or the broker SDK. Keep them that way.

Clients render what the API serves and hold no lists of their own —
`GET /api/v2/capabilities` for the user app and
`GET /admin/backtest-v2/replay/symbols` for the Replay panel.
`GET /admin/research/configuration` was the third and served the Research
Workbench's Configure tab; endpoint and tab were removed together on
2026-08-31. Two tests enforce the rule, one per language:
`test_no_live_module_hard_codes_a_capability_list` and
`test_no_frontend_module_hard_codes_a_capability_list`.

**`/v2/analyse` asks the *platform* timeframe question; `replay/routes.py` asks
the *per-campaign* one.** Do not unify them. Replay refuses an unvalidated run
because its output is evidence; production is gated by the registry instead,
and narrowing it by `validated_timeframes` would turn an incomplete registry
row into an outage. A capability refusal costs no scan credit.

## API authentication

**Every mounted endpoint must transitively depend on `get_current_user` or
`get_current_admin`.** `backend/tests/test_api_authentication.py` parses
`main.py`, resolves every `include_router` call and asserts it, one test per
endpoint. A route that cannot satisfy it must be added to that file's `PUBLIC`
allow-list **with the reason it is safe to serve anonymously** — a deliberate
act that shows up in review, unlike forgetting a guard.

Declare the guard on the `APIRouter`, not per endpoint, so an endpoint added
later inherits it. Super Admin surfaces use `admin.dependencies.
require_super_admin` — the one definition, shared; not a fourth copy, and not
`require_permission`, which a role edit could grant.

**`POST /api/v2/analyse` is the billable scan**, and
[SL-01, the Scan Lifecycle](docs/scanner/scan_lifecycle_billing_rule.md), is the
canonical rule for when a credit is spent — read it before touching
`services/scan_accounting.py`, the confidence threshold, or Confirm Trade
rendering. **A scan is consumed only when the analysis produces at least one
actionable trade the user is allowed to confirm**, once per analysis session
however many Confirm Trade buttons appear. Ratified and implemented 2026-08-06;
it supersedes C-3's "one credit per *completed* analysis"
([impact assessment](docs/scanner/scan_lifecycle_impact_assessment_2026-08-06.md)).

**The decision is made once — `scan_accounting.is_billable_analysis()`, at the
end of the analysis — and frozen onto the session row; nothing recomputes it**
— not Paper Confirm, not Reports, not the Dashboard. A threshold an
administrator changes at 09:17 must never refuse a trade billed at 09:15
([billability snapshot architecture](docs/scanner/scan_billability_snapshot_architecture_2026-08-06.md),
migration `0013`). Four things break it:

- **Comparing confidence against the current threshold anywhere downstream.**
  Outside `scan_accounting` nothing may read `ai.confidence_threshold` to
  decide; `api/v2_routes.py` resolves it once per request and *transports* it.
  An unresolvable threshold makes the analysis **not billable** — the display
  fallback constant must never become a billing input.
- **Re-implementing the predicate in the browser.** The Confirm Trade button
  renders from `signals[].confirmable`, served by the backend. The client-side
  comparison that used to decide it was a billing boundary living in a client.
- **Collapsing `billable` and `scan_counted`.** They are different facts.
  `billable=True, scan_counted=False` is a recoverable under-charge; the
  reverse is a bug signature, and `_charge()` refuses to produce one.
- **Backfilling `billable = scan_counted`.** `billing_rule_version IS NULL`
  marks a row decided under C-3, and such a row is never confirmable.

`/analyse` carries `require_scan_quota` and does the accounting through
`services/scan_accounting.py`; a request the platform could not analyse costs
nothing, and `/api/v2/paper/confirm` performs no accounting at all — it reads
the stored snapshot and re-applies the registry gate, because governance is not
pricing. Both gates fail closed. The five canonical scenarios and SL-V1…SL-V35
are a **mandatory** validation scope —
[docs/v2/validation/v2_scan_lifecycle_validation_scope.md](docs/v2/validation/v2_scan_lifecycle_validation_scope.md).

## The Platform Broker

**Every research, capture, backfill and maintenance job resolves its broker
through `PlatformBrokerService` — the platform's own identity, not a user's.**
Before 2026-08-05 there was no platform-level broker authentication at all:
every non-user path read the process-global `broker_manager._sdk`, which is
only ever populated by a *user's* request, so `scripts/capture_weekly.py`
could not authenticate from cron and platform downloads ran on whichever
customer's session happened to be loaded.
Operations: [docs/v2/operations/platform_broker.md](docs/v2/operations/platform_broker.md).

```python
with PlatformBrokerService.session(source="capture_weekly"):
    backfill_symbol("NIFTY", days_back=7)      # platform identity, downstream unchanged
```

The identity is made **ambient** through a `ContextVar` (`brokers/context.py`)
that `broker_client` consults before falling back to `broker_manager`. That is
why `data/index_scanner.py`, `replay/backfill.py` and `replay/option_backfill.py`
are unmodified — and why the scanner and trading engine were not touched.

Four things to know before editing it:

- **It cannot trade, by construction.** `PlatformBrokerClient` raises
  `PlatformBrokerForbidden` on seven trading methods and audits the attempt.
  A `PlatformBrokerForbidden` in the log is a bug report, not an outage. The
  WebSocket tick feed is **market data and is permitted** (2026-08-10) — but a
  feed outlives the `session()` that started it, and a tick is not a closed
  bar, so the evidence path must keep polling closed candles.
- **`FERNET_KEY` must be set.** `auth/crypto.py` silently generates a
  throwaway key when it is absent, and every stored credential becomes
  `unreadable` after a restart.
- **The user broker stack is untouched and must stay that way** —
  `broker_connections`, `broker_sessions`, `broker_manager`,
  `DefaultBrokerResolver`. A test walks the AST of `platform_broker/` and fails
  on any reference to them.
- **`capture_weekly.py` registers contracts *outside* the broker session**,
  deliberately: registration needs no broker and is the irreversible half.

## Strategy research

**The research infrastructure was retired on 2026-08-30/31. The record it
produced was not.** `backend/research/`, `backend/research_platform/` and
`backend/dataset_programme/`, their three CI gates, their twelve tables and
their Super Admin pages are gone; so is the documentation describing how to
operate or extend them. What every campaign concluded is kept, and this section
is what still binds. The account is
[the retirement record](docs/v2/architecture/v2_research_infrastructure_retirement_2026-08-30.md).

**Every strategy investigation is still a Campaign and still follows
[the Research Governance Framework](docs/v2/governance/research/) (RG-01…RG-13,
less RG-14), ratified 2026-08-02.** Read
[the framework index](docs/v2/governance/research/README.md) before starting any
research, backtest, or "does X work" question — and before opening a campaign,
[RKB-01 the Knowledge Base](docs/v2/governance/research/RKB-01_research_knowledge_base.md)
and [RG-12 the Search Space Map](docs/v2/governance/research/RG-12_search_space_map.md),
which RG-13's CD-0 makes a Gate 1 item. **Fifteen of twenty-one hypothesis
families are closed to new campaigns**; check before proposing one. The rules
survive their tooling: they constrain what may be concluded, and nothing that
was removed was doing the concluding.

The rules that are easiest to break by accident:

- **No production modifications during research** (RR-1). Research raises
  anomalies; it does not fix them.
- **No optimisation recommendation** unless the campaign already demonstrated a
  statistically significant *and* reproducible edge (RR-2). "Try a wider stop" and
  "re-run without the filter" are optimisation recommendations.
- **Every conclusion carries an evidence level L1–L5**, and **L1/L2 may never
  justify a production change**. Levels propagate at the weakest link.
- **Closed campaigns are immutable.** Corrections append to ERRATA; the body is
  never edited. Verify with
  `bash backend/scripts/ci/check_research_archive_integrity.sh`.
- **Datasets are frozen at Stage 1** ([RG-04 §5](docs/v2/governance/research/RG-04_dataset_governance.md#5-the-frozen-partition)).
  Hold-out is touched once. CV folds are session-blocked, never row-level.

**Campaigns A and B are both CLOSED — REJECTED** (2026-08-02, 2026-08-04).
`ema_crossover.v1` is **frozen as the baseline reference implementation**: it may
be executed as a control arm, and must never be modified, optimised, or re-opened
without a new formally-approved research proposal. Campaign B's harness
(`backend/experiments/phase25/`) is frozen the same way — reusable as a
gate-validated instrument, never as a source of trading logic.

**Campaigns C and D are PLANNED, not open.** D is next under RG-08 §4's selection
rule and is assessed **NOT READY**
([report](docs/v2/governance/research/campaign-D-readiness-2026-08-04.md)): P-6
no hold-out partition remains, P-7 RG-12/RG-13 are unratified (RG-14 was the
third and is retired), P-8 D's single-leg vs two-leg scope is undecided. P-1 and
P-2 closed 2026-08-02.

**P-6 has a decision and no resolution, and now no implementation either.**
[DDR-01](docs/v2/governance/research/DDR-01_dataset_generation_v2_2026-08-04.md)
was approved 2026-08-04 and adopted **Option E** — validation data as a
programme-level asset with a ledger, an allocation policy and a lifecycle. It
was built as `backend/dataset_programme/` and RG-14, **held zero slices from the
day it shipped**, never ran the Dataset Audit, and was removed on 2026-08-31
with the rest of the estate. DDR-01 survives as the decision record. **P-6 is
open, Campaign D stays shut, and reopening either starts from the decision, not
from a package.**

**P-5 remains breached, but the loss was smaller than recorded.** Weekly
`ingest_contracts` was missed; the
[Capture Recovery](docs/v2/governance/research/capture-recovery-report-2026-08-04.md)
recovered both August sessions on both paths and took the contract registry
from 2 expiries to 8. A-63 recorded that only the **contract registry** is
unrecoverable once an expiry passes, candles for a registered expiry being
backfillable afterwards — **and A-71 (2026-08-11) measured the deadline on
that**: a week after expiry the broker returns `data: []` for the contract, so
registering before the expiry passes is necessary and not sufficient. Fetch
the candles before it passes too. **A-70** is the same limit intraday: the NFO
15:30–15:39 tail is served for the current session only.
`backend/scripts/capture_weekly.py --status` is the check; nothing schedules it
yet.

**`strategy_registry` is directly administered, and boot reconciles nothing**
(2026-08-30). `services/campaign_registry_sync.py` used to parse the campaign
records at every boot and write each closed campaign's verdict onto the matching
row ([RG-11](docs/v2/governance/research/RG-11_registry_synchronisation.md));
it is retired, along with `db/bootstrap.py`'s phase 3. `SEED_ROADMAP` is the
table's origin — the closed campaigns' terminal states are literals in it, and
migration `0025` carries an older install — and `PATCH /admin/ai/strategies/{id}`
under **`require_super_admin`** is its one authority, with an `admin_audit_log`
row per change. Nothing in the boot plan opens a file under `docs/`, and a test
fails the build if that changes. An edit may not put a row into RELEASED,
PRODUCTION or enabled without a registered implementation — the startup
consistency pass's invariant, now also a write-time refusal. The account is
[the 2026-08-30 direct-authority report](docs/v2/architecture/v2_strategy_registry_direct_authority_2026-08-30.md).

## The Research Platform, the Dataset Programme and the research workloads — retired

All three were removed on 2026-08-30/31 and nothing replaces them. There is no
`python -m research_platform`, no `python -m dataset_programme`, no
`research.specifications` CLI, no replay-matrix or behaviour-study workload, no
`research_*` table and no Research Workbench or Research Matrix page.

**Where the history lives:** campaign verdicts and their checksum manifests in
[`docs/v2/governance/research/registry/`](docs/v2/governance/research/registry/);
the anomaly ledger in
[`docs/v2/validation/systematic/README.md`](docs/v2/validation/systematic/README.md);
every measurement in [`docs/v2/validation/`](docs/v2/validation/). Full account:
[the retirement record](docs/v2/architecture/v2_research_infrastructure_retirement_2026-08-30.md).

**Two things that look retired and are not.** The **Research Dashboard** at
`/research/dashboard` is live — it reads `replay_runs` through
`replay/dashboard_routes.py`, three GET endpoints, and it kept its name because
the name describes replay evidence rather than a package. **`research-archive-integrity`
is still gate 9**: closed campaigns must stay byte-identical to their state at
closure, and that matters more now that the code which produced them is gone.

**RG-07's immutability rule is unchanged and is the reason this cleanup kept
what it kept.** Closed campaigns are immutable; corrections append to ERRATA.

## Optimisation and evidence

[docs/v2/v2_optimisation_research_principles.md](docs/v2/v2_optimisation_research_principles.md)
is a permanent standing mandate and remains in force. One variable per
experiment; compare against the immutable Phase 12 baseline; state sample size
and minimum detectable effect **before** interpreting any result;
**Inconclusive is a valid, expected outcome.** Where it and the Research
Governance Framework both speak, the stricter applies.

## Engineering conventions

- **Specs are frozen.** A spec changes only by a dated amendment document beside
  it, never by editing the original.
- **Nothing ships without a governance pass.** Strategy and filter specs are
  reviewed and ratified before implementation.
- **Phase numbers are namespaced.** V1 (`docs/confidence_engine/`, phases 1–15)
  and V2 phases are unrelated sequences. Always say which.
- **Replay executes production code.** Research harnesses are clients of
  production and own no **undeclared or unproved** trading logic — amended
  2026-08-03 by
  [RG-10](docs/v2/governance/research/RG-10_research_engine_conformance.md).
  A formula transcribed into `backend/experiments/` must be declared in
  `experiments/conformance/registry.py` and proved numerically equal to
  production in `backend/tests/test_research_engine_conformance.py`, **in the
  same change that introduces it**. CI fails otherwise.
- An active strategy row must have a registered implementation — enforced at
  startup ([consistency pass](docs/v2/governance/v2_strategy_registry_consistency_pass_2026-07-31.md)).