Cleanup

DESIGN.md

@@ -1,8 +1,11 @@
-# Design Notes — BNetzA Amateurfunk Question Catalog Downloader
+# Design Notes — BNetzA Amateurfunk Question Catalog → Anki
 
-Status: **design only, no code yet.** This document captures the
-source-discovery work and the proposed shape of the tool. Everything
-below was verified against the live BNetzA site in May 2026.
+Status: **both stages implemented.** Stage 1 (`amateurfunk_fetch.py`)
+downloads and validates the upstream ZIP; Stage 2
+(`amateurfunk_anki.py`) turns the extracted data into Anki decks. This
+document captures the source-discovery work, the JSON schema, and the
+design contracts that both scripts implement. Everything below was
+verified against the live BNetzA site in May 2026.
 
 ---
 
@@ -234,7 +237,7 @@ from the data.
 
 ---
 
-## 4. Tool design (v1)
+## 4. Stage 1 — Fetcher (`amateurfunk_fetch.py`)
 
 ### Goal
 
@@ -430,21 +433,223 @@ These were open in the first draft and have been settled:
    `missing_pictures`, do not fail). Rationale: avoid bricking the
    tool on an upstream typo.
 
-## 6. Open questions
+## 6. Open questions (Stage 1)
 
-These do not block writing v1:
+These do not block the Stage 1 implementation:
 
 1. **PNG fallbacks** — the archive ships PNGs for ~a handful of
-   figures alongside the SVGs. v1 just extracts everything as-is. A
-   future renderer can prefer SVG and fall back to PNG transparently.
+   figures alongside the SVGs. The fetcher just extracts everything
+   as-is. Stage 2 prefers SVG and falls back to PNG by stem:
+   `MediaRegistry.resolve()` tries `<stem>.svg` first, then
+   `<stem>.png`. So an upstream entry that ships only a PNG (or one
+   whose SVG fails to render in some future consumer) still resolves.
 2. **Schema drift safety** — if a future edition adds or renames
    fields, the validator should warn but not fail. v1 fails closed
    on missing required top-level/metadata/question keys but tolerates
    extra keys. We can soften this further if BNetzA evolves the
    format.
-3. **Packaging** — v1 stays a single-file script with `argparse`
-   (`amateurfunk_fetch.py`). Considered and declined: a
-   `pyproject.toml` package with a console-script entry point. For a
-   tool that runs at most quarterly, the packaging ceremony does not
-   pay for itself. Revisit if the scope grows (e.g. importable from
-   another project, distributed on PyPI).
+3. **Packaging** — both scripts stay single-file with `argparse`
+   (`amateurfunk_fetch.py`, `amateurfunk_anki.py`). Considered and
+   declined: a `pyproject.toml` package with console-script entry
+   points. For tools that run at most quarterly, the packaging
+   ceremony does not pay for itself. Revisit if the scope grows
+   (e.g. importable from another project, distributed on PyPI).
+
+---
+
+## 7. Stage 2 — Anki deck builder (`amateurfunk_anki.py`)
+
+### Goal
+
+Given the per-edition directory produced by Stage 1, build a set of
+Anki `.apkg` files that turn every catalog question into a flash card.
+Same input must produce byte-identical output across runs — this is
+important so generated decks can be checksummed and cached cleanly.
+
+### CLI shape
+
+```
+amateurfunk-anki [--data DIR] [--out DIR] [--seed STR] [--epoch INT]
+```
+
+- `--data DIR` — fetch output root (default `./data`). Must contain
+  `manifest-latest.json` pointing at a per-edition directory.
+- `--out DIR` — destination for `.apkg` files (default `./anki`).
+- `--seed STR` — deterministic seed for answer shuffling. The default
+  is fixed; changing it produces a different (but still deterministic)
+  shuffle.
+- `--epoch INT` — override the package timestamp epoch. By default we
+  derive it from the manifest's `fetched_at`; this flag is mainly for
+  tests and explicit rebuilds.
+
+Exit codes: `0` success, `1` configuration / catalog / build error.
+There is no Stage-2 equivalent of the fetcher's `EXIT_BAD_STATE` —
+the builder has no operator-recoverable local state, just generated
+output artifacts.
+
+### Output layout
+
+```
+anki/
+  amateurfunk-technische-kenntnisse.apkg          (~2.5 MB, 1374 cards)
+  amateurfunk-betriebliche-kenntnisse.apkg        (~130 KB,  172 cards)
+  amateurfunk-kenntnisse-von-vorschriften.apkg    (~75 KB,   204 cards)
+```
+
+One `.apkg` per top-level Prüfungsteil. The card counts match the
+study pools from §3 axis 1: Technische = 1374 (cumulative N+E+A);
+Betriebliche = 172 (shared across all candidates); Vorschriften = 204
+(also shared). License class is conveyed via tags
+(`klasse-N` / `klasse-E` / `klasse-A`), not separate packages —
+Anki users browse decks by topic and filter by class.
+
+### Steps
+
+1. **Load** the latest catalog: follow `manifest-latest.json` to a
+   per-edition directory, read the catalog JSON, the per-edition
+   `manifest.json`, and index the `svgs/` folder.
+2. **Categorize** the question tree into three `Category` objects,
+   one per top-level Prüfungsteil. Each category carries the flat
+   list of every question that lives anywhere under it, along with
+   each question's path through the section tree (used for the card
+   breadcrumb and the path tags).
+3. **For each category, render every question** as an Anki note:
+   - Compute a stable per-question seed from `--seed` + question
+     number; shuffle the A/B/C/D choices using it. The displayed
+     label (A/B/C/D) is assigned post-shuffle; the back of the card
+     names the *displayed* position of the correct answer.
+   - Render the question stem and answer texts to HTML via
+     `text_html()`, which tokenizes inline `$...$` math and rewrites
+     it to MathJax `\(...\)` delimiters (Anki 2.1+ ships MathJax
+     built in and recognizes those delimiters; bare `$...$` would
+     show as source). Escapes the rest for HTML safety, preserves
+     line breaks as `<br>`, and preserves `<u>...</u>` tags that
+     the catalog uses to emphasize negation in question stems.
+   - Resolve picture references through a per-category
+     `MediaRegistry`, which records every file actually used so the
+     packager can include only those.
+4. **Build a v11 Anki collection** as an in-memory SQLite database:
+   one `col` row carrying JSON config blobs (deck, model, dconf),
+   one `notes` row per question, one `cards` row per question.
+   Modern Anki understands v11 and upgrades the collection on
+   first open.
+5. **Package** the collection plus the referenced media into a
+   `.apkg` ZIP. Media is addressed by sequential integer keys
+   (Anki's convention); a `media` JSON map at the archive root
+   translates those keys back to filenames.
+6. **Post-process SVGs** at packaging time: inject a white
+   `<rect>` immediately after the opening `<svg>` tag so dark-mode
+   Anki users can still read the black-line BNetzA figures. The
+   extracted source SVGs on disk are never modified. The injection
+   is idempotent (marked with `data-af-white-background="1"`).
+7. **Write `.apkg` atomically**: build into `<out>.tmp`, then
+   `os.replace` into place. Stale `.tmp` directories are
+   overwritten on the next deterministic build — unlike the
+   fetcher's `.bak`/`.tmp` siblings, the Stage-2 temp file holds
+   no operator-recoverable state.
+
+### Determinism
+
+The contract is: same catalog in → same `.apkg` bytes out.
+Determinism rests on three things:
+
+1. **Stable IDs.** `stable_id(namespace, text)` hashes a
+   namespaced key with SHA-1 and squashes into the standard
+   13-digit Anki ID range. Different namespaces
+   (`"deck"` / `"model"` / `"note"` / `"card"`) keep IDs from
+   colliding across kinds; the namespaced inputs are stable
+   (`f"{category.slug}:{number}"` for notes and cards).
+   `stable_guid(text)` produces the 20-character note GUID used
+   for re-import deduplication.
+2. **Stable shuffle.** `randomized_answers()` builds a per-question
+   `random.Random` seeded from SHA-256 of
+   `f"{cli_seed}:{question_number}"`.
+3. **Stable timestamps.** Every `now` value in the collection (the
+   `mod` columns, the JSON config blob timestamps) is fixed to
+   `build_epoch` — derived from the manifest's `fetched_at`, or
+   overridden via `--epoch`. ZIP member timestamps are also fixed
+   via `ZipInfo(name, zip_datetime(build_epoch))`. Without this
+   last step, the inner SQLite would be identical but the
+   archive's per-entry mtimes would still vary between runs.
+
+The combined effect: two runs with the same `data/` produce
+byte-identical sha256 on each `.apkg`. Verified during review.
+
+### Rendering decisions worth knowing
+
+- **A is always correct upstream, then shuffled.** This is the
+  consumer-side caveat from §3 made concrete: we set the
+  `correct` flag on `answer_a` before shuffling, then carry it
+  through. The back of the card reveals the *displayed* position
+  ("Richtige Antwort: B"), not the source position.
+- **LaTeX → MathJax.** The catalog ships ~430 fragments
+  containing inline `$...$` (DESIGN §3 quotes the upstream README
+  on this). Anki 2.1+ MathJax recognizes `\(...\)` natively;
+  `$...$` does not render. The rewrite happens in `text_html()`.
+- **Safe inline tags.** The catalog uses `<u>...</u>` for
+  emphasis. Naive HTML escaping would convert those to
+  `&lt;u&gt;...&lt;/u&gt;` and lose the emphasis. We split on a
+  regex that matches exactly `<u>` and `</u>` and escape only the
+  non-tag pieces.
+- **Slug for filenames.** `slugify()` maps German umlauts and ß
+  to ASCII digraphs before NFKD-normalizing the rest, then keeps
+  only `[a-z0-9-]`. Yields readable, sortable filenames
+  (`amateurfunk-technische-kenntnisse.apkg`).
+- **Tags.** Each note carries `klasse-N|E|A` plus
+  `pfad-<slugified-section>` for every section level below the
+  top-level Prüfungsteil. Per-question number tags are
+  deliberately *not* emitted — they would create ~1750 singletons
+  in Anki's tag tree, and the number is already in the dedicated
+  `Number` field for search.
+- **Breadcrumb consistency.** The visible card breadcrumb and the
+  stored `Path` field both go through one `display_path()`
+  helper, so they never drift. The boilerplate
+  `Prüfungsfragen im Prüfungsteil:` prefix is stripped at this
+  layer.
+- **SVG white background.** BNetzA SVGs are transparent with
+  black line work. In Anki's dark mode the lines disappear into
+  the card background. We inject a white `<rect>` as the first
+  painted element when packaging the SVG into the `.apkg`. The
+  on-disk extracted files are left untouched.
+
+### Testing focus
+
+Network-free, fixture-driven. Real Anki is NOT required to
+verify the output — tests open the `.apkg` ZIP directly, parse
+the embedded SQLite, and check structural invariants.
+
+- Top-level structure: one `.apkg` per Prüfungsteil; expected
+  filename slugs.
+- Note count and field shape per category.
+- Tag mapping: a class-1 question carries `klasse-N`, no
+  `nummer-*` tags.
+- Breadcrumb rendering: both the visible HTML breadcrumb and the
+  stored `Path` field contain the short form; neither contains
+  the Prüfungsteil prefix.
+- LaTeX rewrite: a fixture question with `$...$` produces
+  `\(...\)` in the rendered HTML; a non-math `<` is still
+  escaped to `&lt;`; safe inline `<u>...</u>` tags survive
+  escaping.
+- Answer shuffle: the front HTML contains the answer texts in a
+  non-A-first order; the back names the displayed letter of the
+  correct answer.
+- Picture handling on shuffled answers: a fixture question whose
+  `picture_a`/`picture_b`/... fields point at different files
+  carries the right image with the right shuffled choice.
+- Media inclusion: `MediaRegistry.used_paths` is populated by
+  resolution and only those files end up in the `.apkg`.
+- SVG white background injection: idempotent; the marker rect is
+  emitted exactly once.
+- Deterministic build: two consecutive builds against the same
+  catalog with a fixed `--epoch` produce byte-identical `.apkg`
+  files.
+
+### What we deliberately do NOT do in Stage 2
+
+- No editable cloze cards or "type the answer" templates — the
+  exam is multiple choice, and the deck mirrors that.
+- No per-class packages. Class is a tag axis; package split is
+  by Prüfungsteil.
+- No automatic edition diffs / "new questions since last build"
+  decks. Useful but out of scope.
+- No upload to AnkiWeb. Decks are local artifacts.