Cleanup

CLAUDE.md

@@ -1,13 +1,12 @@
-# Amateurfunk — BNetzA Question Catalog Downloader
+# Amateurfunk — BNetzA Question Catalog → Anki Decks
 
-A small Python tool that downloads the latest German amateur-radio exam
+A small two-stage Python pipeline that downloads the German amateur-radio
-question catalog ("Fragenkatalog") published by the Bundesnetzagentur
+exam question catalog ("Fragenkatalog") published by the Bundesnetzagentur
-(BNetzA) and extracts the structured JSON + SVG image assets for further
+(BNetzA) and turns it into Anki decks.
-use (study apps, flashcards, training tools, etc.).
 
-The full source-discovery notes, JSON schema, and design decisions live
+The full source-discovery notes, JSON schema, exam-structure details, and
-in `DESIGN.md`. This file is a short orientation for anyone (human or
+per-stage design decisions live in `DESIGN.md`. This file is a short
-agent) opening the project.
+orientation for anyone (human or agent) opening the project.
 
 ## What the catalog is
 
@@ -25,7 +24,7 @@ agent) opening the project.
 
 - ZIP (machine-readable): `https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/PruefungsfragenZIP.zip?__blob=publicationFile`
 - Landing page (short link): `https://www.bnetza.de/amateurfunk-fragenkatalog`
-- PDF (human-readable, not used by this tool): same path with
+- PDF (human-readable, not used by this pipeline): same path with
   `Pruefungsfragen.pdf` instead of `PruefungsfragenZIP.zip`.
 
 The ZIP URL is stable across editions — BNetzA replaces the file
@@ -34,32 +33,65 @@ detection. The filename inside the ZIP (`fragenkatalog3b.json`) encodes
 the edition (`3b` = 3rd edition, revision b) and will change on new
 editions, so we discover it from the archive rather than hard-coding.
 
-## Scope of the tool (initial)
+## Pipeline overview
+
+```
+BNetzA ZIP ──[Stage 1: amateurfunk_fetch.py]──► data/<slug>/
+                                                  ├── fragenkatalog*.json
+                                                  ├── svgs/
+                                                  ├── README.txt
+                                                  └── manifest.json
+
+data/ ──[Stage 2: amateurfunk_anki.py]──► anki/
+                                            ├── amateurfunk-technische-kenntnisse.apkg
+                                            ├── amateurfunk-betriebliche-kenntnisse.apkg
+                                            └── amateurfunk-kenntnisse-von-vorschriften.apkg
+```
+
+### Stage 1 — `amateurfunk_fetch.py`
 
 1. Download the ZIP from the canonical URL.
 2. Verify it is a valid ZIP and contains the expected JSON + SVG files.
 3. Extract to a target directory (default: `./data/<edition>/`).
-4. Emit a small `manifest.json` next to the data: source URL, fetched-at
+4. Emit a small `manifest.json` next to the data: source URL,
-   timestamp, `Last-Modified` from the server, JSON edition metadata,
+   fetched-at timestamp, `Last-Modified` from the server, JSON edition
-   sha256 of the ZIP.
+   metadata, sha256 of the ZIP.
 5. Be idempotent — re-running without an upstream change is a no-op.
    The skip key is the HTTP `Last-Modified` header recorded on the
    previous manifest; the ZIP is deleted by default after extraction,
    so the recorded sha256 is provenance, not a re-verification target.
-   See `DESIGN.md` for the full idempotency contract.
+   See `DESIGN.md` §4 for the full idempotency contract.
 
-Out of scope for v1 (kept for later): rendering LaTeX/SVG, building a
+### Stage 2 — `amateurfunk_anki.py`
-study app, multi-edition diffing, mirroring the PDF.
+
+1. Read the latest edition from `data/` (following
+   `manifest-latest.json` to a per-edition directory).
+2. Split the catalog into three categories — one per top-level
+   Prüfungsteil (Technische / Betriebliche / Vorschriften). The
+   license-class axis is mapped into tags, not separate packages.
+3. Render every question as an Anki note: shuffled A/B/C/D choices on
+   the front, the displayed position of the correct answer on the
+   back. Inline `$...$` LaTeX is converted to MathJax `\(...\)`
+   delimiters; the catalog's safe inline markup (`<u>...</u>`) is
+   preserved.
+4. Hand-roll the v11 Anki collection (SQLite + JSON config) and
+   package it as a `.apkg` ZIP with deterministic timestamps. Same
+   input → byte-identical output across runs.
+
+The Anki design decisions (shuffle seeding, deterministic build epoch,
+SVG dark-mode handling, schema choices) live in `DESIGN.md` §7.
 
 ## Repo conventions
 
-- Python 3.11+, standard library only where reasonable (`urllib`,
+- Python 3.11+, standard library only. No third-party dependencies
-  `zipfile`, `hashlib`, `json`, `pathlib`). Add deps only if they pay
+  in either stage.
-  for themselves — none expected for v1.
+- Single-file scripts: `amateurfunk_fetch.py`, `amateurfunk_anki.py`.
-- Single-purpose script: `amateurfunk_fetch.py` (or a tiny package if
+  No frameworks, no CLI library beyond `argparse`.
-  it grows). No framework, no CLI library beyond `argparse`.
+- Style: section banners, commented constants, docstrings on every
-- Downloaded data is treated as a build artifact: kept under `data/`
+  function, inline comments at decision points. The two scripts
-  and gitignored.
+  intentionally read the same way.
+- Outputs are build artifacts: kept under `data/` and `anki/`, both
+  gitignored.
 - License attribution string (required by DL-DE→BY-2.0) is preserved
   verbatim from the upstream `README.txt` whenever we redistribute the
   data.
@@ -67,10 +99,14 @@ study app, multi-edition diffing, mirroring the PDF.
 ## Working on this repo
 
 - Start from `DESIGN.md` — it has the JSON schema, the question/answer
-  conventions (answer A is always correct, B/C/D are distractors), the
+  conventions (answer A is always correct upstream → consumers shuffle
-  LaTeX-in-questions caveat, and the SVG naming convention.
+  before display), the LaTeX-in-questions caveat, the exam-structure
+  rationale for the three Anki packages, and per-stage design notes.
 - Do not invent new download URLs; the ones in `DESIGN.md` were
   verified against the live BNetzA site.
 - When BNetzA publishes a new edition, expect a new
-  `fragenkatalog<N><rev>.json` filename inside the ZIP. The downloader
+  `fragenkatalog<N><rev>.json` filename inside the ZIP. The fetcher
   must not hard-code the current name.
+- Both stages have a fixture-driven test suite. Run with
+  `python3 -m unittest test_amateurfunk_fetch test_amateurfunk_anki`.
+  Network access is only needed for the manual smoke test of Stage 1.

DESIGN.md

@@ -1,8 +1,11 @@
-# Design Notes — BNetzA Amateurfunk Question Catalog Downloader
+# Design Notes — BNetzA Amateurfunk Question Catalog → Anki
 
-Status: **design only, no code yet.** This document captures the
+Status: **both stages implemented.** Stage 1 (`amateurfunk_fetch.py`)
-source-discovery work and the proposed shape of the tool. Everything
+downloads and validates the upstream ZIP; Stage 2
-below was verified against the live BNetzA site in May 2026.
+(`amateurfunk_anki.py`) turns the extracted data into Anki decks. This
+document captures the source-discovery work, the JSON schema, and the
+design contracts that both scripts implement. Everything below was
+verified against the live BNetzA site in May 2026.
 
 ---
 
@@ -234,7 +237,7 @@ from the data.
 
 ---
 
-## 4. Tool design (v1)
+## 4. Stage 1 — Fetcher (`amateurfunk_fetch.py`)
 
 ### Goal
 
@@ -430,21 +433,223 @@ These were open in the first draft and have been settled:
    `missing_pictures`, do not fail). Rationale: avoid bricking the
    tool on an upstream typo.
 
-## 6. Open questions
+## 6. Open questions (Stage 1)
 
-These do not block writing v1:
+These do not block the Stage 1 implementation:
 
 1. **PNG fallbacks** — the archive ships PNGs for ~a handful of
-   figures alongside the SVGs. v1 just extracts everything as-is. A
+   figures alongside the SVGs. The fetcher just extracts everything
-   future renderer can prefer SVG and fall back to PNG transparently.
+   as-is. Stage 2 prefers SVG and falls back to PNG by stem:
+   `MediaRegistry.resolve()` tries `<stem>.svg` first, then
+   `<stem>.png`. So an upstream entry that ships only a PNG (or one
+   whose SVG fails to render in some future consumer) still resolves.
 2. **Schema drift safety** — if a future edition adds or renames
    fields, the validator should warn but not fail. v1 fails closed
    on missing required top-level/metadata/question keys but tolerates
    extra keys. We can soften this further if BNetzA evolves the
    format.
-3. **Packaging** — v1 stays a single-file script with `argparse`
+3. **Packaging** — both scripts stay single-file with `argparse`
-   (`amateurfunk_fetch.py`). Considered and declined: a
+   (`amateurfunk_fetch.py`, `amateurfunk_anki.py`). Considered and
-   `pyproject.toml` package with a console-script entry point. For a
+   declined: a `pyproject.toml` package with console-script entry
-   tool that runs at most quarterly, the packaging ceremony does not
+   points. For tools that run at most quarterly, the packaging
-   pay for itself. Revisit if the scope grows (e.g. importable from
+   ceremony does not pay for itself. Revisit if the scope grows
-   another project, distributed on PyPI).
+   (e.g. importable from another project, distributed on PyPI).
+
+---
+
+## 7. Stage 2 — Anki deck builder (`amateurfunk_anki.py`)
+
+### Goal
+
+Given the per-edition directory produced by Stage 1, build a set of
+Anki `.apkg` files that turn every catalog question into a flash card.
+Same input must produce byte-identical output across runs — this is
+important so generated decks can be checksummed and cached cleanly.
+
+### CLI shape
+
+```
+amateurfunk-anki [--data DIR] [--out DIR] [--seed STR] [--epoch INT]
+```
+
+- `--data DIR` — fetch output root (default `./data`). Must contain
+  `manifest-latest.json` pointing at a per-edition directory.
+- `--out DIR` — destination for `.apkg` files (default `./anki`).
+- `--seed STR` — deterministic seed for answer shuffling. The default
+  is fixed; changing it produces a different (but still deterministic)
+  shuffle.
+- `--epoch INT` — override the package timestamp epoch. By default we
+  derive it from the manifest's `fetched_at`; this flag is mainly for
+  tests and explicit rebuilds.
+
+Exit codes: `0` success, `1` configuration / catalog / build error.
+There is no Stage-2 equivalent of the fetcher's `EXIT_BAD_STATE` —
+the builder has no operator-recoverable local state, just generated
+output artifacts.
+
+### Output layout
+
+```
+anki/
+  amateurfunk-technische-kenntnisse.apkg          (~2.5 MB, 1374 cards)
+  amateurfunk-betriebliche-kenntnisse.apkg        (~130 KB,  172 cards)
+  amateurfunk-kenntnisse-von-vorschriften.apkg    (~75 KB,   204 cards)
+```
+
+One `.apkg` per top-level Prüfungsteil. The card counts match the
+study pools from §3 axis 1: Technische = 1374 (cumulative N+E+A);
+Betriebliche = 172 (shared across all candidates); Vorschriften = 204
+(also shared). License class is conveyed via tags
+(`klasse-N` / `klasse-E` / `klasse-A`), not separate packages —
+Anki users browse decks by topic and filter by class.
+
+### Steps
+
+1. **Load** the latest catalog: follow `manifest-latest.json` to a
+   per-edition directory, read the catalog JSON, the per-edition
+   `manifest.json`, and index the `svgs/` folder.
+2. **Categorize** the question tree into three `Category` objects,
+   one per top-level Prüfungsteil. Each category carries the flat
+   list of every question that lives anywhere under it, along with
+   each question's path through the section tree (used for the card
+   breadcrumb and the path tags).
+3. **For each category, render every question** as an Anki note:
+   - Compute a stable per-question seed from `--seed` + question
+     number; shuffle the A/B/C/D choices using it. The displayed
+     label (A/B/C/D) is assigned post-shuffle; the back of the card
+     names the *displayed* position of the correct answer.
+   - Render the question stem and answer texts to HTML via
+     `text_html()`, which tokenizes inline `$...$` math and rewrites
+     it to MathJax `\(...\)` delimiters (Anki 2.1+ ships MathJax
+     built in and recognizes those delimiters; bare `$...$` would
+     show as source). Escapes the rest for HTML safety, preserves
+     line breaks as `<br>`, and preserves `<u>...</u>` tags that
+     the catalog uses to emphasize negation in question stems.
+   - Resolve picture references through a per-category
+     `MediaRegistry`, which records every file actually used so the
+     packager can include only those.
+4. **Build a v11 Anki collection** as an in-memory SQLite database:
+   one `col` row carrying JSON config blobs (deck, model, dconf),
+   one `notes` row per question, one `cards` row per question.
+   Modern Anki understands v11 and upgrades the collection on
+   first open.
+5. **Package** the collection plus the referenced media into a
+   `.apkg` ZIP. Media is addressed by sequential integer keys
+   (Anki's convention); a `media` JSON map at the archive root
+   translates those keys back to filenames.
+6. **Post-process SVGs** at packaging time: inject a white
+   `<rect>` immediately after the opening `<svg>` tag so dark-mode
+   Anki users can still read the black-line BNetzA figures. The
+   extracted source SVGs on disk are never modified. The injection
+   is idempotent (marked with `data-af-white-background="1"`).
+7. **Write `.apkg` atomically**: build into `<out>.tmp`, then
+   `os.replace` into place. Stale `.tmp` directories are
+   overwritten on the next deterministic build — unlike the
+   fetcher's `.bak`/`.tmp` siblings, the Stage-2 temp file holds
+   no operator-recoverable state.
+
+### Determinism
+
+The contract is: same catalog in → same `.apkg` bytes out.
+Determinism rests on three things:
+
+1. **Stable IDs.** `stable_id(namespace, text)` hashes a
+   namespaced key with SHA-1 and squashes into the standard
+   13-digit Anki ID range. Different namespaces
+   (`"deck"` / `"model"` / `"note"` / `"card"`) keep IDs from
+   colliding across kinds; the namespaced inputs are stable
+   (`f"{category.slug}:{number}"` for notes and cards).
+   `stable_guid(text)` produces the 20-character note GUID used
+   for re-import deduplication.
+2. **Stable shuffle.** `randomized_answers()` builds a per-question
+   `random.Random` seeded from SHA-256 of
+   `f"{cli_seed}:{question_number}"`.
+3. **Stable timestamps.** Every `now` value in the collection (the
+   `mod` columns, the JSON config blob timestamps) is fixed to
+   `build_epoch` — derived from the manifest's `fetched_at`, or
+   overridden via `--epoch`. ZIP member timestamps are also fixed
+   via `ZipInfo(name, zip_datetime(build_epoch))`. Without this
+   last step, the inner SQLite would be identical but the
+   archive's per-entry mtimes would still vary between runs.
+
+The combined effect: two runs with the same `data/` produce
+byte-identical sha256 on each `.apkg`. Verified during review.
+
+### Rendering decisions worth knowing
+
+- **A is always correct upstream, then shuffled.** This is the
+  consumer-side caveat from §3 made concrete: we set the
+  `correct` flag on `answer_a` before shuffling, then carry it
+  through. The back of the card reveals the *displayed* position
+  ("Richtige Antwort: B"), not the source position.
+- **LaTeX → MathJax.** The catalog ships ~430 fragments
+  containing inline `$...$` (DESIGN §3 quotes the upstream README
+  on this). Anki 2.1+ MathJax recognizes `\(...\)` natively;
+  `$...$` does not render. The rewrite happens in `text_html()`.
+- **Safe inline tags.** The catalog uses `<u>...</u>` for
+  emphasis. Naive HTML escaping would convert those to
+  `&lt;u&gt;...&lt;/u&gt;` and lose the emphasis. We split on a
+  regex that matches exactly `<u>` and `</u>` and escape only the
+  non-tag pieces.
+- **Slug for filenames.** `slugify()` maps German umlauts and ß
+  to ASCII digraphs before NFKD-normalizing the rest, then keeps
+  only `[a-z0-9-]`. Yields readable, sortable filenames
+  (`amateurfunk-technische-kenntnisse.apkg`).
+- **Tags.** Each note carries `klasse-N|E|A` plus
+  `pfad-<slugified-section>` for every section level below the
+  top-level Prüfungsteil. Per-question number tags are
+  deliberately *not* emitted — they would create ~1750 singletons
+  in Anki's tag tree, and the number is already in the dedicated
+  `Number` field for search.
+- **Breadcrumb consistency.** The visible card breadcrumb and the
+  stored `Path` field both go through one `display_path()`
+  helper, so they never drift. The boilerplate
+  `Prüfungsfragen im Prüfungsteil:` prefix is stripped at this
+  layer.
+- **SVG white background.** BNetzA SVGs are transparent with
+  black line work. In Anki's dark mode the lines disappear into
+  the card background. We inject a white `<rect>` as the first
+  painted element when packaging the SVG into the `.apkg`. The
+  on-disk extracted files are left untouched.
+
+### Testing focus
+
+Network-free, fixture-driven. Real Anki is NOT required to
+verify the output — tests open the `.apkg` ZIP directly, parse
+the embedded SQLite, and check structural invariants.
+
+- Top-level structure: one `.apkg` per Prüfungsteil; expected
+  filename slugs.
+- Note count and field shape per category.
+- Tag mapping: a class-1 question carries `klasse-N`, no
+  `nummer-*` tags.
+- Breadcrumb rendering: both the visible HTML breadcrumb and the
+  stored `Path` field contain the short form; neither contains
+  the Prüfungsteil prefix.
+- LaTeX rewrite: a fixture question with `$...$` produces
+  `\(...\)` in the rendered HTML; a non-math `<` is still
+  escaped to `&lt;`; safe inline `<u>...</u>` tags survive
+  escaping.
+- Answer shuffle: the front HTML contains the answer texts in a
+  non-A-first order; the back names the displayed letter of the
+  correct answer.
+- Picture handling on shuffled answers: a fixture question whose
+  `picture_a`/`picture_b`/... fields point at different files
+  carries the right image with the right shuffled choice.
+- Media inclusion: `MediaRegistry.used_paths` is populated by
+  resolution and only those files end up in the `.apkg`.
+- SVG white background injection: idempotent; the marker rect is
+  emitted exactly once.
+- Deterministic build: two consecutive builds against the same
+  catalog with a fixed `--epoch` produce byte-identical `.apkg`
+  files.
+
+### What we deliberately do NOT do in Stage 2
+
+- No editable cloze cards or "type the answer" templates — the
+  exam is multiple choice, and the deck mirrors that.
+- No per-class packages. Class is a tag axis; package split is
+  by Prüfungsteil.
+- No automatic edition diffs / "new questions since last build"
+  decks. Useful but out of scope.
+- No upload to AnkiWeb. Decks are local artifacts.

test_amateurfunk_anki.py

@@ -4,16 +4,14 @@ These tests inspect the generated `.apkg` package structure directly:
 ZIP entries, media map, and the SQLite collection database. They do not
 require Anki itself to be installed.
 """
-from __future__ import annotations
 
 import json
 import sqlite3
 import tempfile
+import unittest
 import zipfile
 from pathlib import Path
 
-import unittest
-
 import amateurfunk_anki as aa
 
 

test_amateurfunk_fetch.py

@@ -4,7 +4,6 @@ Network-bound tests are not included; the real BNetzA fetch lives in
 the manual smoke-test invocation. Everything here runs against fixture
 ZIPs built in temp directories.
 """
-from __future__ import annotations
 
 import hashlib
 import json