imple/Amateurfunk-Anki
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
8fc3c0c32ecd93ad9bb72b64fafc77f7fe11a30c
Amateurfunk-Anki/DESIGN.md
T

727 lines
34 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Design Notes — BNetzA Amateurfunk Question Catalog → Anki
Status: **both stages implemented.** Stage 1 (`amateurfunk_fetch.py`)
downloads and validates the upstream ZIP; Stage 2
(`amateurfunk_anki.py`) turns the extracted data into Anki decks. This
document captures the source-discovery work, the JSON schema, and the
design contracts that both scripts implement. Everything below was
verified against the live BNetzA site in May 2026.
---
## 1. Source of the latest questions
The Bundesnetzagentur (BNetzA) is the authoritative publisher. They
provide the exam catalog in two parallel formats from the same landing
page:
- **PDF** — human-readable, ~5 MB.
- **ZIP** — machine-readable, ~3 MB, contains JSON + SVG figures. This
is the format we use.
### Landing page
- Short URL: `https://www.bnetza.de/amateurfunk-fragenkatalog`
- HTTP 301 → `SharedDocs/Downloads/.../Fragenkatalog/KurzURLFragenkatalog.html`
- The short URL points at an HTML page, not at the ZIP itself. We do
not use it for fetching; it is useful only as a citation/attribution
target.
### Direct download URLs (verified 2026-05-20)
- **ZIP (what we fetch):**
`https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/PruefungsfragenZIP.zip?__blob=publicationFile`
- HTTP 200, `Content-Type: application/zip`, `Accept-Ranges: bytes`,
serves a fresh `Last-Modified` header. Suitable for conditional
fetch and idempotency checks.
- **PDF (not used by this tool):**
`https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/Pruefungsfragen.pdf?__blob=publicationFile`
The `?__blob=publicationFile` query string is required — without it the
CMS serves an HTML wrapper, not the binary.
### License
`DL-DE→BY-2.0` (Datenlizenz Deutschland – Namensnennung – Version 2.0,
see `www.govdata.de/dl-de/by-2-0`). Commercial and non-commercial reuse
is permitted with attribution. The exact attribution string required is
spelled out in the ZIP's `README.txt`; we copy it verbatim into any
redistribution and into our `manifest.json`.
### Edition tracking
BNetzA does not version the URL. The same `PruefungsfragenZIP.zip`
path is updated in place when a new edition is published. To detect a
new edition we rely on:
1. The HTTP `Last-Modified` response header.
2. The sha256 of the downloaded ZIP.
3. The `metadata` block inside the JSON (`edition`, `issued_on`,
`valid_from`).
Current edition observed: `3. Auflage, März 2024` (issued 2024-03-20,
valid from 2024-06-24).
---
## 2. ZIP contents (verified by extracting the live file)
```
fragenkatalog3b.json — single JSON file, full question tree (~1.3 MB)
README.txt — license + schema documentation
svgs/ — 700+ figures (mostly SVG, a few PNG fallbacks)
AB108_q.svg
AB404_a.svg
AB404_b.svg
...
NG302_q.png — a couple of questions ship a PNG alongside
NG302_q.svg the SVG for display-problem fallbacks
```
Total: 706 entries, 701 of them files. ~1750 questions across classes
N, E, A (counts observed: N=571, E=463, A=716).
The JSON filename encodes the edition: `fragenkatalog3b.json` =
3rd edition, revision b. A future edition will rename this file
(`fragenkatalog4a.json`, etc.), so the loader must discover it from
the archive (glob `fragenkatalog*.json`), not hard-code the name.
### `README.txt` highlights
- Confirms the two-part structure: JSON for catalog/questions, SVG for
images.
- Confirms the JSON schema (see section 3).
- Notes that question text may contain **LaTeX** for formulas, intended
to be rendered by something like KaTeX.
---
## 3. JSON schema
Top-level object:
```jsonc
{
"metadata": {
"edition": "3. Auflage, März 2024",
"issued_on": "2024-03-20",
"valid_from": "2024-06-24",
"license": "DL-DE->BY-2.0"
},
"sections": [ /* recursive section nodes */ ]
}
```
Each `section` node is either an **inner node** (contains nested
`sections`) or a **leaf** (contains a `questions` list):
```jsonc
// inner
{ "title": "Prüfungsfragen im Prüfungsteil: Technische Kenntnisse",
"sections": [ ... ] }
// leaf
{ "title": "Allgemeine mathematische Grundkenntnisse und Größen",
"questions": [ /* question objects */ ] }
```
Question object (fields per the upstream README):
| field | meaning |
|--------------------|-----------------------------------------------------------|
| `number` | Catalog id, e.g. `NA103`, `AB404`. Also the SVG basename. |
| `class` | License class: `"1"` = N, `"2"` = E, `"3"` = A. |
| `question` | Question text. May contain LaTeX. |
| `answer_a` | **The correct answer.** Always A. |
| `answer_b` | Distractor. |
| `answer_c` | Distractor. |
| `answer_d` | Distractor. |
| `picture_question` | Optional. Figure shown with the question stem. |
| `picture_a`..`_d` | Optional. Per-choice figures. |
The `picture_*` fields, when present, contain a **basename without
extension** (e.g. `AB404_q`, not `AB404_q.svg`). The consumer picks
the extension. Convention observed in the archive:
- `<number>_q` for the question figure (file `<number>_q.svg`),
- `<number>_a` .. `<number>_d` for per-answer figures.
Files live under `svgs/`. A small number of entries ship a `.png`
alongside the `.svg` (display-problem fallback). Consumers should
prefer `.svg` and fall back to `.png` by stem.
This stem-only convention is why the soft picture-reference check
matches stems, not full filenames; matching the literal value
against directory listings would mark every reference missing.
### Important consumer-side conventions
- **Answer A is always correct.** Anything that presents the questions
to a learner must shuffle A/B/C/D before display, otherwise the
exercise is trivial.
- LaTeX in the question text and answers is unescaped — consumers
render it (e.g. KaTeX/MathJax). The downloader does not transform it.
- Classes are stored as string digits, not letter codes — map `"1"→N`,
`"2"→E`, `"3"→A` for display.
Sample question (from `fragenkatalog3b.json`):
```json
{
"number": "NA103",
"class": "1",
"question": "Laut Datenblatt wiegen 100 m eines bestimmten Drahtes 210 g. Ein vorliegendes Drahtstück desselben Materials wiegt 55 g. Wie lang ist das Drahtstück in etwa?",
"answer_a": "26,2 m",
"answer_b": "382 m",
"answer_c": "115 m",
"answer_d": "38,2 m"
}
```
### Exam structure — how the catalog splits into exam parts and classes
The catalog already encodes two orthogonal axes. The downloader does
not slice the data on disk, but any consumer needs to understand both
or they will compute the wrong candidate study pool.
**Axis 1 — Exam part (Prüfungsteil).** The top-level `sections[]`
array has exactly three entries, one per exam part:
| Top-level `title` | Question count | ID prefix |
|----------------------------------------------------------------|---------------:|----------------|
| `Prüfungsfragen im Prüfungsteil: Technische Kenntnisse` | 1374 | `N*`/`E*`/`A*` |
| `Prüfungsfragen im Prüfungsteil: Betriebliche Kenntnisse` | 172 | `B*` |
| `Prüfungsfragen im Prüfungsteil: Kenntnisse von Vorschriften` | 204 | `V*` |
Consumers can split on the section title (canonical) or on the
question `number` first letter (shorthand). The first-letter mapping
is: `A`/`E`/`N` → Technische; `B` → Betriebliche; `V` → Vorschriften.
Inside Technische, the first letter additionally mirrors the license
class (see Axis 2).
**Axis 2 — License class (`class` field on each question).**
Values are `"1"`=N, `"2"`=E, `"3"`=A. Class distribution per exam
part (counts verified against the live catalog, 3rd edition):
| Exam part | class 1 (N) | class 2 (E) | class 3 (A) |
|-------------|------------:|------------:|------------:|
| Technische | 195 | 463 | 716 |
| Betriebliche| 172 | 0 | 0 |
| Vorschriften| 204 | 0 | 0 |
Two things a consumer must know:
1. **Operational + Regulations are class-1-only in the data, but
apply to every candidate.** BNetzA treats these as a shared
foundation. Do not filter them by `class`.
2. **In Technische, the `class` field is a floor, not an equality
marker.** German amateur-radio exam knowledge is cumulative: a
class-E candidate is expected to know everything at class N and
E; a class-A candidate knows class N + E + A. Treating `class`
as equality underreports the E and A study pools.
The candidate study pools work out as:
- **N**: 195 (Tech class 1) + 172 (Betr) + 204 (Vor) = **571**
- **E**: (195 + 463) Tech + 172 + 204 = **1034**
- **A**: (195 + 463 + 716) Tech + 172 + 204 = **1750**
The 1750 total exactly matches the full catalog, which confirms the
floor interpretation: an A candidate's pool is the entire catalog.
**Downloader scope.** v1 does not split data on disk — the JSON tree
carries both axes already and consumers slice it themselves. This
subsection exists so that future consumers (study app, flashcards,
diff tool) implement the slicing correctly without re-deriving it
from the data.
---
## 4. Stage 1 — Fetcher (`amateurfunk_fetch.py`)
### Goal
Given no arguments, fetch the current BNetzA ZIP, extract it into a
clean per-edition directory, and write a manifest. Re-running is a
no-op when the upstream file has not changed.
### CLI shape
```
amateurfunk-fetch [--out DIR] [--force] [--keep-zip]
```
- `--out DIR` — output root (default `./data`). Each edition lands in
`DIR/<edition-slug>/`, e.g. `data/2024-03-20-3-auflage/`.
- `--force` — re-download and re-extract even if the existing manifest
matches.
- `--keep-zip` — keep the raw ZIP alongside the extracted tree (for
archival). Default deletes it after successful extraction.
Exit codes: `0` success (extracted or up-to-date), `1` network /
validation error, `2` invalid local state (e.g. a partial previous run
the tool can't reconcile without `--force`).
### Steps
1. **HEAD** the ZIP URL to read `Last-Modified` (and `Content-Length`
for a basic sanity range). If `DIR/manifest-latest.json` exists
and its `http_last_modified` equals the current server value AND
the target `DIR/<slug>/manifest.json` it points at is present and
parseable, exit 0 unchanged. The manifest — not the raw ZIP — is
the trusted record after a successful validated extraction; the
ZIP sha256 stays in the manifest as provenance, not as something
we re-verify on every run. (We delete the ZIP by default; there
would be nothing to re-verify against.)
2. **GET** the ZIP to a temp file in `DIR/.tmp/`. Stream to disk,
compute sha256 on the fly. Enforce a compressed max size (e.g.
50 MB) and, after open, a total uncompressed max size (e.g.
200 MB) as a guardrail against zip-bomb-style upstream regressions.
3. **Validate**, failing closed on structural problems:
- `zipfile.is_zipfile()` is true.
- All ZIP paths are relative and normalized: reject any entry
whose name is absolute, contains a `..` segment, or whose
`os.path.normpath`-result differs in a way that escapes the
extraction root — defends against zip-slip.
- **No symlink entries.** ZIPs created on Unix encode symlinks in
the upper 16 bits of `ZipInfo.external_attr` (the POSIX mode
field). Reject any entry where
`(zi.external_attr >> 16) & 0o170000 == 0o120000` (S_IFLNK),
in addition to the path checks above.
- Sum of uncompressed sizes is below the configured cap.
- Archive contains **exactly one** root-level `fragenkatalog*.json`.
- Archive contains **more than 100 file entries** whose normalized
path starts with `svgs/`. We do not require a standalone `svgs/`
directory record — many ZIP producers omit directory entries
and only emit file entries like `svgs/AB108_q.svg`.
- The JSON parses and has top-level keys `metadata` and `sections`.
- `metadata` carries the required keys `edition`, `issued_on`,
`valid_from`, `license`. Extra keys are tolerated.
- Every section node has either nested `sections` or a `questions`
list (not both, not neither).
- Every question has `number`, `class`, `question`, `answer_a`,
`answer_b`, `answer_c`, `answer_d`.
- For every `picture_*` reference in a question, the named file
exists under `svgs/` in the archive. **Soft check:** missing
references do not fail the extraction. They are collected into
the manifest as `missing_pictures` so consumers can decide
locally. Rationale: a single upstream typo should not brick the
downloader for every consumer until BNetzA ships a fix.
4. **Derive edition slug** from `metadata.issued_on` plus an edition
ordinal parsed from `metadata.edition`, e.g.
`"3. Auflage, März 2024"` + `2024-03-20` → `2024-03-20-3-auflage`.
The ordinal is taken from the leading `\d+` in the edition string;
if absent, fall back to `unknown-auflage`. Sortable, filesystem-
safe, and independent of German month-name parsing. The full
upstream `metadata.edition` is preserved verbatim in `manifest.json`.
5. **Extract** into `DIR/<slug>.tmp/`. Replacement semantics:
- If `DIR/<slug>/manifest.json` already exists and its recorded
`zip_sha256` matches the freshly downloaded ZIP, skip extraction,
update `manifest-latest.json` if needed, and exit 0.
- If `DIR/<slug>/` exists but does not match and `--force` is not
set, exit 2 with a clear message naming the offending directory.
- With `--force`, after extraction completes into `DIR/<slug>.tmp/`,
rename the existing `DIR/<slug>/` to `DIR/<slug>.bak/`, rename
`DIR/<slug>.tmp/` into place, then remove `DIR/<slug>.bak/`. A
crash between the two renames leaves a recoverable `.bak`
directory.
- **`.bak` collision policy:** if `DIR/<slug>.bak/` already exists
before the forced replacement starts, exit 2 with a clear error
naming the stale `.bak/` path. A leftover `.bak/` is evidence
that a previous run crashed mid-rename; deciding whether to keep
or delete it is the operator's call, not ours. The same applies
to a stale `DIR/<slug>.tmp/`: refuse to overwrite, surface the
path. Both checks happen *before* any rename — no destructive
action without a clean predecessor state.
Always copy the upstream `README.txt` verbatim into the extracted
tree. The manifest also records the attribution string for
convenience, but the file is the source of truth — lossless
preservation beats parser-derived fields.
6. **Write `DIR/<slug>/manifest.json`** with:
- `source_url`
- `fetched_at` (ISO 8601 UTC)
- `http_last_modified` (verbatim from the server)
- `zip_sha256`
- `zip_size`
- `json_filename` (the actual `fragenkatalog*.json` we found)
- the full upstream `metadata` block
- the verbatim attribution string from `README.txt`
- `missing_pictures` (array of `{question_number, field, file}`,
empty when the archive is clean)
7. **Atomically update `DIR/manifest-latest.json`** to point at the
current slug. Precise sequence:
1. Write the new content to a sibling `manifest-latest.json.tmp`
in the same directory as the target.
2. `os.fsync(tmp_fd)` to flush the tmp file's contents to disk.
3. `os.replace(tmp_path, final_path)` to atomically swap.
4. On POSIX, open the containing directory and `os.fsync` its
file descriptor so the rename itself is durable. Wrap in a
`try/except OSError` and ignore on platforms where directory
fsync is not supported (e.g. Windows) — the swap is still
atomic, only its on-disk persistence guarantee differs.
A symlink at `manifest-latest` would be nicer on POSIX but a
small pointer file is portable.
8. **Clean up** the temp ZIP (unless `--keep-zip`).
### Idempotency and safety
- Never extract directly into the final directory — always into a
sibling `*.tmp` that is renamed on success. A crash mid-extract leaves
the previous good edition untouched.
- Network calls go through `urllib.request` with a sane User-Agent
(`amateurfunk-fetch/<version> (+contact)`) and a timeout. Single
retry on transient errors (no exponential-backoff library needed for
a once-a-quarter download).
- No telemetry. No mutation outside `--out`.
### Testing focus
The most valuable tests are behavioral, not network-bound. The real
BNetzA fetch stays as an opt-in integration test (skipped by default);
everything else runs against fixture ZIPs.
- Slug generation from sample `metadata` (covers normal, missing
ordinal, unusual edition strings).
- ZIP path rejection: absolute paths, `..` segments, symlinks.
- Uncompressed-size cap triggers cleanly.
- Validation failures: missing JSON, multiple `fragenkatalog*.json`,
missing `svgs/`, malformed JSON, missing required top-level keys,
missing required metadata/question keys, malformed section nodes.
- `missing_pictures` is populated but extraction still succeeds when
a `picture_*` reference doesn't resolve (soft check).
- Manifest is well-formed and contains the upstream attribution and
`metadata` block verbatim.
- **`Last-Modified` round-trip**: after a successful extraction, a
rerun with the same `Last-Modified` header on the HEAD response
skips the GET entirely. This is the actual contract idempotency
hangs on now that the ZIP is deleted by default.
- `--force` path: existing non-matching `DIR/<slug>/` is replaced via
the temp/bak rename dance, and a simulated crash between the two
renames leaves a recoverable `.bak/`.
- Atomic write of `manifest-latest.json`: a write that fails partway
does not corrupt the existing pointer file.
### What we deliberately do NOT do in v1
- No PDF mirroring.
- No question rendering (LaTeX, SVG).
- No splitting by class / chapter on disk — the JSON tree already
carries that structure and consumers can slice it themselves.
- No diffing between editions. Useful, but a separate tool that reads
two manifest dirs.
- No mirror to S3 / a release artifact. Out of scope unless asked.
---
## 5. Resolved during review
These were open in the first draft and have been settled:
1. **Edition slug format** — resolved: derive from `issued_on` +
numeric edition ordinal, e.g. `2024-03-20-3-auflage`. Avoids
parsing German month names; preserves day-precision; sortable.
2. **`manifest-latest.json` location** — at `DIR/` root (one stable
pointer for consumers); per-edition `manifest.json` is the
immutable record.
3. **Idempotency without a retained ZIP** — resolved: the manifest is
the trusted record after extraction. `Last-Modified` match is
sufficient to skip the download; `zip_sha256` is recorded for
provenance only.
4. **README.txt preservation** — copy verbatim into the extracted
tree alongside any derived attribution field in the manifest.
5. **Missing picture references** — soft-validate (record in
`missing_pictures`, do not fail). Rationale: avoid bricking the
tool on an upstream typo.
## 6. Open questions (Stage 1)
These do not block the Stage 1 implementation:
1. **PNG fallbacks** — the archive ships PNGs for ~a handful of
figures alongside the SVGs. The fetcher just extracts everything
as-is. Stage 2 prefers SVG and falls back to PNG by stem:
`MediaRegistry.resolve()` tries `<stem>.svg` first, then
`<stem>.png`. So an upstream entry that ships only a PNG (or one
whose SVG fails to render in some future consumer) still resolves.
2. **Schema drift safety** — if a future edition adds or renames
fields, the validator should warn but not fail. v1 fails closed
on missing required top-level/metadata/question keys but tolerates
extra keys. We can soften this further if BNetzA evolves the
format.
3. **Packaging** — both scripts stay single-file with `argparse`
(`amateurfunk_fetch.py`, `amateurfunk_anki.py`). Considered and
declined: a `pyproject.toml` package with console-script entry
points. For tools that run at most quarterly, the packaging
ceremony does not pay for itself. Revisit if the scope grows
(e.g. importable from another project, distributed on PyPI).
---
## 7. Stage 2 — Anki deck builder (`amateurfunk_anki.py`)
### Goal
Given the per-edition directory produced by Stage 1, build a set of
Anki `.apkg` files that turn every catalog question into a flash card.
By default, each build uses a fresh shuffle seed so repeated imports
vary the answer order. When `--seed` and `--epoch` are supplied,
the same input must produce byte-identical output across runs — useful
for tests, checksums, and cacheable release builds.
### CLI shape
```
amateurfunk-anki [--data DIR] [--out DIR] [--seed STR] [--epoch INT]
```
- `--data DIR` — fetch output root (default `./data`). Must contain
`manifest-latest.json` pointing at a per-edition directory.
- `--out DIR` — destination for `.apkg` files (default `./anki`).
- `--seed STR` — deterministic seed for answer shuffling. Omit it
for the normal study-deck behavior: a fresh seed is generated for
each build, so answers are reshuffled every time decks are rebuilt.
- `--epoch INT` — override the package timestamp epoch. By default we
derive it from the manifest's `fetched_at`; this flag is mainly for
tests and explicit rebuilds.
Exit codes: `0` success, `1` configuration / catalog / build error.
There is no Stage-2 equivalent of the fetcher's `EXIT_BAD_STATE` —
the builder has no operator-recoverable local state, just generated
output artifacts.
### Output layout
```
anki/
amateurfunk-technische-kenntnisse-n.apkg (195 cards)
amateurfunk-technische-kenntnisse-e.apkg (463 cards, 11 sub-decks)
amateurfunk-technische-kenntnisse-a.apkg (716 cards, 11 sub-decks)
amateurfunk-betriebliche-kenntnisse.apkg (172 cards)
amateurfunk-kenntnisse-von-vorschriften.apkg (204 cards)
```
Betriebliche and Vorschriften are shared across every candidate
(class-1-only in the data per §3 axis 2) and stay as one deck each.
Technische is fanned out per license class using a **strict equality
split** on the question's `class` field — class-1 questions land in
the N deck only, class-2 in E only, class-3 in A only. The card counts
therefore equal the new-at-this-class slices from §3, not the
cumulative study pools: a candidate studying for class E imports
Technische-N + Technische-E + Betriebliche + Vorschriften.
The class-E (463) and class-A (716) Technische pools are large enough
that a single flat deck is unwieldy, so each package is built as a deck
*tree* rather than fanned out into separate files: one `.apkg`
containing one sub-deck per first-level catalog topic (the 11
subsections under the Prüfungsteil, e.g. "Sender und Empfänger",
"Antennen und Übertragungsleitungen") plus an anchoring `…::E` / `…::A`
parent deck. Each card is filed under its topic via the `did` column;
grouping is by the first section title below the Prüfungsteil
(`item.path[1]`). Anki sorts the sub-decks alphabetically on import (it
has no per-deck manual ordering), not in catalog order. N (195) is
small enough to stay one flat deck. The split classes are listed in
`TOPIC_SPLIT_CLASSES`. These multi-deck packages list every sub-deck in
the `col.decks` blob (see `build_apkg_for_category` and
`insert_collection_metadata`).
Note that re-importing a reworked package does not move cards that
already exist in a collection — Anki only files *new* cards by the
package's decks and leaves existing cards in their current deck.
The Technische deck names use Anki's `::` hierarchy separator
(`Amateurfunk::Technische Kenntnisse::N`, and one level deeper for E/A —
`Amateurfunk::Technische Kenntnisse::A::Sender und Empfänger`) so the
decks render as a nested tree in Anki's deck browser. The
`klasse-N` / `klasse-E` / `klasse-A` tag is still emitted on every
note — redundant within each Technische deck but useful in Betr/Vor
for in-Anki filtering, and harmless besides.
### Steps
1. **Load** the latest catalog: follow `manifest-latest.json` to a
per-edition directory, read the catalog JSON, the per-edition
`manifest.json`, and index the `svgs/` folder.
2. **Categorize** the question tree into five `Category` objects.
Betriebliche and Vorschriften get one each. Technische is
additionally split into three sub-categories (one per license
class) via a strict equality match on the question's `class`
field. Each category carries the flat list of every question that
lives anywhere under it, along with each question's path through
the section tree (used for the card breadcrumb and the path
tags).
3. **For each category, render every question** as an Anki note:
- Compute a stable per-question seed from `--seed` + question
number; shuffle the A/B/C/D choices using it. The displayed
label (A/B/C/D) is assigned post-shuffle; the back of the card
names the *displayed* position of the correct answer.
- Render the question stem and answer texts to HTML via
`text_html()`, which tokenizes inline `$...$` math and rewrites
it to MathJax `\(...\)` delimiters (Anki 2.1+ ships MathJax
built in and recognizes those delimiters; bare `$...$` would
show as source). Escapes the rest for HTML safety, preserves
line breaks as `<br>`, and preserves `<u>...</u>` tags that
the catalog uses to emphasize negation in question stems.
- Resolve picture references through a per-category
`MediaRegistry`, which records every file actually used so the
packager can include only those.
4. **Build a v11 Anki collection** as an in-memory SQLite database:
one `col` row carrying JSON config blobs (deck, model, dconf),
one `notes` row per question, one `cards` row per question.
Modern Anki understands v11 and upgrades the collection on
first open.
5. **Package** the collection plus the referenced media into a
`.apkg` ZIP. Media is addressed by sequential integer keys
(Anki's convention); a `media` JSON map at the archive root
translates those keys back to filenames.
6. **Post-process SVGs** at packaging time: inject a white
`<rect>` immediately after the opening `<svg>` tag so dark-mode
Anki users can still read the black-line BNetzA figures. The
extracted source SVGs on disk are never modified. The injection
is idempotent (marked with `data-af-white-background="1"`).
7. **Write `.apkg` atomically**: build into `<out>.tmp`, then
`os.replace` into place. Stale `.tmp` directories are
overwritten on the next deterministic build — unlike the
fetcher's `.bak`/`.tmp` siblings, the Stage-2 temp file holds
no operator-recoverable state.
### Determinism
The default CLI intentionally is not byte-deterministic because it
generates a fresh shuffle seed each run. The reproducible-build
contract is: same catalog + same `--seed` + same timestamp inputs →
same `.apkg` bytes out. Determinism rests on three things:
1. **Stable IDs.** `stable_id(namespace, text)` hashes a
namespaced key with SHA-1 and squashes into the standard
13-digit Anki ID range. Different namespaces
(`"deck"` / `"model"` / `"note"` / `"card"`) keep IDs from
colliding across kinds; the namespaced inputs are stable
(`f"{category.slug}:{number}"` for notes and cards).
`stable_guid(text)` produces the 20-character note GUID used
for re-import deduplication.
2. **Stable shuffle.** `randomized_answers()` builds a per-question
`random.Random` seeded from SHA-256 of
`f"{cli_seed}:{question_number}"`.
3. **Stable timestamps.** Every `now` value in the collection (the
`mod` columns, the JSON config blob timestamps) is fixed to
`build_epoch` — derived from the manifest's `fetched_at`, or
overridden via `--epoch`. ZIP member timestamps are also fixed
via `ZipInfo(name, zip_datetime(build_epoch))`. Without this
last step, the inner SQLite would be identical but the
archive's per-entry mtimes would still vary between runs.
The combined effect: two runs with the same `data/`, `--seed`, and
timestamp inputs produce byte-identical sha256 on each `.apkg`.
Verified during review.
### Rendering decisions worth knowing
- **A is always correct upstream, then shuffled.** This is the
consumer-side caveat from §3 made concrete: we set the
`correct` flag on `answer_a` before shuffling, then carry it
through. The back of the card reveals the *displayed* position
("Richtige Antwort: B"), not the source position.
- **LaTeX → MathJax.** The catalog ships ~430 fragments
containing inline `$...$` (DESIGN §3 quotes the upstream README
on this). Anki 2.1+ MathJax recognizes `\(...\)` natively;
`$...$` does not render. The rewrite happens in `text_html()`.
- **Safe inline tags.** The catalog uses `<u>...</u>` for
emphasis. Naive HTML escaping would convert those to
`&lt;u&gt;...&lt;/u&gt;` and lose the emphasis. We split on a
regex that matches exactly `<u>` and `</u>` and escape only the
non-tag pieces.
- **Slug for filenames.** `slugify()` maps German umlauts and ß
to ASCII digraphs before NFKD-normalizing the rest, then keeps
only `[a-z0-9-]`. Yields readable, sortable filenames
(`amateurfunk-technische-kenntnisse-n.apkg`). The three
Technische decks append the class letter (`-n`, `-e`, `-a`)
to the shared base slug; Betriebliche and Vorschriften use the
base slug as-is.
- **Tags.** Each note carries `klasse-N|E|A` plus
`pfad-<slugified-section>` for every section level below the
top-level Prüfungsteil. The `klasse-*` tag is redundant within
the three Technische decks (every card already shares one class)
but is still emitted there for uniformity, and remains the
primary filter axis inside the Betriebliche/Vorschriften decks.
Per-question number tags are deliberately *not* emitted — they
would create ~1750 singletons in Anki's tag tree, and the number
is already in the dedicated `Number` field for search.
- **Breadcrumb consistency.** The visible card breadcrumb and the
stored `Path` field both go through one `display_path()`
helper, so they never drift. The boilerplate
`Prüfungsfragen im Prüfungsteil:` prefix is stripped at this
layer.
- **SVG white background.** BNetzA SVGs are transparent with
black line work. In Anki's dark mode the lines disappear into
the card background. We inject a white `<rect>` as the first
painted element when packaging the SVG into the `.apkg`. The
on-disk extracted files are left untouched.
- **Explanations layer (optional).** `amateurfunk_anki.py` loads
`explanations.json` (CLI: `--explanations`) and, for any
question number found there, appends an English explanation
block to the back of the card. The block is styled distinctly
(serif body, sans-serif metadata, top border) and shows
a small "low confidence" badge when the entry's `confidence`
field is below `LOW_CONFIDENCE_THRESHOLD` (= 7). `revision` and
the raw `confidence` number are editorial-only and never
displayed. A missing file is treated as an empty database; a
malformed entry is a hard `AnkiBuildError`. The editorial
contract (schema, sourcing, agent workflows) lives in
`EXPLANATIONS.md`; this file documents only the wiring on the
build side. Note-GUID stability across explanation edits: the
GUID is keyed on `category.slug:number`, not on field content,
so adding/editing an entry updates the existing Anki note on
re-import rather than producing a duplicate.
### Testing focus
Network-free, fixture-driven. Real Anki is NOT required to
verify the output — tests open the `.apkg` ZIP directly, parse
the embedded SQLite, and check structural invariants.
- Top-level structure: five `.apkg` files (Betriebliche +
Vorschriften + Technische×{N,E,A}); expected filename slugs.
- Technische class partition: each Technische deck contains
exactly the questions whose `class` field matches that letter,
and no others.
- Note count and field shape per category.
- Tag mapping: a class-1 question carries `klasse-N`, no
`nummer-*` tags.
- Breadcrumb rendering: both the visible HTML breadcrumb and the
stored `Path` field contain the short form; neither contains
the Prüfungsteil prefix.
- LaTeX rewrite: a fixture question with `$...$` produces
`\(...\)` in the rendered HTML; a non-math `<` is still
escaped to `&lt;`; safe inline `<u>...</u>` tags survive
escaping.
- Answer shuffle: the front HTML contains the answer texts in a
non-A-first order; the back names the displayed letter of the
correct answer.
- Picture handling on shuffled answers: a fixture question whose
`picture_a`/`picture_b`/... fields point at different files
carries the right image with the right shuffled choice.
- Media inclusion: `MediaRegistry.used_paths` is populated by
resolution and only those files end up in the `.apkg`.
- SVG white background injection: idempotent; the marker rect is
emitted exactly once.
- Deterministic build: two consecutive builds against the same
catalog with a fixed `--epoch` produce byte-identical `.apkg`
files.
- Explanations layer: a present entry produces a styled
explanation block on the back card; a missing entry leaves the
card unchanged; `confidence < 7` surfaces the "low confidence"
badge; URL sources render as `<a>`, citation sources render as
escaped text; a missing `explanations.json` is silently empty;
a malformed entry (bad type, out-of-range confidence, missing
field) raises `AnkiBuildError`.
### What we deliberately do NOT do in Stage 2
- No editable cloze cards or "type the answer" templates — the
exam is multiple choice, and the deck mirrors that.
- No per-class packages. Class is a tag axis; package split is
by Prüfungsteil.
- No automatic edition diffs / "new questions since last build"
decks. Useful but out of scope.
- No upload to AnkiWeb. Decks are local artifacts.
Reference in New Issue View Git Blame Copy Permalink