# Design Notes — BNetzA Amateurfunk Question Catalog Downloader Status: **design only, no code yet.** This document captures the source-discovery work and the proposed shape of the tool. Everything below was verified against the live BNetzA site in May 2026. --- ## 1. Source of the latest questions The Bundesnetzagentur (BNetzA) is the authoritative publisher. They provide the exam catalog in two parallel formats from the same landing page: - **PDF** — human-readable, ~5 MB. - **ZIP** — machine-readable, ~3 MB, contains JSON + SVG figures. This is the format we use. ### Landing page - Short URL: `https://www.bnetza.de/amateurfunk-fragenkatalog` - HTTP 301 → `SharedDocs/Downloads/.../Fragenkatalog/KurzURLFragenkatalog.html` - The short URL points at an HTML page, not at the ZIP itself. We do not use it for fetching; it is useful only as a citation/attribution target. ### Direct download URLs (verified 2026-05-20) - **ZIP (what we fetch):** `https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/PruefungsfragenZIP.zip?__blob=publicationFile` - HTTP 200, `Content-Type: application/zip`, `Accept-Ranges: bytes`, serves a fresh `Last-Modified` header. Suitable for conditional fetch and idempotency checks. - **PDF (not used by this tool):** `https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/Pruefungsfragen.pdf?__blob=publicationFile` The `?__blob=publicationFile` query string is required — without it the CMS serves an HTML wrapper, not the binary. ### License `DL-DE→BY-2.0` (Datenlizenz Deutschland – Namensnennung – Version 2.0, see `www.govdata.de/dl-de/by-2-0`). Commercial and non-commercial reuse is permitted with attribution. The exact attribution string required is spelled out in the ZIP's `README.txt`; we copy it verbatim into any redistribution and into our `manifest.json`. ### Edition tracking BNetzA does not version the URL. The same `PruefungsfragenZIP.zip` path is updated in place when a new edition is published. To detect a new edition we rely on: 1. The HTTP `Last-Modified` response header. 2. The sha256 of the downloaded ZIP. 3. The `metadata` block inside the JSON (`edition`, `issued_on`, `valid_from`). Current edition observed: `3. Auflage, März 2024` (issued 2024-03-20, valid from 2024-06-24). --- ## 2. ZIP contents (verified by extracting the live file) ``` fragenkatalog3b.json — single JSON file, full question tree (~1.3 MB) README.txt — license + schema documentation svgs/ — 700+ figures (mostly SVG, a few PNG fallbacks) AB108_q.svg AB404_a.svg AB404_b.svg ... NG302_q.png — a couple of questions ship a PNG alongside NG302_q.svg the SVG for display-problem fallbacks ``` Total: 706 entries, 701 of them files. ~1750 questions across classes N, E, A (counts observed: N=571, E=463, A=716). The JSON filename encodes the edition: `fragenkatalog3b.json` = 3rd edition, revision b. A future edition will rename this file (`fragenkatalog4a.json`, etc.), so the loader must discover it from the archive (glob `fragenkatalog*.json`), not hard-code the name. ### `README.txt` highlights - Confirms the two-part structure: JSON for catalog/questions, SVG for images. - Confirms the JSON schema (see section 3). - Notes that question text may contain **LaTeX** for formulas, intended to be rendered by something like KaTeX. --- ## 3. JSON schema Top-level object: ```jsonc { "metadata": { "edition": "3. Auflage, März 2024", "issued_on": "2024-03-20", "valid_from": "2024-06-24", "license": "DL-DE->BY-2.0" }, "sections": [ /* recursive section nodes */ ] } ``` Each `section` node is either an **inner node** (contains nested `sections`) or a **leaf** (contains a `questions` list): ```jsonc // inner { "title": "Prüfungsfragen im Prüfungsteil: Technische Kenntnisse", "sections": [ ... ] } // leaf { "title": "Allgemeine mathematische Grundkenntnisse und Größen", "questions": [ /* question objects */ ] } ``` Question object (fields per the upstream README): | field | meaning | |--------------------|-----------------------------------------------------------| | `number` | Catalog id, e.g. `NA103`, `AB404`. Also the SVG basename. | | `class` | License class: `"1"` = N, `"2"` = E, `"3"` = A. | | `question` | Question text. May contain LaTeX. | | `answer_a` | **The correct answer.** Always A. | | `answer_b` | Distractor. | | `answer_c` | Distractor. | | `answer_d` | Distractor. | | `picture_question` | Optional. Figure shown with the question stem. | | `picture_a`..`_d` | Optional. Per-choice figures. | The `picture_*` fields, when present, contain a filename (e.g. `AB404_q.svg`) that resolves into `svgs/`. Convention observed in the archive: `_q.svg` for the question figure, `_a.svg` .. `_d.svg` for per-answer figures. A small number of entries also ship a `.png` next to the `.svg`. ### Important consumer-side conventions - **Answer A is always correct.** Anything that presents the questions to a learner must shuffle A/B/C/D before display, otherwise the exercise is trivial. - LaTeX in the question text and answers is unescaped — consumers render it (e.g. KaTeX/MathJax). The downloader does not transform it. - Classes are stored as string digits, not letter codes — map `"1"→N`, `"2"→E`, `"3"→A` for display. Sample question (from `fragenkatalog3b.json`): ```json { "number": "NA103", "class": "1", "question": "Laut Datenblatt wiegen 100 m eines bestimmten Drahtes 210 g. Ein vorliegendes Drahtstück desselben Materials wiegt 55 g. Wie lang ist das Drahtstück in etwa?", "answer_a": "26,2 m", "answer_b": "382 m", "answer_c": "115 m", "answer_d": "38,2 m" } ``` ### Exam structure — how the catalog splits into exam parts and classes The catalog already encodes two orthogonal axes. The downloader does not slice the data on disk, but any consumer needs to understand both or they will compute the wrong candidate study pool. **Axis 1 — Exam part (Prüfungsteil).** The top-level `sections[]` array has exactly three entries, one per exam part: | Top-level `title` | Question count | ID prefix | |----------------------------------------------------------------|---------------:|----------------| | `Prüfungsfragen im Prüfungsteil: Technische Kenntnisse` | 1374 | `N*`/`E*`/`A*` | | `Prüfungsfragen im Prüfungsteil: Betriebliche Kenntnisse` | 172 | `B*` | | `Prüfungsfragen im Prüfungsteil: Kenntnisse von Vorschriften` | 204 | `V*` | Consumers can split on the section title (canonical) or on the question `number` first letter (shorthand). The first-letter mapping is: `A`/`E`/`N` → Technische; `B` → Betriebliche; `V` → Vorschriften. Inside Technische, the first letter additionally mirrors the license class (see Axis 2). **Axis 2 — License class (`class` field on each question).** Values are `"1"`=N, `"2"`=E, `"3"`=A. Class distribution per exam part (counts verified against the live catalog, 3rd edition): | Exam part | class 1 (N) | class 2 (E) | class 3 (A) | |-------------|------------:|------------:|------------:| | Technische | 195 | 463 | 716 | | Betriebliche| 172 | 0 | 0 | | Vorschriften| 204 | 0 | 0 | Two things a consumer must know: 1. **Operational + Regulations are class-1-only in the data, but apply to every candidate.** BNetzA treats these as a shared foundation. Do not filter them by `class`. 2. **In Technische, the `class` field is a floor, not an equality marker.** German amateur-radio exam knowledge is cumulative: a class-E candidate is expected to know everything at class N and E; a class-A candidate knows class N + E + A. Treating `class` as equality underreports the E and A study pools. The candidate study pools work out as: - **N**: 195 (Tech class 1) + 172 (Betr) + 204 (Vor) = **571** - **E**: (195 + 463) Tech + 172 + 204 = **1034** - **A**: (195 + 463 + 716) Tech + 172 + 204 = **1750** The 1750 total exactly matches the full catalog, which confirms the floor interpretation: an A candidate's pool is the entire catalog. **Downloader scope.** v1 does not split data on disk — the JSON tree carries both axes already and consumers slice it themselves. This subsection exists so that future consumers (study app, flashcards, diff tool) implement the slicing correctly without re-deriving it from the data. --- ## 4. Tool design (v1) ### Goal Given no arguments, fetch the current BNetzA ZIP, extract it into a clean per-edition directory, and write a manifest. Re-running is a no-op when the upstream file has not changed. ### CLI shape ``` amateurfunk-fetch [--out DIR] [--force] [--keep-zip] ``` - `--out DIR` — output root (default `./data`). Each edition lands in `DIR//`, e.g. `data/2024-03-20-3-auflage/`. - `--force` — re-download and re-extract even if the existing manifest matches. - `--keep-zip` — keep the raw ZIP alongside the extracted tree (for archival). Default deletes it after successful extraction. Exit codes: `0` success (extracted or up-to-date), `1` network / validation error, `2` invalid local state (e.g. a partial previous run the tool can't reconcile without `--force`). ### Steps 1. **HEAD** the ZIP URL to read `Last-Modified` (and `Content-Length` for a basic sanity range). If `DIR/manifest-latest.json` exists and its `http_last_modified` equals the current server value AND the target `DIR//manifest.json` it points at is present and parseable, exit 0 unchanged. The manifest — not the raw ZIP — is the trusted record after a successful validated extraction; the ZIP sha256 stays in the manifest as provenance, not as something we re-verify on every run. (We delete the ZIP by default; there would be nothing to re-verify against.) 2. **GET** the ZIP to a temp file in `DIR/.tmp/`. Stream to disk, compute sha256 on the fly. Enforce a compressed max size (e.g. 50 MB) and, after open, a total uncompressed max size (e.g. 200 MB) as a guardrail against zip-bomb-style upstream regressions. 3. **Validate**, failing closed on structural problems: - `zipfile.is_zipfile()` is true. - All ZIP paths are relative and normalized: reject any entry whose name is absolute, contains a `..` segment, or whose `os.path.normpath`-result differs in a way that escapes the extraction root — defends against zip-slip. - **No symlink entries.** ZIPs created on Unix encode symlinks in the upper 16 bits of `ZipInfo.external_attr` (the POSIX mode field). Reject any entry where `(zi.external_attr >> 16) & 0o170000 == 0o120000` (S_IFLNK), in addition to the path checks above. - Sum of uncompressed sizes is below the configured cap. - Archive contains **exactly one** root-level `fragenkatalog*.json`. - Archive contains **more than 100 file entries** whose normalized path starts with `svgs/`. We do not require a standalone `svgs/` directory record — many ZIP producers omit directory entries and only emit file entries like `svgs/AB108_q.svg`. - The JSON parses and has top-level keys `metadata` and `sections`. - `metadata` carries the required keys `edition`, `issued_on`, `valid_from`, `license`. Extra keys are tolerated. - Every section node has either nested `sections` or a `questions` list (not both, not neither). - Every question has `number`, `class`, `question`, `answer_a`, `answer_b`, `answer_c`, `answer_d`. - For every `picture_*` reference in a question, the named file exists under `svgs/` in the archive. **Soft check:** missing references do not fail the extraction. They are collected into the manifest as `missing_pictures` so consumers can decide locally. Rationale: a single upstream typo should not brick the downloader for every consumer until BNetzA ships a fix. 4. **Derive edition slug** from `metadata.issued_on` plus an edition ordinal parsed from `metadata.edition`, e.g. `"3. Auflage, März 2024"` + `2024-03-20` → `2024-03-20-3-auflage`. The ordinal is taken from the leading `\d+` in the edition string; if absent, fall back to `unknown-auflage`. Sortable, filesystem- safe, and independent of German month-name parsing. The full upstream `metadata.edition` is preserved verbatim in `manifest.json`. 5. **Extract** into `DIR/.tmp/`. Replacement semantics: - If `DIR//manifest.json` already exists and its recorded `zip_sha256` matches the freshly downloaded ZIP, skip extraction, update `manifest-latest.json` if needed, and exit 0. - If `DIR//` exists but does not match and `--force` is not set, exit 2 with a clear message naming the offending directory. - With `--force`, after extraction completes into `DIR/.tmp/`, rename the existing `DIR//` to `DIR/.bak/`, rename `DIR/.tmp/` into place, then remove `DIR/.bak/`. A crash between the two renames leaves a recoverable `.bak` directory. - **`.bak` collision policy:** if `DIR/.bak/` already exists before the forced replacement starts, exit 2 with a clear error naming the stale `.bak/` path. A leftover `.bak/` is evidence that a previous run crashed mid-rename; deciding whether to keep or delete it is the operator's call, not ours. The same applies to a stale `DIR/.tmp/`: refuse to overwrite, surface the path. Both checks happen *before* any rename — no destructive action without a clean predecessor state. Always copy the upstream `README.txt` verbatim into the extracted tree. The manifest also records the attribution string for convenience, but the file is the source of truth — lossless preservation beats parser-derived fields. 6. **Write `DIR//manifest.json`** with: - `source_url` - `fetched_at` (ISO 8601 UTC) - `http_last_modified` (verbatim from the server) - `zip_sha256` - `zip_size` - `json_filename` (the actual `fragenkatalog*.json` we found) - the full upstream `metadata` block - the verbatim attribution string from `README.txt` - `missing_pictures` (array of `{question_number, field, file}`, empty when the archive is clean) 7. **Atomically update `DIR/manifest-latest.json`** to point at the current slug. Precise sequence: 1. Write the new content to a sibling `manifest-latest.json.tmp` in the same directory as the target. 2. `os.fsync(tmp_fd)` to flush the tmp file's contents to disk. 3. `os.replace(tmp_path, final_path)` to atomically swap. 4. On POSIX, open the containing directory and `os.fsync` its file descriptor so the rename itself is durable. Wrap in a `try/except OSError` and ignore on platforms where directory fsync is not supported (e.g. Windows) — the swap is still atomic, only its on-disk persistence guarantee differs. A symlink at `manifest-latest` would be nicer on POSIX but a small pointer file is portable. 8. **Clean up** the temp ZIP (unless `--keep-zip`). ### Idempotency and safety - Never extract directly into the final directory — always into a sibling `*.tmp` that is renamed on success. A crash mid-extract leaves the previous good edition untouched. - Network calls go through `urllib.request` with a sane User-Agent (`amateurfunk-fetch/ (+contact)`) and a timeout. Single retry on transient errors (no exponential-backoff library needed for a once-a-quarter download). - No telemetry. No mutation outside `--out`. ### Testing focus The most valuable tests are behavioral, not network-bound. The real BNetzA fetch stays as an opt-in integration test (skipped by default); everything else runs against fixture ZIPs. - Slug generation from sample `metadata` (covers normal, missing ordinal, unusual edition strings). - ZIP path rejection: absolute paths, `..` segments, symlinks. - Uncompressed-size cap triggers cleanly. - Validation failures: missing JSON, multiple `fragenkatalog*.json`, missing `svgs/`, malformed JSON, missing required top-level keys, missing required metadata/question keys, malformed section nodes. - `missing_pictures` is populated but extraction still succeeds when a `picture_*` reference doesn't resolve (soft check). - Manifest is well-formed and contains the upstream attribution and `metadata` block verbatim. - **`Last-Modified` round-trip**: after a successful extraction, a rerun with the same `Last-Modified` header on the HEAD response skips the GET entirely. This is the actual contract idempotency hangs on now that the ZIP is deleted by default. - `--force` path: existing non-matching `DIR//` is replaced via the temp/bak rename dance, and a simulated crash between the two renames leaves a recoverable `.bak/`. - Atomic write of `manifest-latest.json`: a write that fails partway does not corrupt the existing pointer file. ### What we deliberately do NOT do in v1 - No PDF mirroring. - No question rendering (LaTeX, SVG). - No splitting by class / chapter on disk — the JSON tree already carries that structure and consumers can slice it themselves. - No diffing between editions. Useful, but a separate tool that reads two manifest dirs. - No mirror to S3 / a release artifact. Out of scope unless asked. --- ## 5. Resolved during review These were open in the first draft and have been settled: 1. **Edition slug format** — resolved: derive from `issued_on` + numeric edition ordinal, e.g. `2024-03-20-3-auflage`. Avoids parsing German month names; preserves day-precision; sortable. 2. **`manifest-latest.json` location** — at `DIR/` root (one stable pointer for consumers); per-edition `manifest.json` is the immutable record. 3. **Idempotency without a retained ZIP** — resolved: the manifest is the trusted record after extraction. `Last-Modified` match is sufficient to skip the download; `zip_sha256` is recorded for provenance only. 4. **README.txt preservation** — copy verbatim into the extracted tree alongside any derived attribution field in the manifest. 5. **Missing picture references** — soft-validate (record in `missing_pictures`, do not fail). Rationale: avoid bricking the tool on an upstream typo. ## 6. Open questions These do not block writing v1: 1. **PNG fallbacks** — the archive ships PNGs for ~a handful of figures alongside the SVGs. v1 just extracts everything as-is. A future renderer can prefer SVG and fall back to PNG transparently. 2. **Schema drift safety** — if a future edition adds or renames fields, the validator should warn but not fail. v1 fails closed on missing required top-level/metadata/question keys but tolerates extra keys. We can soften this further if BNetzA evolves the format. 3. **Packaging** — v1 stays a single-file script with `argparse` (`amateurfunk_fetch.py`). Considered and declined: a `pyproject.toml` package with a console-script entry point. For a tool that runs at most quarterly, the packaging ceremony does not pay for itself. Revisit if the scope grows (e.g. importable from another project, distributed on PyPI).