Initial design

2026-05-20 13:21:09 +02:00
commit 8c97c16885
2 changed files with 517 additions and 0 deletions
@@ -0,0 +1,76 @@
 # Amateurfunk — BNetzA Question Catalog Downloader
 A small Python tool that downloads the latest German amateur-radio exam
 question catalog ("Fragenkatalog") published by the Bundesnetzagentur
 (BNetzA) and extracts the structured JSON + SVG image assets for further
 use (study apps, flashcards, training tools, etc.).
 The full source-discovery notes, JSON schema, and design decisions live
 in `DESIGN.md`. This file is a short orientation for anyone (human or
 agent) opening the project.
 ## What the catalog is
 - Official German amateur-radio exam questions for classes **N, E, A**
  (German license tiers).
 - Published by the Bundesnetzagentur under the **DL-DE→BY-2.0** open
  data license (free reuse, attribution required).
 - Distributed as a single ZIP containing one JSON file with the full
  question tree, plus a `svgs/` folder with figures referenced by
  individual questions.
 - Current edition at time of writing: **3. Auflage, März 2024**
  (issued 2024-03-20, valid from 2024-06-24, ~1750 questions).
 ## Canonical source
 - ZIP (machine-readable): `https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/PruefungsfragenZIP.zip?__blob=publicationFile`
 - Landing page (short link): `https://www.bnetza.de/amateurfunk-fragenkatalog`
 - PDF (human-readable, not used by this tool): same path with
  `Pruefungsfragen.pdf` instead of `PruefungsfragenZIP.zip`.
 The ZIP URL is stable across editions — BNetzA replaces the file
 in-place. The `Last-Modified` HTTP header is reliable for change
 detection. The filename inside the ZIP (`fragenkatalog3b.json`) encodes
 the edition (`3b` = 3rd edition, revision b) and will change on new
 editions, so we discover it from the archive rather than hard-coding.
 ## Scope of the tool (initial)
 1. Download the ZIP from the canonical URL.
 2. Verify it is a valid ZIP and contains the expected JSON + SVG files.
 3. Extract to a target directory (default: `./data/<edition>/`).
 4. Emit a small `manifest.json` next to the data: source URL, fetched-at
   timestamp, `Last-Modified` from the server, JSON edition metadata,
   sha256 of the ZIP.
 5. Be idempotent — re-running without an upstream change is a no-op.
   The skip key is the HTTP `Last-Modified` header recorded on the
   previous manifest; the ZIP is deleted by default after extraction,
   so the recorded sha256 is provenance, not a re-verification target.
   See `DESIGN.md` for the full idempotency contract.
 Out of scope for v1 (kept for later): rendering LaTeX/SVG, building a
 study app, multi-edition diffing, mirroring the PDF.
 ## Repo conventions
 - Python 3.11+, standard library only where reasonable (`urllib`,
  `zipfile`, `hashlib`, `json`, `pathlib`). Add deps only if they pay
  for themselves — none expected for v1.
 - Single-purpose script: `amateurfunk_fetch.py` (or a tiny package if
  it grows). No framework, no CLI library beyond `argparse`.
 - Downloaded data is treated as a build artifact: kept under `data/`
  and gitignored.
 - License attribution string (required by DL-DE→BY-2.0) is preserved
  verbatim from the upstream `README.txt` whenever we redistribute the
  data.
 ## Working on this repo
 - Start from `DESIGN.md` — it has the JSON schema, the question/answer
  conventions (answer A is always correct, B/C/D are distractors), the
  LaTeX-in-questions caveat, and the SVG naming convention.
 - Do not invent new download URLs; the ones in `DESIGN.md` were
  verified against the live BNetzA site.
 - When BNetzA publishes a new edition, expect a new
  `fragenkatalog<N><rev>.json` filename inside the ZIP. The downloader
  must not hard-code the current name.
@@ -0,0 +1,441 @@
 # Design Notes — BNetzA Amateurfunk Question Catalog Downloader
 Status: **design only, no code yet.** This document captures the
 source-discovery work and the proposed shape of the tool. Everything
 below was verified against the live BNetzA site in May 2026.
 ---
 ## 1. Source of the latest questions
 The Bundesnetzagentur (BNetzA) is the authoritative publisher. They
 provide the exam catalog in two parallel formats from the same landing
 page:
 - **PDF** — human-readable, ~5 MB.
 - **ZIP** — machine-readable, ~3 MB, contains JSON + SVG figures. This
  is the format we use.
 ### Landing page
 - Short URL: `https://www.bnetza.de/amateurfunk-fragenkatalog`
  - HTTP 301 → `SharedDocs/Downloads/.../Fragenkatalog/KurzURLFragenkatalog.html`
  - The short URL points at an HTML page, not at the ZIP itself. We do
    not use it for fetching; it is useful only as a citation/attribution
    target.
 ### Direct download URLs (verified 2026-05-20)
 - **ZIP (what we fetch):**
  `https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/PruefungsfragenZIP.zip?__blob=publicationFile`
  - HTTP 200, `Content-Type: application/zip`, `Accept-Ranges: bytes`,
    serves a fresh `Last-Modified` header. Suitable for conditional
    fetch and idempotency checks.
 - **PDF (not used by this tool):**
  `https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/Pruefungsfragen.pdf?__blob=publicationFile`
 The `?__blob=publicationFile` query string is required — without it the
 CMS serves an HTML wrapper, not the binary.
 ### License
 `DL-DE→BY-2.0` (Datenlizenz Deutschland – Namensnennung – Version 2.0,
 see `www.govdata.de/dl-de/by-2-0`). Commercial and non-commercial reuse
 is permitted with attribution. The exact attribution string required is
 spelled out in the ZIP's `README.txt`; we copy it verbatim into any
 redistribution and into our `manifest.json`.
 ### Edition tracking
 BNetzA does not version the URL. The same `PruefungsfragenZIP.zip`
 path is updated in place when a new edition is published. To detect a
 new edition we rely on:
 1. The HTTP `Last-Modified` response header.
 2. The sha256 of the downloaded ZIP.
 3. The `metadata` block inside the JSON (`edition`, `issued_on`,
   `valid_from`).
 Current edition observed: `3. Auflage, März 2024` (issued 2024-03-20,
 valid from 2024-06-24).
 ---
 ## 2. ZIP contents (verified by extracting the live file)
 ```
 fragenkatalog3b.json         — single JSON file, full question tree (~1.3 MB)
 README.txt                   — license + schema documentation
 svgs/                        — 700+ figures (mostly SVG, a few PNG fallbacks)
  AB108_q.svg
  AB404_a.svg
  AB404_b.svg
  ...
  NG302_q.png                — a couple of questions ship a PNG alongside
  NG302_q.svg                  the SVG for display-problem fallbacks
 ```
 Total: 706 entries, 701 of them files. ~1750 questions across classes
 N, E, A (counts observed: N=571, E=463, A=716).
 The JSON filename encodes the edition: `fragenkatalog3b.json` =
 3rd edition, revision b. A future edition will rename this file
 (`fragenkatalog4a.json`, etc.), so the loader must discover it from
 the archive (glob `fragenkatalog*.json`), not hard-code the name.
 ### `README.txt` highlights
 - Confirms the two-part structure: JSON for catalog/questions, SVG for
  images.
 - Confirms the JSON schema (see section 3).
 - Notes that question text may contain **LaTeX** for formulas, intended
  to be rendered by something like KaTeX.
 ---
 ## 3. JSON schema
 Top-level object:
 ```jsonc
 {
  "metadata": {
    "edition":    "3. Auflage, März 2024",
    "issued_on":  "2024-03-20",
    "valid_from": "2024-06-24",
    "license":    "DL-DE->BY-2.0"
  },
  "sections": [ /* recursive section nodes */ ]
 }
 ```
 Each `section` node is either an **inner node** (contains nested
 `sections`) or a **leaf** (contains a `questions` list):
 ```jsonc
 // inner
 { "title": "Prüfungsfragen im Prüfungsteil: Technische Kenntnisse",
  "sections": [ ... ] }
 // leaf
 { "title": "Allgemeine mathematische Grundkenntnisse und Größen",
  "questions": [ /* question objects */ ] }
 ```
 Question object (fields per the upstream README):
 | field              | meaning                                                   |
 |--------------------|-----------------------------------------------------------|
 | `number`           | Catalog id, e.g. `NA103`, `AB404`. Also the SVG basename. |
 | `class`            | License class: `"1"` = N, `"2"` = E, `"3"` = A.           |
 | `question`         | Question text. May contain LaTeX.                         |
 | `answer_a`         | **The correct answer.** Always A.                         |
 | `answer_b`         | Distractor.                                               |
 | `answer_c`         | Distractor.                                               |
 | `answer_d`         | Distractor.                                               |
 | `picture_question` | Optional. Figure shown with the question stem.            |
 | `picture_a`..`_d`  | Optional. Per-choice figures.                             |
 The `picture_*` fields, when present, contain a filename (e.g.
 `AB404_q.svg`) that resolves into `svgs/`. Convention observed in the
 archive: `<number>_q.svg` for the question figure, `<number>_a.svg` ..
 `<number>_d.svg` for per-answer figures. A small number of entries also
 ship a `.png` next to the `.svg`.
 ### Important consumer-side conventions
 - **Answer A is always correct.** Anything that presents the questions
  to a learner must shuffle A/B/C/D before display, otherwise the
  exercise is trivial.
 - LaTeX in the question text and answers is unescaped — consumers
  render it (e.g. KaTeX/MathJax). The downloader does not transform it.
 - Classes are stored as string digits, not letter codes — map `"1"→N`,
  `"2"→E`, `"3"→A` for display.
 Sample question (from `fragenkatalog3b.json`):
 ```json
 {
  "number": "NA103",
  "class": "1",
  "question": "Laut Datenblatt wiegen 100 m eines bestimmten Drahtes 210 g. Ein vorliegendes Drahtstück desselben Materials wiegt 55 g. Wie lang ist das Drahtstück in etwa?",
  "answer_a": "26,2 m",
  "answer_b": "382 m",
  "answer_c": "115 m",
  "answer_d": "38,2 m"
 }
 ```
 ### Exam structure — how the catalog splits into exam parts and classes
 The catalog already encodes two orthogonal axes. The downloader does
 not slice the data on disk, but any consumer needs to understand both
 or they will compute the wrong candidate study pool.
 **Axis 1 — Exam part (Prüfungsteil).** The top-level `sections[]`
 array has exactly three entries, one per exam part:
 | Top-level `title`                                              | Question count | ID prefix      |
 |----------------------------------------------------------------|---------------:|----------------|
 | `Prüfungsfragen im Prüfungsteil: Technische Kenntnisse`        | 1374           | `N*`/`E*`/`A*` |
 | `Prüfungsfragen im Prüfungsteil: Betriebliche Kenntnisse`      |  172           | `B*`           |
 | `Prüfungsfragen im Prüfungsteil: Kenntnisse von Vorschriften`  |  204           | `V*`           |
 Consumers can split on the section title (canonical) or on the
 question `number` first letter (shorthand). The first-letter mapping
 is: `A`/`E`/`N` → Technische; `B` → Betriebliche; `V` → Vorschriften.
 Inside Technische, the first letter additionally mirrors the license
 class (see Axis 2).
 **Axis 2 — License class (`class` field on each question).**
 Values are `"1"`=N, `"2"`=E, `"3"`=A. Class distribution per exam
 part (counts verified against the live catalog, 3rd edition):
 | Exam part   | class 1 (N) | class 2 (E) | class 3 (A) |
 |-------------|------------:|------------:|------------:|
 | Technische  |        195  |        463  |        716  |
 | Betriebliche|        172  |          0  |          0  |
 | Vorschriften|        204  |          0  |          0  |
 Two things a consumer must know:
 1. **Operational + Regulations are class-1-only in the data, but
   apply to every candidate.** BNetzA treats these as a shared
   foundation. Do not filter them by `class`.
 2. **In Technische, the `class` field is a floor, not an equality
   marker.** German amateur-radio exam knowledge is cumulative: a
   class-E candidate is expected to know everything at class N and
   E; a class-A candidate knows class N + E + A. Treating `class`
   as equality underreports the E and A study pools.
 The candidate study pools work out as:
 - **N**: 195 (Tech class 1) + 172 (Betr) + 204 (Vor) = **571**
 - **E**: (195 + 463) Tech + 172 + 204                = **1034**
 - **A**: (195 + 463 + 716) Tech + 172 + 204          = **1750**
 The 1750 total exactly matches the full catalog, which confirms the
 floor interpretation: an A candidate's pool is the entire catalog.
 **Downloader scope.** v1 does not split data on disk — the JSON tree
 carries both axes already and consumers slice it themselves. This
 subsection exists so that future consumers (study app, flashcards,
 diff tool) implement the slicing correctly without re-deriving it
 from the data.
 ---
 ## 4. Tool design (v1)
 ### Goal
 Given no arguments, fetch the current BNetzA ZIP, extract it into a
 clean per-edition directory, and write a manifest. Re-running is a
 no-op when the upstream file has not changed.
 ### CLI shape
 ```
 amateurfunk-fetch [--out DIR] [--force] [--keep-zip]
 ```
 - `--out DIR` — output root (default `./data`). Each edition lands in
  `DIR/<edition-slug>/`, e.g. `data/2024-03-20-3-auflage/`.
 - `--force` — re-download and re-extract even if the existing manifest
  matches.
 - `--keep-zip` — keep the raw ZIP alongside the extracted tree (for
  archival). Default deletes it after successful extraction.
 Exit codes: `0` success (extracted or up-to-date), `1` network /
 validation error, `2` invalid local state (e.g. a partial previous run
 the tool can't reconcile without `--force`).
 ### Steps
 1. **HEAD** the ZIP URL to read `Last-Modified` (and `Content-Length`
   for a basic sanity range). If `DIR/manifest-latest.json` exists
   and its `http_last_modified` equals the current server value AND
   the target `DIR/<slug>/manifest.json` it points at is present and
   parseable, exit 0 unchanged. The manifest — not the raw ZIP — is
   the trusted record after a successful validated extraction; the
   ZIP sha256 stays in the manifest as provenance, not as something
   we re-verify on every run. (We delete the ZIP by default; there
   would be nothing to re-verify against.)
 2. **GET** the ZIP to a temp file in `DIR/.tmp/`. Stream to disk,
   compute sha256 on the fly. Enforce a compressed max size (e.g.
   50 MB) and, after open, a total uncompressed max size (e.g.
   200 MB) as a guardrail against zip-bomb-style upstream regressions.
 3. **Validate**, failing closed on structural problems:
   - `zipfile.is_zipfile()` is true.
   - All ZIP paths are relative and normalized: reject any entry
     whose name is absolute, contains a `..` segment, or whose
     `os.path.normpath`-result differs in a way that escapes the
     extraction root — defends against zip-slip.
   - **No symlink entries.** ZIPs created on Unix encode symlinks in
     the upper 16 bits of `ZipInfo.external_attr` (the POSIX mode
     field). Reject any entry where
     `(zi.external_attr >> 16) & 0o170000 == 0o120000` (S_IFLNK),
     in addition to the path checks above.
   - Sum of uncompressed sizes is below the configured cap.
   - Archive contains **exactly one** root-level `fragenkatalog*.json`.
   - Archive contains **more than 100 file entries** whose normalized
     path starts with `svgs/`. We do not require a standalone `svgs/`
     directory record — many ZIP producers omit directory entries
     and only emit file entries like `svgs/AB108_q.svg`.
   - The JSON parses and has top-level keys `metadata` and `sections`.
   - `metadata` carries the required keys `edition`, `issued_on`,
     `valid_from`, `license`. Extra keys are tolerated.
   - Every section node has either nested `sections` or a `questions`
     list (not both, not neither).
   - Every question has `number`, `class`, `question`, `answer_a`,
     `answer_b`, `answer_c`, `answer_d`.
   - For every `picture_*` reference in a question, the named file
     exists under `svgs/` in the archive. **Soft check:** missing
     references do not fail the extraction. They are collected into
     the manifest as `missing_pictures` so consumers can decide
     locally. Rationale: a single upstream typo should not brick the
     downloader for every consumer until BNetzA ships a fix.
 4. **Derive edition slug** from `metadata.issued_on` plus an edition
   ordinal parsed from `metadata.edition`, e.g.
   `"3. Auflage, März 2024"` + `2024-03-20` → `2024-03-20-3-auflage`.
   The ordinal is taken from the leading `\d+` in the edition string;
   if absent, fall back to `unknown-auflage`. Sortable, filesystem-
   safe, and independent of German month-name parsing. The full
   upstream `metadata.edition` is preserved verbatim in `manifest.json`.
 5. **Extract** into `DIR/<slug>.tmp/`. Replacement semantics:
   - If `DIR/<slug>/manifest.json` already exists and its recorded
     `zip_sha256` matches the freshly downloaded ZIP, skip extraction,
     update `manifest-latest.json` if needed, and exit 0.
   - If `DIR/<slug>/` exists but does not match and `--force` is not
     set, exit 2 with a clear message naming the offending directory.
   - With `--force`, after extraction completes into `DIR/<slug>.tmp/`,
     rename the existing `DIR/<slug>/` to `DIR/<slug>.bak/`, rename
     `DIR/<slug>.tmp/` into place, then remove `DIR/<slug>.bak/`. A
     crash between the two renames leaves a recoverable `.bak`
     directory.
   - **`.bak` collision policy:** if `DIR/<slug>.bak/` already exists
     before the forced replacement starts, exit 2 with a clear error
     naming the stale `.bak/` path. A leftover `.bak/` is evidence
     that a previous run crashed mid-rename; deciding whether to keep
     or delete it is the operator's call, not ours. The same applies
     to a stale `DIR/<slug>.tmp/`: refuse to overwrite, surface the
     path. Both checks happen *before* any rename — no destructive
     action without a clean predecessor state.
   Always copy the upstream `README.txt` verbatim into the extracted
   tree. The manifest also records the attribution string for
   convenience, but the file is the source of truth — lossless
   preservation beats parser-derived fields.
 6. **Write `DIR/<slug>/manifest.json`** with:
   - `source_url`
   - `fetched_at` (ISO 8601 UTC)
   - `http_last_modified` (verbatim from the server)
   - `zip_sha256`
   - `zip_size`
   - `json_filename` (the actual `fragenkatalog*.json` we found)
   - the full upstream `metadata` block
   - the verbatim attribution string from `README.txt`
   - `missing_pictures` (array of `{question_number, field, file}`,
     empty when the archive is clean)
 7. **Atomically update `DIR/manifest-latest.json`** to point at the
   current slug. Precise sequence:
   1. Write the new content to a sibling `manifest-latest.json.tmp`
      in the same directory as the target.
   2. `os.fsync(tmp_fd)` to flush the tmp file's contents to disk.
   3. `os.replace(tmp_path, final_path)` to atomically swap.
   4. On POSIX, open the containing directory and `os.fsync` its
      file descriptor so the rename itself is durable. Wrap in a
      `try/except OSError` and ignore on platforms where directory
      fsync is not supported (e.g. Windows) — the swap is still
      atomic, only its on-disk persistence guarantee differs.
   A symlink at `manifest-latest` would be nicer on POSIX but a
   small pointer file is portable.
 8. **Clean up** the temp ZIP (unless `--keep-zip`).
 ### Idempotency and safety
 - Never extract directly into the final directory — always into a
  sibling `*.tmp` that is renamed on success. A crash mid-extract leaves
  the previous good edition untouched.
 - Network calls go through `urllib.request` with a sane User-Agent
  (`amateurfunk-fetch/<version> (+contact)`) and a timeout. Single
  retry on transient errors (no exponential-backoff library needed for
  a once-a-quarter download).
 - No telemetry. No mutation outside `--out`.
 ### Testing focus
 The most valuable tests are behavioral, not network-bound. The real
 BNetzA fetch stays as an opt-in integration test (skipped by default);
 everything else runs against fixture ZIPs.
 - Slug generation from sample `metadata` (covers normal, missing
  ordinal, unusual edition strings).
 - ZIP path rejection: absolute paths, `..` segments, symlinks.
 - Uncompressed-size cap triggers cleanly.
 - Validation failures: missing JSON, multiple `fragenkatalog*.json`,
  missing `svgs/`, malformed JSON, missing required top-level keys,
  missing required metadata/question keys, malformed section nodes.
 - `missing_pictures` is populated but extraction still succeeds when
  a `picture_*` reference doesn't resolve (soft check).
 - Manifest is well-formed and contains the upstream attribution and
  `metadata` block verbatim.
 - **`Last-Modified` round-trip**: after a successful extraction, a
  rerun with the same `Last-Modified` header on the HEAD response
  skips the GET entirely. This is the actual contract idempotency
  hangs on now that the ZIP is deleted by default.
 - `--force` path: existing non-matching `DIR/<slug>/` is replaced via
  the temp/bak rename dance, and a simulated crash between the two
  renames leaves a recoverable `.bak/`.
 - Atomic write of `manifest-latest.json`: a write that fails partway
  does not corrupt the existing pointer file.
 ### What we deliberately do NOT do in v1
 - No PDF mirroring.
 - No question rendering (LaTeX, SVG).
 - No splitting by class / chapter on disk — the JSON tree already
  carries that structure and consumers can slice it themselves.
 - No diffing between editions. Useful, but a separate tool that reads
  two manifest dirs.
 - No mirror to S3 / a release artifact. Out of scope unless asked.
 ---
 ## 5. Resolved during review
 These were open in the first draft and have been settled:
 1. **Edition slug format** — resolved: derive from `issued_on` +
   numeric edition ordinal, e.g. `2024-03-20-3-auflage`. Avoids
   parsing German month names; preserves day-precision; sortable.
 2. **`manifest-latest.json` location** — at `DIR/` root (one stable
   pointer for consumers); per-edition `manifest.json` is the
   immutable record.
 3. **Idempotency without a retained ZIP** — resolved: the manifest is
   the trusted record after extraction. `Last-Modified` match is
   sufficient to skip the download; `zip_sha256` is recorded for
   provenance only.
 4. **README.txt preservation** — copy verbatim into the extracted
   tree alongside any derived attribution field in the manifest.
 5. **Missing picture references** — soft-validate (record in
   `missing_pictures`, do not fail). Rationale: avoid bricking the
   tool on an upstream typo.
 ## 6. Open questions
 These do not block writing v1:
 1. **PNG fallbacks** — the archive ships PNGs for ~a handful of
   figures alongside the SVGs. v1 just extracts everything as-is. A
   future renderer can prefer SVG and fall back to PNG transparently.
 2. **Schema drift safety** — if a future edition adds or renames
   fields, the validator should warn but not fail. v1 fails closed
   on missing required top-level/metadata/question keys but tolerates
   extra keys. We can soften this further if BNetzA evolves the
   format.
 3. **Packaging** — v1 stays a single-file script with `argparse`
   (`amateurfunk_fetch.py`). Considered and declined: a
   `pyproject.toml` package with a console-script entry point. For a
   tool that runs at most quarterly, the packaging ceremony does not
   pay for itself. Revisit if the scope grows (e.g. importable from
   another project, distributed on PyPI).