From 8c97c168855e21c71e8061ddf07f89c454ae9715 Mon Sep 17 00:00:00 2001 From: Renat Nurgaliyev Date: Wed, 20 May 2026 13:21:09 +0200 Subject: [PATCH] Initial design --- CLAUDE.md | 76 ++++++++++ DESIGN.md | 441 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 517 insertions(+) create mode 100644 CLAUDE.md create mode 100644 DESIGN.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..4731f9e --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,76 @@ +# Amateurfunk — BNetzA Question Catalog Downloader + +A small Python tool that downloads the latest German amateur-radio exam +question catalog ("Fragenkatalog") published by the Bundesnetzagentur +(BNetzA) and extracts the structured JSON + SVG image assets for further +use (study apps, flashcards, training tools, etc.). + +The full source-discovery notes, JSON schema, and design decisions live +in `DESIGN.md`. This file is a short orientation for anyone (human or +agent) opening the project. + +## What the catalog is + +- Official German amateur-radio exam questions for classes **N, E, A** + (German license tiers). +- Published by the Bundesnetzagentur under the **DL-DE→BY-2.0** open + data license (free reuse, attribution required). +- Distributed as a single ZIP containing one JSON file with the full + question tree, plus a `svgs/` folder with figures referenced by + individual questions. +- Current edition at time of writing: **3. Auflage, März 2024** + (issued 2024-03-20, valid from 2024-06-24, ~1750 questions). + +## Canonical source + +- ZIP (machine-readable): `https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/PruefungsfragenZIP.zip?__blob=publicationFile` +- Landing page (short link): `https://www.bnetza.de/amateurfunk-fragenkatalog` +- PDF (human-readable, not used by this tool): same path with + `Pruefungsfragen.pdf` instead of `PruefungsfragenZIP.zip`. + +The ZIP URL is stable across editions — BNetzA replaces the file +in-place. The `Last-Modified` HTTP header is reliable for change +detection. The filename inside the ZIP (`fragenkatalog3b.json`) encodes +the edition (`3b` = 3rd edition, revision b) and will change on new +editions, so we discover it from the archive rather than hard-coding. + +## Scope of the tool (initial) + +1. Download the ZIP from the canonical URL. +2. Verify it is a valid ZIP and contains the expected JSON + SVG files. +3. Extract to a target directory (default: `./data//`). +4. Emit a small `manifest.json` next to the data: source URL, fetched-at + timestamp, `Last-Modified` from the server, JSON edition metadata, + sha256 of the ZIP. +5. Be idempotent — re-running without an upstream change is a no-op. + The skip key is the HTTP `Last-Modified` header recorded on the + previous manifest; the ZIP is deleted by default after extraction, + so the recorded sha256 is provenance, not a re-verification target. + See `DESIGN.md` for the full idempotency contract. + +Out of scope for v1 (kept for later): rendering LaTeX/SVG, building a +study app, multi-edition diffing, mirroring the PDF. + +## Repo conventions + +- Python 3.11+, standard library only where reasonable (`urllib`, + `zipfile`, `hashlib`, `json`, `pathlib`). Add deps only if they pay + for themselves — none expected for v1. +- Single-purpose script: `amateurfunk_fetch.py` (or a tiny package if + it grows). No framework, no CLI library beyond `argparse`. +- Downloaded data is treated as a build artifact: kept under `data/` + and gitignored. +- License attribution string (required by DL-DE→BY-2.0) is preserved + verbatim from the upstream `README.txt` whenever we redistribute the + data. + +## Working on this repo + +- Start from `DESIGN.md` — it has the JSON schema, the question/answer + conventions (answer A is always correct, B/C/D are distractors), the + LaTeX-in-questions caveat, and the SVG naming convention. +- Do not invent new download URLs; the ones in `DESIGN.md` were + verified against the live BNetzA site. +- When BNetzA publishes a new edition, expect a new + `fragenkatalog.json` filename inside the ZIP. The downloader + must not hard-code the current name. diff --git a/DESIGN.md b/DESIGN.md new file mode 100644 index 0000000..f851503 --- /dev/null +++ b/DESIGN.md @@ -0,0 +1,441 @@ +# Design Notes — BNetzA Amateurfunk Question Catalog Downloader + +Status: **design only, no code yet.** This document captures the +source-discovery work and the proposed shape of the tool. Everything +below was verified against the live BNetzA site in May 2026. + +--- + +## 1. Source of the latest questions + +The Bundesnetzagentur (BNetzA) is the authoritative publisher. They +provide the exam catalog in two parallel formats from the same landing +page: + +- **PDF** — human-readable, ~5 MB. +- **ZIP** — machine-readable, ~3 MB, contains JSON + SVG figures. This + is the format we use. + +### Landing page + +- Short URL: `https://www.bnetza.de/amateurfunk-fragenkatalog` + - HTTP 301 → `SharedDocs/Downloads/.../Fragenkatalog/KurzURLFragenkatalog.html` + - The short URL points at an HTML page, not at the ZIP itself. We do + not use it for fetching; it is useful only as a citation/attribution + target. + +### Direct download URLs (verified 2026-05-20) + +- **ZIP (what we fetch):** + `https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/PruefungsfragenZIP.zip?__blob=publicationFile` + - HTTP 200, `Content-Type: application/zip`, `Accept-Ranges: bytes`, + serves a fresh `Last-Modified` header. Suitable for conditional + fetch and idempotency checks. +- **PDF (not used by this tool):** + `https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/Pruefungsfragen.pdf?__blob=publicationFile` + +The `?__blob=publicationFile` query string is required — without it the +CMS serves an HTML wrapper, not the binary. + +### License + +`DL-DE→BY-2.0` (Datenlizenz Deutschland – Namensnennung – Version 2.0, +see `www.govdata.de/dl-de/by-2-0`). Commercial and non-commercial reuse +is permitted with attribution. The exact attribution string required is +spelled out in the ZIP's `README.txt`; we copy it verbatim into any +redistribution and into our `manifest.json`. + +### Edition tracking + +BNetzA does not version the URL. The same `PruefungsfragenZIP.zip` +path is updated in place when a new edition is published. To detect a +new edition we rely on: + +1. The HTTP `Last-Modified` response header. +2. The sha256 of the downloaded ZIP. +3. The `metadata` block inside the JSON (`edition`, `issued_on`, + `valid_from`). + +Current edition observed: `3. Auflage, März 2024` (issued 2024-03-20, +valid from 2024-06-24). + +--- + +## 2. ZIP contents (verified by extracting the live file) + +``` +fragenkatalog3b.json — single JSON file, full question tree (~1.3 MB) +README.txt — license + schema documentation +svgs/ — 700+ figures (mostly SVG, a few PNG fallbacks) + AB108_q.svg + AB404_a.svg + AB404_b.svg + ... + NG302_q.png — a couple of questions ship a PNG alongside + NG302_q.svg the SVG for display-problem fallbacks +``` + +Total: 706 entries, 701 of them files. ~1750 questions across classes +N, E, A (counts observed: N=571, E=463, A=716). + +The JSON filename encodes the edition: `fragenkatalog3b.json` = +3rd edition, revision b. A future edition will rename this file +(`fragenkatalog4a.json`, etc.), so the loader must discover it from +the archive (glob `fragenkatalog*.json`), not hard-code the name. + +### `README.txt` highlights + +- Confirms the two-part structure: JSON for catalog/questions, SVG for + images. +- Confirms the JSON schema (see section 3). +- Notes that question text may contain **LaTeX** for formulas, intended + to be rendered by something like KaTeX. + +--- + +## 3. JSON schema + +Top-level object: + +```jsonc +{ + "metadata": { + "edition": "3. Auflage, März 2024", + "issued_on": "2024-03-20", + "valid_from": "2024-06-24", + "license": "DL-DE->BY-2.0" + }, + "sections": [ /* recursive section nodes */ ] +} +``` + +Each `section` node is either an **inner node** (contains nested +`sections`) or a **leaf** (contains a `questions` list): + +```jsonc +// inner +{ "title": "Prüfungsfragen im Prüfungsteil: Technische Kenntnisse", + "sections": [ ... ] } + +// leaf +{ "title": "Allgemeine mathematische Grundkenntnisse und Größen", + "questions": [ /* question objects */ ] } +``` + +Question object (fields per the upstream README): + +| field | meaning | +|--------------------|-----------------------------------------------------------| +| `number` | Catalog id, e.g. `NA103`, `AB404`. Also the SVG basename. | +| `class` | License class: `"1"` = N, `"2"` = E, `"3"` = A. | +| `question` | Question text. May contain LaTeX. | +| `answer_a` | **The correct answer.** Always A. | +| `answer_b` | Distractor. | +| `answer_c` | Distractor. | +| `answer_d` | Distractor. | +| `picture_question` | Optional. Figure shown with the question stem. | +| `picture_a`..`_d` | Optional. Per-choice figures. | + +The `picture_*` fields, when present, contain a filename (e.g. +`AB404_q.svg`) that resolves into `svgs/`. Convention observed in the +archive: `_q.svg` for the question figure, `_a.svg` .. +`_d.svg` for per-answer figures. A small number of entries also +ship a `.png` next to the `.svg`. + +### Important consumer-side conventions + +- **Answer A is always correct.** Anything that presents the questions + to a learner must shuffle A/B/C/D before display, otherwise the + exercise is trivial. +- LaTeX in the question text and answers is unescaped — consumers + render it (e.g. KaTeX/MathJax). The downloader does not transform it. +- Classes are stored as string digits, not letter codes — map `"1"→N`, + `"2"→E`, `"3"→A` for display. + +Sample question (from `fragenkatalog3b.json`): + +```json +{ + "number": "NA103", + "class": "1", + "question": "Laut Datenblatt wiegen 100 m eines bestimmten Drahtes 210 g. Ein vorliegendes Drahtstück desselben Materials wiegt 55 g. Wie lang ist das Drahtstück in etwa?", + "answer_a": "26,2 m", + "answer_b": "382 m", + "answer_c": "115 m", + "answer_d": "38,2 m" +} +``` + +### Exam structure — how the catalog splits into exam parts and classes + +The catalog already encodes two orthogonal axes. The downloader does +not slice the data on disk, but any consumer needs to understand both +or they will compute the wrong candidate study pool. + +**Axis 1 — Exam part (Prüfungsteil).** The top-level `sections[]` +array has exactly three entries, one per exam part: + +| Top-level `title` | Question count | ID prefix | +|----------------------------------------------------------------|---------------:|----------------| +| `Prüfungsfragen im Prüfungsteil: Technische Kenntnisse` | 1374 | `N*`/`E*`/`A*` | +| `Prüfungsfragen im Prüfungsteil: Betriebliche Kenntnisse` | 172 | `B*` | +| `Prüfungsfragen im Prüfungsteil: Kenntnisse von Vorschriften` | 204 | `V*` | + +Consumers can split on the section title (canonical) or on the +question `number` first letter (shorthand). The first-letter mapping +is: `A`/`E`/`N` → Technische; `B` → Betriebliche; `V` → Vorschriften. +Inside Technische, the first letter additionally mirrors the license +class (see Axis 2). + +**Axis 2 — License class (`class` field on each question).** +Values are `"1"`=N, `"2"`=E, `"3"`=A. Class distribution per exam +part (counts verified against the live catalog, 3rd edition): + +| Exam part | class 1 (N) | class 2 (E) | class 3 (A) | +|-------------|------------:|------------:|------------:| +| Technische | 195 | 463 | 716 | +| Betriebliche| 172 | 0 | 0 | +| Vorschriften| 204 | 0 | 0 | + +Two things a consumer must know: + +1. **Operational + Regulations are class-1-only in the data, but + apply to every candidate.** BNetzA treats these as a shared + foundation. Do not filter them by `class`. +2. **In Technische, the `class` field is a floor, not an equality + marker.** German amateur-radio exam knowledge is cumulative: a + class-E candidate is expected to know everything at class N and + E; a class-A candidate knows class N + E + A. Treating `class` + as equality underreports the E and A study pools. + +The candidate study pools work out as: + +- **N**: 195 (Tech class 1) + 172 (Betr) + 204 (Vor) = **571** +- **E**: (195 + 463) Tech + 172 + 204 = **1034** +- **A**: (195 + 463 + 716) Tech + 172 + 204 = **1750** + +The 1750 total exactly matches the full catalog, which confirms the +floor interpretation: an A candidate's pool is the entire catalog. + +**Downloader scope.** v1 does not split data on disk — the JSON tree +carries both axes already and consumers slice it themselves. This +subsection exists so that future consumers (study app, flashcards, +diff tool) implement the slicing correctly without re-deriving it +from the data. + +--- + +## 4. Tool design (v1) + +### Goal + +Given no arguments, fetch the current BNetzA ZIP, extract it into a +clean per-edition directory, and write a manifest. Re-running is a +no-op when the upstream file has not changed. + +### CLI shape + +``` +amateurfunk-fetch [--out DIR] [--force] [--keep-zip] +``` + +- `--out DIR` — output root (default `./data`). Each edition lands in + `DIR//`, e.g. `data/2024-03-20-3-auflage/`. +- `--force` — re-download and re-extract even if the existing manifest + matches. +- `--keep-zip` — keep the raw ZIP alongside the extracted tree (for + archival). Default deletes it after successful extraction. + +Exit codes: `0` success (extracted or up-to-date), `1` network / +validation error, `2` invalid local state (e.g. a partial previous run +the tool can't reconcile without `--force`). + +### Steps + +1. **HEAD** the ZIP URL to read `Last-Modified` (and `Content-Length` + for a basic sanity range). If `DIR/manifest-latest.json` exists + and its `http_last_modified` equals the current server value AND + the target `DIR//manifest.json` it points at is present and + parseable, exit 0 unchanged. The manifest — not the raw ZIP — is + the trusted record after a successful validated extraction; the + ZIP sha256 stays in the manifest as provenance, not as something + we re-verify on every run. (We delete the ZIP by default; there + would be nothing to re-verify against.) +2. **GET** the ZIP to a temp file in `DIR/.tmp/`. Stream to disk, + compute sha256 on the fly. Enforce a compressed max size (e.g. + 50 MB) and, after open, a total uncompressed max size (e.g. + 200 MB) as a guardrail against zip-bomb-style upstream regressions. +3. **Validate**, failing closed on structural problems: + - `zipfile.is_zipfile()` is true. + - All ZIP paths are relative and normalized: reject any entry + whose name is absolute, contains a `..` segment, or whose + `os.path.normpath`-result differs in a way that escapes the + extraction root — defends against zip-slip. + - **No symlink entries.** ZIPs created on Unix encode symlinks in + the upper 16 bits of `ZipInfo.external_attr` (the POSIX mode + field). Reject any entry where + `(zi.external_attr >> 16) & 0o170000 == 0o120000` (S_IFLNK), + in addition to the path checks above. + - Sum of uncompressed sizes is below the configured cap. + - Archive contains **exactly one** root-level `fragenkatalog*.json`. + - Archive contains **more than 100 file entries** whose normalized + path starts with `svgs/`. We do not require a standalone `svgs/` + directory record — many ZIP producers omit directory entries + and only emit file entries like `svgs/AB108_q.svg`. + - The JSON parses and has top-level keys `metadata` and `sections`. + - `metadata` carries the required keys `edition`, `issued_on`, + `valid_from`, `license`. Extra keys are tolerated. + - Every section node has either nested `sections` or a `questions` + list (not both, not neither). + - Every question has `number`, `class`, `question`, `answer_a`, + `answer_b`, `answer_c`, `answer_d`. + - For every `picture_*` reference in a question, the named file + exists under `svgs/` in the archive. **Soft check:** missing + references do not fail the extraction. They are collected into + the manifest as `missing_pictures` so consumers can decide + locally. Rationale: a single upstream typo should not brick the + downloader for every consumer until BNetzA ships a fix. +4. **Derive edition slug** from `metadata.issued_on` plus an edition + ordinal parsed from `metadata.edition`, e.g. + `"3. Auflage, März 2024"` + `2024-03-20` → `2024-03-20-3-auflage`. + The ordinal is taken from the leading `\d+` in the edition string; + if absent, fall back to `unknown-auflage`. Sortable, filesystem- + safe, and independent of German month-name parsing. The full + upstream `metadata.edition` is preserved verbatim in `manifest.json`. +5. **Extract** into `DIR/.tmp/`. Replacement semantics: + - If `DIR//manifest.json` already exists and its recorded + `zip_sha256` matches the freshly downloaded ZIP, skip extraction, + update `manifest-latest.json` if needed, and exit 0. + - If `DIR//` exists but does not match and `--force` is not + set, exit 2 with a clear message naming the offending directory. + - With `--force`, after extraction completes into `DIR/.tmp/`, + rename the existing `DIR//` to `DIR/.bak/`, rename + `DIR/.tmp/` into place, then remove `DIR/.bak/`. A + crash between the two renames leaves a recoverable `.bak` + directory. + - **`.bak` collision policy:** if `DIR/.bak/` already exists + before the forced replacement starts, exit 2 with a clear error + naming the stale `.bak/` path. A leftover `.bak/` is evidence + that a previous run crashed mid-rename; deciding whether to keep + or delete it is the operator's call, not ours. The same applies + to a stale `DIR/.tmp/`: refuse to overwrite, surface the + path. Both checks happen *before* any rename — no destructive + action without a clean predecessor state. + Always copy the upstream `README.txt` verbatim into the extracted + tree. The manifest also records the attribution string for + convenience, but the file is the source of truth — lossless + preservation beats parser-derived fields. +6. **Write `DIR//manifest.json`** with: + - `source_url` + - `fetched_at` (ISO 8601 UTC) + - `http_last_modified` (verbatim from the server) + - `zip_sha256` + - `zip_size` + - `json_filename` (the actual `fragenkatalog*.json` we found) + - the full upstream `metadata` block + - the verbatim attribution string from `README.txt` + - `missing_pictures` (array of `{question_number, field, file}`, + empty when the archive is clean) +7. **Atomically update `DIR/manifest-latest.json`** to point at the + current slug. Precise sequence: + 1. Write the new content to a sibling `manifest-latest.json.tmp` + in the same directory as the target. + 2. `os.fsync(tmp_fd)` to flush the tmp file's contents to disk. + 3. `os.replace(tmp_path, final_path)` to atomically swap. + 4. On POSIX, open the containing directory and `os.fsync` its + file descriptor so the rename itself is durable. Wrap in a + `try/except OSError` and ignore on platforms where directory + fsync is not supported (e.g. Windows) — the swap is still + atomic, only its on-disk persistence guarantee differs. + A symlink at `manifest-latest` would be nicer on POSIX but a + small pointer file is portable. +8. **Clean up** the temp ZIP (unless `--keep-zip`). + +### Idempotency and safety + +- Never extract directly into the final directory — always into a + sibling `*.tmp` that is renamed on success. A crash mid-extract leaves + the previous good edition untouched. +- Network calls go through `urllib.request` with a sane User-Agent + (`amateurfunk-fetch/ (+contact)`) and a timeout. Single + retry on transient errors (no exponential-backoff library needed for + a once-a-quarter download). +- No telemetry. No mutation outside `--out`. + +### Testing focus + +The most valuable tests are behavioral, not network-bound. The real +BNetzA fetch stays as an opt-in integration test (skipped by default); +everything else runs against fixture ZIPs. + +- Slug generation from sample `metadata` (covers normal, missing + ordinal, unusual edition strings). +- ZIP path rejection: absolute paths, `..` segments, symlinks. +- Uncompressed-size cap triggers cleanly. +- Validation failures: missing JSON, multiple `fragenkatalog*.json`, + missing `svgs/`, malformed JSON, missing required top-level keys, + missing required metadata/question keys, malformed section nodes. +- `missing_pictures` is populated but extraction still succeeds when + a `picture_*` reference doesn't resolve (soft check). +- Manifest is well-formed and contains the upstream attribution and + `metadata` block verbatim. +- **`Last-Modified` round-trip**: after a successful extraction, a + rerun with the same `Last-Modified` header on the HEAD response + skips the GET entirely. This is the actual contract idempotency + hangs on now that the ZIP is deleted by default. +- `--force` path: existing non-matching `DIR//` is replaced via + the temp/bak rename dance, and a simulated crash between the two + renames leaves a recoverable `.bak/`. +- Atomic write of `manifest-latest.json`: a write that fails partway + does not corrupt the existing pointer file. + +### What we deliberately do NOT do in v1 + +- No PDF mirroring. +- No question rendering (LaTeX, SVG). +- No splitting by class / chapter on disk — the JSON tree already + carries that structure and consumers can slice it themselves. +- No diffing between editions. Useful, but a separate tool that reads + two manifest dirs. +- No mirror to S3 / a release artifact. Out of scope unless asked. + +--- + +## 5. Resolved during review + +These were open in the first draft and have been settled: + +1. **Edition slug format** — resolved: derive from `issued_on` + + numeric edition ordinal, e.g. `2024-03-20-3-auflage`. Avoids + parsing German month names; preserves day-precision; sortable. +2. **`manifest-latest.json` location** — at `DIR/` root (one stable + pointer for consumers); per-edition `manifest.json` is the + immutable record. +3. **Idempotency without a retained ZIP** — resolved: the manifest is + the trusted record after extraction. `Last-Modified` match is + sufficient to skip the download; `zip_sha256` is recorded for + provenance only. +4. **README.txt preservation** — copy verbatim into the extracted + tree alongside any derived attribution field in the manifest. +5. **Missing picture references** — soft-validate (record in + `missing_pictures`, do not fail). Rationale: avoid bricking the + tool on an upstream typo. + +## 6. Open questions + +These do not block writing v1: + +1. **PNG fallbacks** — the archive ships PNGs for ~a handful of + figures alongside the SVGs. v1 just extracts everything as-is. A + future renderer can prefer SVG and fall back to PNG transparently. +2. **Schema drift safety** — if a future edition adds or renames + fields, the validator should warn but not fail. v1 fails closed + on missing required top-level/metadata/question keys but tolerates + extra keys. We can soften this further if BNetzA evolves the + format. +3. **Packaging** — v1 stays a single-file script with `argparse` + (`amateurfunk_fetch.py`). Considered and declined: a + `pyproject.toml` package with a console-script entry point. For a + tool that runs at most quarterly, the packaging ceremony does not + pay for itself. Revisit if the scope grows (e.g. importable from + another project, distributed on PyPI).