imple/Amateurfunk-Anki
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
8c97c168855e21c71e8061ddf07f89c454ae9715
Amateurfunk-Anki/DESIGN.md
T
imple 8c97c16885 Initial design
2026-05-20 13:21:09 +02:00

20 KiB
Raw Blame History

Design Notes — BNetzA Amateurfunk Question Catalog Downloader

Status: design only, no code yet. This document captures the source-discovery work and the proposed shape of the tool. Everything below was verified against the live BNetzA site in May 2026.

1. Source of the latest questions

The Bundesnetzagentur (BNetzA) is the authoritative publisher. They provide the exam catalog in two parallel formats from the same landing page:

  • PDF — human-readable, ~5 MB.
  • ZIP — machine-readable, ~3 MB, contains JSON + SVG figures. This is the format we use.

Landing page

  • Short URL: https://www.bnetza.de/amateurfunk-fragenkatalog
    • HTTP 301 → SharedDocs/Downloads/.../Fragenkatalog/KurzURLFragenkatalog.html
    • The short URL points at an HTML page, not at the ZIP itself. We do not use it for fetching; it is useful only as a citation/attribution target.

Direct download URLs (verified 2026-05-20)

  • ZIP (what we fetch): https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/PruefungsfragenZIP.zip?__blob=publicationFile
    • HTTP 200, Content-Type: application/zip, Accept-Ranges: bytes, serves a fresh Last-Modified header. Suitable for conditional fetch and idempotency checks.
  • PDF (not used by this tool): https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/Pruefungsfragen.pdf?__blob=publicationFile

The ?__blob=publicationFile query string is required — without it the CMS serves an HTML wrapper, not the binary.

License

DL-DE→BY-2.0 (Datenlizenz Deutschland – Namensnennung – Version 2.0, see www.govdata.de/dl-de/by-2-0). Commercial and non-commercial reuse is permitted with attribution. The exact attribution string required is spelled out in the ZIP's README.txt; we copy it verbatim into any redistribution and into our manifest.json.

Edition tracking

BNetzA does not version the URL. The same PruefungsfragenZIP.zip path is updated in place when a new edition is published. To detect a new edition we rely on:

  1. The HTTP Last-Modified response header.
  2. The sha256 of the downloaded ZIP.
  3. The metadata block inside the JSON (edition, issued_on, valid_from).

Current edition observed: 3. Auflage, März 2024 (issued 2024-03-20, valid from 2024-06-24).

2. ZIP contents (verified by extracting the live file)

fragenkatalog3b.json         — single JSON file, full question tree (~1.3 MB)
README.txt                   — license + schema documentation
svgs/                        — 700+ figures (mostly SVG, a few PNG fallbacks)
  AB108_q.svg
  AB404_a.svg
  AB404_b.svg
  ...
  NG302_q.png                — a couple of questions ship a PNG alongside
  NG302_q.svg                  the SVG for display-problem fallbacks

Total: 706 entries, 701 of them files. ~1750 questions across classes N, E, A (counts observed: N=571, E=463, A=716).

The JSON filename encodes the edition: fragenkatalog3b.json = 3rd edition, revision b. A future edition will rename this file (fragenkatalog4a.json, etc.), so the loader must discover it from the archive (glob fragenkatalog*.json), not hard-code the name.

README.txt highlights

  • Confirms the two-part structure: JSON for catalog/questions, SVG for images.
  • Confirms the JSON schema (see section 3).
  • Notes that question text may contain LaTeX for formulas, intended to be rendered by something like KaTeX.

3. JSON schema

Top-level object:

{
  "metadata": {
    "edition":    "3. Auflage, März 2024",
    "issued_on":  "2024-03-20",
    "valid_from": "2024-06-24",
    "license":    "DL-DE->BY-2.0"
  },
  "sections": [ /* recursive section nodes */ ]
}

Each section node is either an inner node (contains nested sections) or a leaf (contains a questions list):

// inner
{ "title": "Prüfungsfragen im Prüfungsteil: Technische Kenntnisse",
  "sections": [ ... ] }

// leaf
{ "title": "Allgemeine mathematische Grundkenntnisse und Größen",
  "questions": [ /* question objects */ ] }

Question object (fields per the upstream README):

field meaning
number Catalog id, e.g. NA103, AB404. Also the SVG basename.
class License class: "1" = N, "2" = E, "3" = A.
question Question text. May contain LaTeX.
answer_a The correct answer. Always A.
answer_b Distractor.
answer_c Distractor.
answer_d Distractor.
picture_question Optional. Figure shown with the question stem.
picture_a.._d Optional. Per-choice figures.

The picture_* fields, when present, contain a filename (e.g. AB404_q.svg) that resolves into svgs/. Convention observed in the archive: <number>_q.svg for the question figure, <number>_a.svg .. <number>_d.svg for per-answer figures. A small number of entries also ship a .png next to the .svg.

Important consumer-side conventions

  • Answer A is always correct. Anything that presents the questions to a learner must shuffle A/B/C/D before display, otherwise the exercise is trivial.
  • LaTeX in the question text and answers is unescaped — consumers render it (e.g. KaTeX/MathJax). The downloader does not transform it.
  • Classes are stored as string digits, not letter codes — map "1"→N, "2"→E, "3"→A for display.

Sample question (from fragenkatalog3b.json):

{
  "number": "NA103",
  "class": "1",
  "question": "Laut Datenblatt wiegen 100 m eines bestimmten Drahtes 210 g. Ein vorliegendes Drahtstück desselben Materials wiegt 55 g. Wie lang ist das Drahtstück in etwa?",
  "answer_a": "26,2 m",
  "answer_b": "382 m",
  "answer_c": "115 m",
  "answer_d": "38,2 m"
}

Exam structure — how the catalog splits into exam parts and classes

The catalog already encodes two orthogonal axes. The downloader does not slice the data on disk, but any consumer needs to understand both or they will compute the wrong candidate study pool.

Axis 1 — Exam part (Prüfungsteil). The top-level sections[] array has exactly three entries, one per exam part:

Top-level title Question count ID prefix
Prüfungsfragen im Prüfungsteil: Technische Kenntnisse 1374 N*/E*/A*
Prüfungsfragen im Prüfungsteil: Betriebliche Kenntnisse 172 B*
Prüfungsfragen im Prüfungsteil: Kenntnisse von Vorschriften 204 V*

Consumers can split on the section title (canonical) or on the question number first letter (shorthand). The first-letter mapping is: A/E/N → Technische; B → Betriebliche; V → Vorschriften. Inside Technische, the first letter additionally mirrors the license class (see Axis 2).

Axis 2 — License class (class field on each question). Values are "1"=N, "2"=E, "3"=A. Class distribution per exam part (counts verified against the live catalog, 3rd edition):

Exam part class 1 (N) class 2 (E) class 3 (A)
Technische 195 463 716
Betriebliche 172 0 0
Vorschriften 204 0 0

Two things a consumer must know:

  1. Operational + Regulations are class-1-only in the data, but apply to every candidate. BNetzA treats these as a shared foundation. Do not filter them by class.
  2. In Technische, the class field is a floor, not an equality marker. German amateur-radio exam knowledge is cumulative: a class-E candidate is expected to know everything at class N and E; a class-A candidate knows class N + E + A. Treating class as equality underreports the E and A study pools.

The candidate study pools work out as:

  • N: 195 (Tech class 1) + 172 (Betr) + 204 (Vor) = 571
  • E: (195 + 463) Tech + 172 + 204 = 1034
  • A: (195 + 463 + 716) Tech + 172 + 204 = 1750

The 1750 total exactly matches the full catalog, which confirms the floor interpretation: an A candidate's pool is the entire catalog.

Downloader scope. v1 does not split data on disk — the JSON tree carries both axes already and consumers slice it themselves. This subsection exists so that future consumers (study app, flashcards, diff tool) implement the slicing correctly without re-deriving it from the data.

4. Tool design (v1)

Goal

Given no arguments, fetch the current BNetzA ZIP, extract it into a clean per-edition directory, and write a manifest. Re-running is a no-op when the upstream file has not changed.

CLI shape

amateurfunk-fetch [--out DIR] [--force] [--keep-zip]
  • --out DIR — output root (default ./data). Each edition lands in DIR/<edition-slug>/, e.g. data/2024-03-20-3-auflage/.
  • --force — re-download and re-extract even if the existing manifest matches.
  • --keep-zip — keep the raw ZIP alongside the extracted tree (for archival). Default deletes it after successful extraction.

Exit codes: 0 success (extracted or up-to-date), 1 network / validation error, 2 invalid local state (e.g. a partial previous run the tool can't reconcile without --force).

Steps

  1. HEAD the ZIP URL to read Last-Modified (and Content-Length for a basic sanity range). If DIR/manifest-latest.json exists and its http_last_modified equals the current server value AND the target DIR/<slug>/manifest.json it points at is present and parseable, exit 0 unchanged. The manifest — not the raw ZIP — is the trusted record after a successful validated extraction; the ZIP sha256 stays in the manifest as provenance, not as something we re-verify on every run. (We delete the ZIP by default; there would be nothing to re-verify against.)
  2. GET the ZIP to a temp file in DIR/.tmp/. Stream to disk, compute sha256 on the fly. Enforce a compressed max size (e.g. 50 MB) and, after open, a total uncompressed max size (e.g. 200 MB) as a guardrail against zip-bomb-style upstream regressions.
  3. Validate, failing closed on structural problems:
    • zipfile.is_zipfile() is true.
    • All ZIP paths are relative and normalized: reject any entry whose name is absolute, contains a .. segment, or whose os.path.normpath-result differs in a way that escapes the extraction root — defends against zip-slip.
    • No symlink entries. ZIPs created on Unix encode symlinks in the upper 16 bits of ZipInfo.external_attr (the POSIX mode field). Reject any entry where (zi.external_attr >> 16) & 0o170000 == 0o120000 (S_IFLNK), in addition to the path checks above.
    • Sum of uncompressed sizes is below the configured cap.
    • Archive contains exactly one root-level fragenkatalog*.json.
    • Archive contains more than 100 file entries whose normalized path starts with svgs/. We do not require a standalone svgs/ directory record — many ZIP producers omit directory entries and only emit file entries like svgs/AB108_q.svg.
    • The JSON parses and has top-level keys metadata and sections.
    • metadata carries the required keys edition, issued_on, valid_from, license. Extra keys are tolerated.
    • Every section node has either nested sections or a questions list (not both, not neither).
    • Every question has number, class, question, answer_a, answer_b, answer_c, answer_d.
    • For every picture_* reference in a question, the named file exists under svgs/ in the archive. Soft check: missing references do not fail the extraction. They are collected into the manifest as missing_pictures so consumers can decide locally. Rationale: a single upstream typo should not brick the downloader for every consumer until BNetzA ships a fix.
  4. Derive edition slug from metadata.issued_on plus an edition ordinal parsed from metadata.edition, e.g. "3. Auflage, März 2024" + 2024-03-20 → 2024-03-20-3-auflage. The ordinal is taken from the leading \d+ in the edition string; if absent, fall back to unknown-auflage. Sortable, filesystem- safe, and independent of German month-name parsing. The full upstream metadata.edition is preserved verbatim in manifest.json.
  5. Extract into DIR/<slug>.tmp/. Replacement semantics:
    • If DIR/<slug>/manifest.json already exists and its recorded zip_sha256 matches the freshly downloaded ZIP, skip extraction, update manifest-latest.json if needed, and exit 0.
    • If DIR/<slug>/ exists but does not match and --force is not set, exit 2 with a clear message naming the offending directory.
    • With --force, after extraction completes into DIR/<slug>.tmp/, rename the existing DIR/<slug>/ to DIR/<slug>.bak/, rename DIR/<slug>.tmp/ into place, then remove DIR/<slug>.bak/. A crash between the two renames leaves a recoverable .bak directory.
    • .bak collision policy: if DIR/<slug>.bak/ already exists before the forced replacement starts, exit 2 with a clear error naming the stale .bak/ path. A leftover .bak/ is evidence that a previous run crashed mid-rename; deciding whether to keep or delete it is the operator's call, not ours. The same applies to a stale DIR/<slug>.tmp/: refuse to overwrite, surface the path. Both checks happen before any rename — no destructive action without a clean predecessor state. Always copy the upstream README.txt verbatim into the extracted tree. The manifest also records the attribution string for convenience, but the file is the source of truth — lossless preservation beats parser-derived fields.
  6. Write DIR/<slug>/manifest.json with:
    • source_url
    • fetched_at (ISO 8601 UTC)
    • http_last_modified (verbatim from the server)
    • zip_sha256
    • zip_size
    • json_filename (the actual fragenkatalog*.json we found)
    • the full upstream metadata block
    • the verbatim attribution string from README.txt
    • missing_pictures (array of {question_number, field, file}, empty when the archive is clean)
  7. Atomically update DIR/manifest-latest.json to point at the current slug. Precise sequence:
    1. Write the new content to a sibling manifest-latest.json.tmp in the same directory as the target.
    2. os.fsync(tmp_fd) to flush the tmp file's contents to disk.
    3. os.replace(tmp_path, final_path) to atomically swap.
    4. On POSIX, open the containing directory and os.fsync its file descriptor so the rename itself is durable. Wrap in a try/except OSError and ignore on platforms where directory fsync is not supported (e.g. Windows) — the swap is still atomic, only its on-disk persistence guarantee differs. A symlink at manifest-latest would be nicer on POSIX but a small pointer file is portable.
  8. Clean up the temp ZIP (unless --keep-zip).

Idempotency and safety

  • Never extract directly into the final directory — always into a sibling *.tmp that is renamed on success. A crash mid-extract leaves the previous good edition untouched.
  • Network calls go through urllib.request with a sane User-Agent (amateurfunk-fetch/<version> (+contact)) and a timeout. Single retry on transient errors (no exponential-backoff library needed for a once-a-quarter download).
  • No telemetry. No mutation outside --out.

Testing focus

The most valuable tests are behavioral, not network-bound. The real BNetzA fetch stays as an opt-in integration test (skipped by default); everything else runs against fixture ZIPs.

  • Slug generation from sample metadata (covers normal, missing ordinal, unusual edition strings).
  • ZIP path rejection: absolute paths, .. segments, symlinks.
  • Uncompressed-size cap triggers cleanly.
  • Validation failures: missing JSON, multiple fragenkatalog*.json, missing svgs/, malformed JSON, missing required top-level keys, missing required metadata/question keys, malformed section nodes.
  • missing_pictures is populated but extraction still succeeds when a picture_* reference doesn't resolve (soft check).
  • Manifest is well-formed and contains the upstream attribution and metadata block verbatim.
  • Last-Modified round-trip: after a successful extraction, a rerun with the same Last-Modified header on the HEAD response skips the GET entirely. This is the actual contract idempotency hangs on now that the ZIP is deleted by default.
  • --force path: existing non-matching DIR/<slug>/ is replaced via the temp/bak rename dance, and a simulated crash between the two renames leaves a recoverable .bak/.
  • Atomic write of manifest-latest.json: a write that fails partway does not corrupt the existing pointer file.

What we deliberately do NOT do in v1

  • No PDF mirroring.
  • No question rendering (LaTeX, SVG).
  • No splitting by class / chapter on disk — the JSON tree already carries that structure and consumers can slice it themselves.
  • No diffing between editions. Useful, but a separate tool that reads two manifest dirs.
  • No mirror to S3 / a release artifact. Out of scope unless asked.

5. Resolved during review

These were open in the first draft and have been settled:

  1. Edition slug format — resolved: derive from issued_on + numeric edition ordinal, e.g. 2024-03-20-3-auflage. Avoids parsing German month names; preserves day-precision; sortable.
  2. manifest-latest.json location — at DIR/ root (one stable pointer for consumers); per-edition manifest.json is the immutable record.
  3. Idempotency without a retained ZIP — resolved: the manifest is the trusted record after extraction. Last-Modified match is sufficient to skip the download; zip_sha256 is recorded for provenance only.
  4. README.txt preservation — copy verbatim into the extracted tree alongside any derived attribution field in the manifest.
  5. Missing picture references — soft-validate (record in missing_pictures, do not fail). Rationale: avoid bricking the tool on an upstream typo.

6. Open questions

These do not block writing v1:

  1. PNG fallbacks — the archive ships PNGs for ~a handful of figures alongside the SVGs. v1 just extracts everything as-is. A future renderer can prefer SVG and fall back to PNG transparently.
  2. Schema drift safety — if a future edition adds or renames fields, the validator should warn but not fail. v1 fails closed on missing required top-level/metadata/question keys but tolerates extra keys. We can soften this further if BNetzA evolves the format.
  3. Packaging — v1 stays a single-file script with argparse (amateurfunk_fetch.py). Considered and declined: a pyproject.toml package with a console-script entry point. For a tool that runs at most quarterly, the packaging ceremony does not pay for itself. Revisit if the scope grows (e.g. importable from another project, distributed on PyPI).
Reference in New Issue View Git Blame Copy Permalink