451 lines
20 KiB
Markdown
451 lines
20 KiB
Markdown
# Design Notes — BNetzA Amateurfunk Question Catalog Downloader
|
||
|
||
Status: **design only, no code yet.** This document captures the
|
||
source-discovery work and the proposed shape of the tool. Everything
|
||
below was verified against the live BNetzA site in May 2026.
|
||
|
||
---
|
||
|
||
## 1. Source of the latest questions
|
||
|
||
The Bundesnetzagentur (BNetzA) is the authoritative publisher. They
|
||
provide the exam catalog in two parallel formats from the same landing
|
||
page:
|
||
|
||
- **PDF** — human-readable, ~5 MB.
|
||
- **ZIP** — machine-readable, ~3 MB, contains JSON + SVG figures. This
|
||
is the format we use.
|
||
|
||
### Landing page
|
||
|
||
- Short URL: `https://www.bnetza.de/amateurfunk-fragenkatalog`
|
||
- HTTP 301 → `SharedDocs/Downloads/.../Fragenkatalog/KurzURLFragenkatalog.html`
|
||
- The short URL points at an HTML page, not at the ZIP itself. We do
|
||
not use it for fetching; it is useful only as a citation/attribution
|
||
target.
|
||
|
||
### Direct download URLs (verified 2026-05-20)
|
||
|
||
- **ZIP (what we fetch):**
|
||
`https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/PruefungsfragenZIP.zip?__blob=publicationFile`
|
||
- HTTP 200, `Content-Type: application/zip`, `Accept-Ranges: bytes`,
|
||
serves a fresh `Last-Modified` header. Suitable for conditional
|
||
fetch and idempotency checks.
|
||
- **PDF (not used by this tool):**
|
||
`https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/Pruefungsfragen.pdf?__blob=publicationFile`
|
||
|
||
The `?__blob=publicationFile` query string is required — without it the
|
||
CMS serves an HTML wrapper, not the binary.
|
||
|
||
### License
|
||
|
||
`DL-DE→BY-2.0` (Datenlizenz Deutschland – Namensnennung – Version 2.0,
|
||
see `www.govdata.de/dl-de/by-2-0`). Commercial and non-commercial reuse
|
||
is permitted with attribution. The exact attribution string required is
|
||
spelled out in the ZIP's `README.txt`; we copy it verbatim into any
|
||
redistribution and into our `manifest.json`.
|
||
|
||
### Edition tracking
|
||
|
||
BNetzA does not version the URL. The same `PruefungsfragenZIP.zip`
|
||
path is updated in place when a new edition is published. To detect a
|
||
new edition we rely on:
|
||
|
||
1. The HTTP `Last-Modified` response header.
|
||
2. The sha256 of the downloaded ZIP.
|
||
3. The `metadata` block inside the JSON (`edition`, `issued_on`,
|
||
`valid_from`).
|
||
|
||
Current edition observed: `3. Auflage, März 2024` (issued 2024-03-20,
|
||
valid from 2024-06-24).
|
||
|
||
---
|
||
|
||
## 2. ZIP contents (verified by extracting the live file)
|
||
|
||
```
|
||
fragenkatalog3b.json — single JSON file, full question tree (~1.3 MB)
|
||
README.txt — license + schema documentation
|
||
svgs/ — 700+ figures (mostly SVG, a few PNG fallbacks)
|
||
AB108_q.svg
|
||
AB404_a.svg
|
||
AB404_b.svg
|
||
...
|
||
NG302_q.png — a couple of questions ship a PNG alongside
|
||
NG302_q.svg the SVG for display-problem fallbacks
|
||
```
|
||
|
||
Total: 706 entries, 701 of them files. ~1750 questions across classes
|
||
N, E, A (counts observed: N=571, E=463, A=716).
|
||
|
||
The JSON filename encodes the edition: `fragenkatalog3b.json` =
|
||
3rd edition, revision b. A future edition will rename this file
|
||
(`fragenkatalog4a.json`, etc.), so the loader must discover it from
|
||
the archive (glob `fragenkatalog*.json`), not hard-code the name.
|
||
|
||
### `README.txt` highlights
|
||
|
||
- Confirms the two-part structure: JSON for catalog/questions, SVG for
|
||
images.
|
||
- Confirms the JSON schema (see section 3).
|
||
- Notes that question text may contain **LaTeX** for formulas, intended
|
||
to be rendered by something like KaTeX.
|
||
|
||
---
|
||
|
||
## 3. JSON schema
|
||
|
||
Top-level object:
|
||
|
||
```jsonc
|
||
{
|
||
"metadata": {
|
||
"edition": "3. Auflage, März 2024",
|
||
"issued_on": "2024-03-20",
|
||
"valid_from": "2024-06-24",
|
||
"license": "DL-DE->BY-2.0"
|
||
},
|
||
"sections": [ /* recursive section nodes */ ]
|
||
}
|
||
```
|
||
|
||
Each `section` node is either an **inner node** (contains nested
|
||
`sections`) or a **leaf** (contains a `questions` list):
|
||
|
||
```jsonc
|
||
// inner
|
||
{ "title": "Prüfungsfragen im Prüfungsteil: Technische Kenntnisse",
|
||
"sections": [ ... ] }
|
||
|
||
// leaf
|
||
{ "title": "Allgemeine mathematische Grundkenntnisse und Größen",
|
||
"questions": [ /* question objects */ ] }
|
||
```
|
||
|
||
Question object (fields per the upstream README):
|
||
|
||
| field | meaning |
|
||
|--------------------|-----------------------------------------------------------|
|
||
| `number` | Catalog id, e.g. `NA103`, `AB404`. Also the SVG basename. |
|
||
| `class` | License class: `"1"` = N, `"2"` = E, `"3"` = A. |
|
||
| `question` | Question text. May contain LaTeX. |
|
||
| `answer_a` | **The correct answer.** Always A. |
|
||
| `answer_b` | Distractor. |
|
||
| `answer_c` | Distractor. |
|
||
| `answer_d` | Distractor. |
|
||
| `picture_question` | Optional. Figure shown with the question stem. |
|
||
| `picture_a`..`_d` | Optional. Per-choice figures. |
|
||
|
||
The `picture_*` fields, when present, contain a **basename without
|
||
extension** (e.g. `AB404_q`, not `AB404_q.svg`). The consumer picks
|
||
the extension. Convention observed in the archive:
|
||
|
||
- `<number>_q` for the question figure (file `<number>_q.svg`),
|
||
- `<number>_a` .. `<number>_d` for per-answer figures.
|
||
|
||
Files live under `svgs/`. A small number of entries ship a `.png`
|
||
alongside the `.svg` (display-problem fallback). Consumers should
|
||
prefer `.svg` and fall back to `.png` by stem.
|
||
|
||
This stem-only convention is why the soft picture-reference check
|
||
matches stems, not full filenames; matching the literal value
|
||
against directory listings would mark every reference missing.
|
||
|
||
### Important consumer-side conventions
|
||
|
||
- **Answer A is always correct.** Anything that presents the questions
|
||
to a learner must shuffle A/B/C/D before display, otherwise the
|
||
exercise is trivial.
|
||
- LaTeX in the question text and answers is unescaped — consumers
|
||
render it (e.g. KaTeX/MathJax). The downloader does not transform it.
|
||
- Classes are stored as string digits, not letter codes — map `"1"→N`,
|
||
`"2"→E`, `"3"→A` for display.
|
||
|
||
Sample question (from `fragenkatalog3b.json`):
|
||
|
||
```json
|
||
{
|
||
"number": "NA103",
|
||
"class": "1",
|
||
"question": "Laut Datenblatt wiegen 100 m eines bestimmten Drahtes 210 g. Ein vorliegendes Drahtstück desselben Materials wiegt 55 g. Wie lang ist das Drahtstück in etwa?",
|
||
"answer_a": "26,2 m",
|
||
"answer_b": "382 m",
|
||
"answer_c": "115 m",
|
||
"answer_d": "38,2 m"
|
||
}
|
||
```
|
||
|
||
### Exam structure — how the catalog splits into exam parts and classes
|
||
|
||
The catalog already encodes two orthogonal axes. The downloader does
|
||
not slice the data on disk, but any consumer needs to understand both
|
||
or they will compute the wrong candidate study pool.
|
||
|
||
**Axis 1 — Exam part (Prüfungsteil).** The top-level `sections[]`
|
||
array has exactly three entries, one per exam part:
|
||
|
||
| Top-level `title` | Question count | ID prefix |
|
||
|----------------------------------------------------------------|---------------:|----------------|
|
||
| `Prüfungsfragen im Prüfungsteil: Technische Kenntnisse` | 1374 | `N*`/`E*`/`A*` |
|
||
| `Prüfungsfragen im Prüfungsteil: Betriebliche Kenntnisse` | 172 | `B*` |
|
||
| `Prüfungsfragen im Prüfungsteil: Kenntnisse von Vorschriften` | 204 | `V*` |
|
||
|
||
Consumers can split on the section title (canonical) or on the
|
||
question `number` first letter (shorthand). The first-letter mapping
|
||
is: `A`/`E`/`N` → Technische; `B` → Betriebliche; `V` → Vorschriften.
|
||
Inside Technische, the first letter additionally mirrors the license
|
||
class (see Axis 2).
|
||
|
||
**Axis 2 — License class (`class` field on each question).**
|
||
Values are `"1"`=N, `"2"`=E, `"3"`=A. Class distribution per exam
|
||
part (counts verified against the live catalog, 3rd edition):
|
||
|
||
| Exam part | class 1 (N) | class 2 (E) | class 3 (A) |
|
||
|-------------|------------:|------------:|------------:|
|
||
| Technische | 195 | 463 | 716 |
|
||
| Betriebliche| 172 | 0 | 0 |
|
||
| Vorschriften| 204 | 0 | 0 |
|
||
|
||
Two things a consumer must know:
|
||
|
||
1. **Operational + Regulations are class-1-only in the data, but
|
||
apply to every candidate.** BNetzA treats these as a shared
|
||
foundation. Do not filter them by `class`.
|
||
2. **In Technische, the `class` field is a floor, not an equality
|
||
marker.** German amateur-radio exam knowledge is cumulative: a
|
||
class-E candidate is expected to know everything at class N and
|
||
E; a class-A candidate knows class N + E + A. Treating `class`
|
||
as equality underreports the E and A study pools.
|
||
|
||
The candidate study pools work out as:
|
||
|
||
- **N**: 195 (Tech class 1) + 172 (Betr) + 204 (Vor) = **571**
|
||
- **E**: (195 + 463) Tech + 172 + 204 = **1034**
|
||
- **A**: (195 + 463 + 716) Tech + 172 + 204 = **1750**
|
||
|
||
The 1750 total exactly matches the full catalog, which confirms the
|
||
floor interpretation: an A candidate's pool is the entire catalog.
|
||
|
||
**Downloader scope.** v1 does not split data on disk — the JSON tree
|
||
carries both axes already and consumers slice it themselves. This
|
||
subsection exists so that future consumers (study app, flashcards,
|
||
diff tool) implement the slicing correctly without re-deriving it
|
||
from the data.
|
||
|
||
---
|
||
|
||
## 4. Tool design (v1)
|
||
|
||
### Goal
|
||
|
||
Given no arguments, fetch the current BNetzA ZIP, extract it into a
|
||
clean per-edition directory, and write a manifest. Re-running is a
|
||
no-op when the upstream file has not changed.
|
||
|
||
### CLI shape
|
||
|
||
```
|
||
amateurfunk-fetch [--out DIR] [--force] [--keep-zip]
|
||
```
|
||
|
||
- `--out DIR` — output root (default `./data`). Each edition lands in
|
||
`DIR/<edition-slug>/`, e.g. `data/2024-03-20-3-auflage/`.
|
||
- `--force` — re-download and re-extract even if the existing manifest
|
||
matches.
|
||
- `--keep-zip` — keep the raw ZIP alongside the extracted tree (for
|
||
archival). Default deletes it after successful extraction.
|
||
|
||
Exit codes: `0` success (extracted or up-to-date), `1` network /
|
||
validation error, `2` invalid local state (e.g. a partial previous run
|
||
the tool can't reconcile without `--force`).
|
||
|
||
### Steps
|
||
|
||
1. **HEAD** the ZIP URL to read `Last-Modified` (and `Content-Length`
|
||
for a basic sanity range). If `DIR/manifest-latest.json` exists
|
||
and its `http_last_modified` equals the current server value AND
|
||
the target `DIR/<slug>/manifest.json` it points at is present and
|
||
parseable, exit 0 unchanged. The manifest — not the raw ZIP — is
|
||
the trusted record after a successful validated extraction; the
|
||
ZIP sha256 stays in the manifest as provenance, not as something
|
||
we re-verify on every run. (We delete the ZIP by default; there
|
||
would be nothing to re-verify against.)
|
||
2. **GET** the ZIP to a temp file in `DIR/.tmp/`. Stream to disk,
|
||
compute sha256 on the fly. Enforce a compressed max size (e.g.
|
||
50 MB) and, after open, a total uncompressed max size (e.g.
|
||
200 MB) as a guardrail against zip-bomb-style upstream regressions.
|
||
3. **Validate**, failing closed on structural problems:
|
||
- `zipfile.is_zipfile()` is true.
|
||
- All ZIP paths are relative and normalized: reject any entry
|
||
whose name is absolute, contains a `..` segment, or whose
|
||
`os.path.normpath`-result differs in a way that escapes the
|
||
extraction root — defends against zip-slip.
|
||
- **No symlink entries.** ZIPs created on Unix encode symlinks in
|
||
the upper 16 bits of `ZipInfo.external_attr` (the POSIX mode
|
||
field). Reject any entry where
|
||
`(zi.external_attr >> 16) & 0o170000 == 0o120000` (S_IFLNK),
|
||
in addition to the path checks above.
|
||
- Sum of uncompressed sizes is below the configured cap.
|
||
- Archive contains **exactly one** root-level `fragenkatalog*.json`.
|
||
- Archive contains **more than 100 file entries** whose normalized
|
||
path starts with `svgs/`. We do not require a standalone `svgs/`
|
||
directory record — many ZIP producers omit directory entries
|
||
and only emit file entries like `svgs/AB108_q.svg`.
|
||
- The JSON parses and has top-level keys `metadata` and `sections`.
|
||
- `metadata` carries the required keys `edition`, `issued_on`,
|
||
`valid_from`, `license`. Extra keys are tolerated.
|
||
- Every section node has either nested `sections` or a `questions`
|
||
list (not both, not neither).
|
||
- Every question has `number`, `class`, `question`, `answer_a`,
|
||
`answer_b`, `answer_c`, `answer_d`.
|
||
- For every `picture_*` reference in a question, the named file
|
||
exists under `svgs/` in the archive. **Soft check:** missing
|
||
references do not fail the extraction. They are collected into
|
||
the manifest as `missing_pictures` so consumers can decide
|
||
locally. Rationale: a single upstream typo should not brick the
|
||
downloader for every consumer until BNetzA ships a fix.
|
||
4. **Derive edition slug** from `metadata.issued_on` plus an edition
|
||
ordinal parsed from `metadata.edition`, e.g.
|
||
`"3. Auflage, März 2024"` + `2024-03-20` → `2024-03-20-3-auflage`.
|
||
The ordinal is taken from the leading `\d+` in the edition string;
|
||
if absent, fall back to `unknown-auflage`. Sortable, filesystem-
|
||
safe, and independent of German month-name parsing. The full
|
||
upstream `metadata.edition` is preserved verbatim in `manifest.json`.
|
||
5. **Extract** into `DIR/<slug>.tmp/`. Replacement semantics:
|
||
- If `DIR/<slug>/manifest.json` already exists and its recorded
|
||
`zip_sha256` matches the freshly downloaded ZIP, skip extraction,
|
||
update `manifest-latest.json` if needed, and exit 0.
|
||
- If `DIR/<slug>/` exists but does not match and `--force` is not
|
||
set, exit 2 with a clear message naming the offending directory.
|
||
- With `--force`, after extraction completes into `DIR/<slug>.tmp/`,
|
||
rename the existing `DIR/<slug>/` to `DIR/<slug>.bak/`, rename
|
||
`DIR/<slug>.tmp/` into place, then remove `DIR/<slug>.bak/`. A
|
||
crash between the two renames leaves a recoverable `.bak`
|
||
directory.
|
||
- **`.bak` collision policy:** if `DIR/<slug>.bak/` already exists
|
||
before the forced replacement starts, exit 2 with a clear error
|
||
naming the stale `.bak/` path. A leftover `.bak/` is evidence
|
||
that a previous run crashed mid-rename; deciding whether to keep
|
||
or delete it is the operator's call, not ours. The same applies
|
||
to a stale `DIR/<slug>.tmp/`: refuse to overwrite, surface the
|
||
path. Both checks happen *before* any rename — no destructive
|
||
action without a clean predecessor state.
|
||
Always copy the upstream `README.txt` verbatim into the extracted
|
||
tree. The manifest also records the attribution string for
|
||
convenience, but the file is the source of truth — lossless
|
||
preservation beats parser-derived fields.
|
||
6. **Write `DIR/<slug>/manifest.json`** with:
|
||
- `source_url`
|
||
- `fetched_at` (ISO 8601 UTC)
|
||
- `http_last_modified` (verbatim from the server)
|
||
- `zip_sha256`
|
||
- `zip_size`
|
||
- `json_filename` (the actual `fragenkatalog*.json` we found)
|
||
- the full upstream `metadata` block
|
||
- the verbatim attribution string from `README.txt`
|
||
- `missing_pictures` (array of `{question_number, field, file}`,
|
||
empty when the archive is clean)
|
||
7. **Atomically update `DIR/manifest-latest.json`** to point at the
|
||
current slug. Precise sequence:
|
||
1. Write the new content to a sibling `manifest-latest.json.tmp`
|
||
in the same directory as the target.
|
||
2. `os.fsync(tmp_fd)` to flush the tmp file's contents to disk.
|
||
3. `os.replace(tmp_path, final_path)` to atomically swap.
|
||
4. On POSIX, open the containing directory and `os.fsync` its
|
||
file descriptor so the rename itself is durable. Wrap in a
|
||
`try/except OSError` and ignore on platforms where directory
|
||
fsync is not supported (e.g. Windows) — the swap is still
|
||
atomic, only its on-disk persistence guarantee differs.
|
||
A symlink at `manifest-latest` would be nicer on POSIX but a
|
||
small pointer file is portable.
|
||
8. **Clean up** the temp ZIP (unless `--keep-zip`).
|
||
|
||
### Idempotency and safety
|
||
|
||
- Never extract directly into the final directory — always into a
|
||
sibling `*.tmp` that is renamed on success. A crash mid-extract leaves
|
||
the previous good edition untouched.
|
||
- Network calls go through `urllib.request` with a sane User-Agent
|
||
(`amateurfunk-fetch/<version> (+contact)`) and a timeout. Single
|
||
retry on transient errors (no exponential-backoff library needed for
|
||
a once-a-quarter download).
|
||
- No telemetry. No mutation outside `--out`.
|
||
|
||
### Testing focus
|
||
|
||
The most valuable tests are behavioral, not network-bound. The real
|
||
BNetzA fetch stays as an opt-in integration test (skipped by default);
|
||
everything else runs against fixture ZIPs.
|
||
|
||
- Slug generation from sample `metadata` (covers normal, missing
|
||
ordinal, unusual edition strings).
|
||
- ZIP path rejection: absolute paths, `..` segments, symlinks.
|
||
- Uncompressed-size cap triggers cleanly.
|
||
- Validation failures: missing JSON, multiple `fragenkatalog*.json`,
|
||
missing `svgs/`, malformed JSON, missing required top-level keys,
|
||
missing required metadata/question keys, malformed section nodes.
|
||
- `missing_pictures` is populated but extraction still succeeds when
|
||
a `picture_*` reference doesn't resolve (soft check).
|
||
- Manifest is well-formed and contains the upstream attribution and
|
||
`metadata` block verbatim.
|
||
- **`Last-Modified` round-trip**: after a successful extraction, a
|
||
rerun with the same `Last-Modified` header on the HEAD response
|
||
skips the GET entirely. This is the actual contract idempotency
|
||
hangs on now that the ZIP is deleted by default.
|
||
- `--force` path: existing non-matching `DIR/<slug>/` is replaced via
|
||
the temp/bak rename dance, and a simulated crash between the two
|
||
renames leaves a recoverable `.bak/`.
|
||
- Atomic write of `manifest-latest.json`: a write that fails partway
|
||
does not corrupt the existing pointer file.
|
||
|
||
### What we deliberately do NOT do in v1
|
||
|
||
- No PDF mirroring.
|
||
- No question rendering (LaTeX, SVG).
|
||
- No splitting by class / chapter on disk — the JSON tree already
|
||
carries that structure and consumers can slice it themselves.
|
||
- No diffing between editions. Useful, but a separate tool that reads
|
||
two manifest dirs.
|
||
- No mirror to S3 / a release artifact. Out of scope unless asked.
|
||
|
||
---
|
||
|
||
## 5. Resolved during review
|
||
|
||
These were open in the first draft and have been settled:
|
||
|
||
1. **Edition slug format** — resolved: derive from `issued_on` +
|
||
numeric edition ordinal, e.g. `2024-03-20-3-auflage`. Avoids
|
||
parsing German month names; preserves day-precision; sortable.
|
||
2. **`manifest-latest.json` location** — at `DIR/` root (one stable
|
||
pointer for consumers); per-edition `manifest.json` is the
|
||
immutable record.
|
||
3. **Idempotency without a retained ZIP** — resolved: the manifest is
|
||
the trusted record after extraction. `Last-Modified` match is
|
||
sufficient to skip the download; `zip_sha256` is recorded for
|
||
provenance only.
|
||
4. **README.txt preservation** — copy verbatim into the extracted
|
||
tree alongside any derived attribution field in the manifest.
|
||
5. **Missing picture references** — soft-validate (record in
|
||
`missing_pictures`, do not fail). Rationale: avoid bricking the
|
||
tool on an upstream typo.
|
||
|
||
## 6. Open questions
|
||
|
||
These do not block writing v1:
|
||
|
||
1. **PNG fallbacks** — the archive ships PNGs for ~a handful of
|
||
figures alongside the SVGs. v1 just extracts everything as-is. A
|
||
future renderer can prefer SVG and fall back to PNG transparently.
|
||
2. **Schema drift safety** — if a future edition adds or renames
|
||
fields, the validator should warn but not fail. v1 fails closed
|
||
on missing required top-level/metadata/question keys but tolerates
|
||
extra keys. We can soften this further if BNetzA evolves the
|
||
format.
|
||
3. **Packaging** — v1 stays a single-file script with `argparse`
|
||
(`amateurfunk_fetch.py`). Considered and declined: a
|
||
`pyproject.toml` package with a console-script entry point. For a
|
||
tool that runs at most quarterly, the packaging ceremony does not
|
||
pay for itself. Revisit if the scope grows (e.g. importable from
|
||
another project, distributed on PyPI).
|