imple/Amateurfunk-Anki
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial design

Browse Source
This commit is contained in:
Renat Nurgaliyev
2026-05-20 13:21:09 +02:00
commit 8c97c16885
2 changed files with 517 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+76
View File
@@ -0,0 +1,76 @@
# Amateurfunk — BNetzA Question Catalog Downloader
A small Python tool that downloads the latest German amateur-radio exam
question catalog ("Fragenkatalog") published by the Bundesnetzagentur
(BNetzA) and extracts the structured JSON + SVG image assets for further
use (study apps, flashcards, training tools, etc.).
The full source-discovery notes, JSON schema, and design decisions live
in `DESIGN.md`. This file is a short orientation for anyone (human or
agent) opening the project.
## What the catalog is
- Official German amateur-radio exam questions for classes **N, E, A**
(German license tiers).
- Published by the Bundesnetzagentur under the **DL-DE→BY-2.0** open
data license (free reuse, attribution required).
- Distributed as a single ZIP containing one JSON file with the full
question tree, plus a `svgs/` folder with figures referenced by
individual questions.
- Current edition at time of writing: **3. Auflage, März 2024**
(issued 2024-03-20, valid from 2024-06-24, ~1750 questions).
## Canonical source
- ZIP (machine-readable): `https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/PruefungsfragenZIP.zip?__blob=publicationFile`
- Landing page (short link): `https://www.bnetza.de/amateurfunk-fragenkatalog`
- PDF (human-readable, not used by this tool): same path with
`Pruefungsfragen.pdf` instead of `PruefungsfragenZIP.zip`.
The ZIP URL is stable across editions — BNetzA replaces the file
in-place. The `Last-Modified` HTTP header is reliable for change
detection. The filename inside the ZIP (`fragenkatalog3b.json`) encodes
the edition (`3b` = 3rd edition, revision b) and will change on new
editions, so we discover it from the archive rather than hard-coding.
## Scope of the tool (initial)
1. Download the ZIP from the canonical URL.
2. Verify it is a valid ZIP and contains the expected JSON + SVG files.
3. Extract to a target directory (default: `./data/<edition>/`).
4. Emit a small `manifest.json` next to the data: source URL, fetched-at
timestamp, `Last-Modified` from the server, JSON edition metadata,
sha256 of the ZIP.
5. Be idempotent — re-running without an upstream change is a no-op.
The skip key is the HTTP `Last-Modified` header recorded on the
previous manifest; the ZIP is deleted by default after extraction,
so the recorded sha256 is provenance, not a re-verification target.
See `DESIGN.md` for the full idempotency contract.
Out of scope for v1 (kept for later): rendering LaTeX/SVG, building a
study app, multi-edition diffing, mirroring the PDF.
## Repo conventions
- Python 3.11+, standard library only where reasonable (`urllib`,
`zipfile`, `hashlib`, `json`, `pathlib`). Add deps only if they pay
for themselves — none expected for v1.
- Single-purpose script: `amateurfunk_fetch.py` (or a tiny package if
it grows). No framework, no CLI library beyond `argparse`.
- Downloaded data is treated as a build artifact: kept under `data/`
and gitignored.
- License attribution string (required by DL-DE→BY-2.0) is preserved
verbatim from the upstream `README.txt` whenever we redistribute the
data.
## Working on this repo
- Start from `DESIGN.md` — it has the JSON schema, the question/answer
conventions (answer A is always correct, B/C/D are distractors), the
LaTeX-in-questions caveat, and the SVG naming convention.
- Do not invent new download URLs; the ones in `DESIGN.md` were
verified against the live BNetzA site.
- When BNetzA publishes a new edition, expect a new
`fragenkatalog<N><rev>.json` filename inside the ZIP. The downloader
must not hard-code the current name.
DESIGN.md
+441
View File
@@ -0,0 +1,441 @@
# Design Notes — BNetzA Amateurfunk Question Catalog Downloader
Status: **design only, no code yet.** This document captures the
source-discovery work and the proposed shape of the tool. Everything
below was verified against the live BNetzA site in May 2026.
---
## 1. Source of the latest questions
The Bundesnetzagentur (BNetzA) is the authoritative publisher. They
provide the exam catalog in two parallel formats from the same landing
page:
- **PDF** — human-readable, ~5 MB.
- **ZIP** — machine-readable, ~3 MB, contains JSON + SVG figures. This
is the format we use.
### Landing page
- Short URL: `https://www.bnetza.de/amateurfunk-fragenkatalog`
- HTTP 301 → `SharedDocs/Downloads/.../Fragenkatalog/KurzURLFragenkatalog.html`
- The short URL points at an HTML page, not at the ZIP itself. We do
not use it for fetching; it is useful only as a citation/attribution
target.
### Direct download URLs (verified 2026-05-20)
- **ZIP (what we fetch):**
`https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/PruefungsfragenZIP.zip?__blob=publicationFile`
- HTTP 200, `Content-Type: application/zip`, `Accept-Ranges: bytes`,
serves a fresh `Last-Modified` header. Suitable for conditional
fetch and idempotency checks.
- **PDF (not used by this tool):**
`https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Telekommunikation/Unternehmen_Institutionen/Frequenzen/Amateurfunk/Fragenkatalog/Pruefungsfragen.pdf?__blob=publicationFile`
The `?__blob=publicationFile` query string is required — without it the
CMS serves an HTML wrapper, not the binary.
### License
`DL-DE→BY-2.0` (Datenlizenz Deutschland – Namensnennung – Version 2.0,
see `www.govdata.de/dl-de/by-2-0`). Commercial and non-commercial reuse
is permitted with attribution. The exact attribution string required is
spelled out in the ZIP's `README.txt`; we copy it verbatim into any
redistribution and into our `manifest.json`.
### Edition tracking
BNetzA does not version the URL. The same `PruefungsfragenZIP.zip`
path is updated in place when a new edition is published. To detect a
new edition we rely on:
1. The HTTP `Last-Modified` response header.
2. The sha256 of the downloaded ZIP.
3. The `metadata` block inside the JSON (`edition`, `issued_on`,
`valid_from`).
Current edition observed: `3. Auflage, März 2024` (issued 2024-03-20,
valid from 2024-06-24).
---
## 2. ZIP contents (verified by extracting the live file)
```
fragenkatalog3b.json — single JSON file, full question tree (~1.3 MB)
README.txt — license + schema documentation
svgs/ — 700+ figures (mostly SVG, a few PNG fallbacks)
AB108_q.svg
AB404_a.svg
AB404_b.svg
...
NG302_q.png — a couple of questions ship a PNG alongside
NG302_q.svg the SVG for display-problem fallbacks
```
Total: 706 entries, 701 of them files. ~1750 questions across classes
N, E, A (counts observed: N=571, E=463, A=716).
The JSON filename encodes the edition: `fragenkatalog3b.json` =
3rd edition, revision b. A future edition will rename this file
(`fragenkatalog4a.json`, etc.), so the loader must discover it from
the archive (glob `fragenkatalog*.json`), not hard-code the name.
### `README.txt` highlights
- Confirms the two-part structure: JSON for catalog/questions, SVG for
images.
- Confirms the JSON schema (see section 3).
- Notes that question text may contain **LaTeX** for formulas, intended
to be rendered by something like KaTeX.
---
## 3. JSON schema
Top-level object:
```jsonc
{
"metadata": {
"edition": "3. Auflage, März 2024",
"issued_on": "2024-03-20",
"valid_from": "2024-06-24",
"license": "DL-DE->BY-2.0"
},
"sections": [ /* recursive section nodes */ ]
}
```
Each `section` node is either an **inner node** (contains nested
`sections`) or a **leaf** (contains a `questions` list):
```jsonc
// inner
{ "title": "Prüfungsfragen im Prüfungsteil: Technische Kenntnisse",
"sections": [ ... ] }
// leaf
{ "title": "Allgemeine mathematische Grundkenntnisse und Größen",
"questions": [ /* question objects */ ] }
```
Question object (fields per the upstream README):
| field | meaning |
|--------------------|-----------------------------------------------------------|
| `number` | Catalog id, e.g. `NA103`, `AB404`. Also the SVG basename. |
| `class` | License class: `"1"` = N, `"2"` = E, `"3"` = A. |
| `question` | Question text. May contain LaTeX. |
| `answer_a` | **The correct answer.** Always A. |
| `answer_b` | Distractor. |
| `answer_c` | Distractor. |
| `answer_d` | Distractor. |
| `picture_question` | Optional. Figure shown with the question stem. |
| `picture_a`..`_d` | Optional. Per-choice figures. |
The `picture_*` fields, when present, contain a filename (e.g.
`AB404_q.svg`) that resolves into `svgs/`. Convention observed in the
archive: `<number>_q.svg` for the question figure, `<number>_a.svg` ..
`<number>_d.svg` for per-answer figures. A small number of entries also
ship a `.png` next to the `.svg`.
### Important consumer-side conventions
- **Answer A is always correct.** Anything that presents the questions
to a learner must shuffle A/B/C/D before display, otherwise the
exercise is trivial.
- LaTeX in the question text and answers is unescaped — consumers
render it (e.g. KaTeX/MathJax). The downloader does not transform it.
- Classes are stored as string digits, not letter codes — map `"1"→N`,
`"2"→E`, `"3"→A` for display.
Sample question (from `fragenkatalog3b.json`):
```json
{
"number": "NA103",
"class": "1",
"question": "Laut Datenblatt wiegen 100 m eines bestimmten Drahtes 210 g. Ein vorliegendes Drahtstück desselben Materials wiegt 55 g. Wie lang ist das Drahtstück in etwa?",
"answer_a": "26,2 m",
"answer_b": "382 m",
"answer_c": "115 m",
"answer_d": "38,2 m"
}
```
### Exam structure — how the catalog splits into exam parts and classes
The catalog already encodes two orthogonal axes. The downloader does
not slice the data on disk, but any consumer needs to understand both
or they will compute the wrong candidate study pool.
**Axis 1 — Exam part (Prüfungsteil).** The top-level `sections[]`
array has exactly three entries, one per exam part:
| Top-level `title` | Question count | ID prefix |
|----------------------------------------------------------------|---------------:|----------------|
| `Prüfungsfragen im Prüfungsteil: Technische Kenntnisse` | 1374 | `N*`/`E*`/`A*` |
| `Prüfungsfragen im Prüfungsteil: Betriebliche Kenntnisse` | 172 | `B*` |
| `Prüfungsfragen im Prüfungsteil: Kenntnisse von Vorschriften` | 204 | `V*` |
Consumers can split on the section title (canonical) or on the
question `number` first letter (shorthand). The first-letter mapping
is: `A`/`E`/`N` → Technische; `B` → Betriebliche; `V` → Vorschriften.
Inside Technische, the first letter additionally mirrors the license
class (see Axis 2).
**Axis 2 — License class (`class` field on each question).**
Values are `"1"`=N, `"2"`=E, `"3"`=A. Class distribution per exam
part (counts verified against the live catalog, 3rd edition):
| Exam part | class 1 (N) | class 2 (E) | class 3 (A) |
|-------------|------------:|------------:|------------:|
| Technische | 195 | 463 | 716 |
| Betriebliche| 172 | 0 | 0 |
| Vorschriften| 204 | 0 | 0 |
Two things a consumer must know:
1. **Operational + Regulations are class-1-only in the data, but
apply to every candidate.** BNetzA treats these as a shared
foundation. Do not filter them by `class`.
2. **In Technische, the `class` field is a floor, not an equality
marker.** German amateur-radio exam knowledge is cumulative: a
class-E candidate is expected to know everything at class N and
E; a class-A candidate knows class N + E + A. Treating `class`
as equality underreports the E and A study pools.
The candidate study pools work out as:
- **N**: 195 (Tech class 1) + 172 (Betr) + 204 (Vor) = **571**
- **E**: (195 + 463) Tech + 172 + 204 = **1034**
- **A**: (195 + 463 + 716) Tech + 172 + 204 = **1750**
The 1750 total exactly matches the full catalog, which confirms the
floor interpretation: an A candidate's pool is the entire catalog.
**Downloader scope.** v1 does not split data on disk — the JSON tree
carries both axes already and consumers slice it themselves. This
subsection exists so that future consumers (study app, flashcards,
diff tool) implement the slicing correctly without re-deriving it
from the data.
---
## 4. Tool design (v1)
### Goal
Given no arguments, fetch the current BNetzA ZIP, extract it into a
clean per-edition directory, and write a manifest. Re-running is a
no-op when the upstream file has not changed.
### CLI shape
```
amateurfunk-fetch [--out DIR] [--force] [--keep-zip]
```
- `--out DIR` — output root (default `./data`). Each edition lands in
`DIR/<edition-slug>/`, e.g. `data/2024-03-20-3-auflage/`.
- `--force` — re-download and re-extract even if the existing manifest
matches.
- `--keep-zip` — keep the raw ZIP alongside the extracted tree (for
archival). Default deletes it after successful extraction.
Exit codes: `0` success (extracted or up-to-date), `1` network /
validation error, `2` invalid local state (e.g. a partial previous run
the tool can't reconcile without `--force`).
### Steps
1. **HEAD** the ZIP URL to read `Last-Modified` (and `Content-Length`
for a basic sanity range). If `DIR/manifest-latest.json` exists
and its `http_last_modified` equals the current server value AND
the target `DIR/<slug>/manifest.json` it points at is present and
parseable, exit 0 unchanged. The manifest — not the raw ZIP — is
the trusted record after a successful validated extraction; the
ZIP sha256 stays in the manifest as provenance, not as something
we re-verify on every run. (We delete the ZIP by default; there
would be nothing to re-verify against.)
2. **GET** the ZIP to a temp file in `DIR/.tmp/`. Stream to disk,
compute sha256 on the fly. Enforce a compressed max size (e.g.
50 MB) and, after open, a total uncompressed max size (e.g.
200 MB) as a guardrail against zip-bomb-style upstream regressions.
3. **Validate**, failing closed on structural problems:
- `zipfile.is_zipfile()` is true.
- All ZIP paths are relative and normalized: reject any entry
whose name is absolute, contains a `..` segment, or whose
`os.path.normpath`-result differs in a way that escapes the
extraction root — defends against zip-slip.
- **No symlink entries.** ZIPs created on Unix encode symlinks in
the upper 16 bits of `ZipInfo.external_attr` (the POSIX mode
field). Reject any entry where
`(zi.external_attr >> 16) & 0o170000 == 0o120000` (S_IFLNK),
in addition to the path checks above.
- Sum of uncompressed sizes is below the configured cap.
- Archive contains **exactly one** root-level `fragenkatalog*.json`.
- Archive contains **more than 100 file entries** whose normalized
path starts with `svgs/`. We do not require a standalone `svgs/`
directory record — many ZIP producers omit directory entries
and only emit file entries like `svgs/AB108_q.svg`.
- The JSON parses and has top-level keys `metadata` and `sections`.
- `metadata` carries the required keys `edition`, `issued_on`,
`valid_from`, `license`. Extra keys are tolerated.
- Every section node has either nested `sections` or a `questions`
list (not both, not neither).
- Every question has `number`, `class`, `question`, `answer_a`,
`answer_b`, `answer_c`, `answer_d`.
- For every `picture_*` reference in a question, the named file
exists under `svgs/` in the archive. **Soft check:** missing
references do not fail the extraction. They are collected into
the manifest as `missing_pictures` so consumers can decide
locally. Rationale: a single upstream typo should not brick the
downloader for every consumer until BNetzA ships a fix.
4. **Derive edition slug** from `metadata.issued_on` plus an edition
ordinal parsed from `metadata.edition`, e.g.
`"3. Auflage, März 2024"` + `2024-03-20` → `2024-03-20-3-auflage`.
The ordinal is taken from the leading `\d+` in the edition string;
if absent, fall back to `unknown-auflage`. Sortable, filesystem-
safe, and independent of German month-name parsing. The full
upstream `metadata.edition` is preserved verbatim in `manifest.json`.
5. **Extract** into `DIR/<slug>.tmp/`. Replacement semantics:
- If `DIR/<slug>/manifest.json` already exists and its recorded
`zip_sha256` matches the freshly downloaded ZIP, skip extraction,
update `manifest-latest.json` if needed, and exit 0.
- If `DIR/<slug>/` exists but does not match and `--force` is not
set, exit 2 with a clear message naming the offending directory.
- With `--force`, after extraction completes into `DIR/<slug>.tmp/`,
rename the existing `DIR/<slug>/` to `DIR/<slug>.bak/`, rename
`DIR/<slug>.tmp/` into place, then remove `DIR/<slug>.bak/`. A
crash between the two renames leaves a recoverable `.bak`
directory.
- **`.bak` collision policy:** if `DIR/<slug>.bak/` already exists
before the forced replacement starts, exit 2 with a clear error
naming the stale `.bak/` path. A leftover `.bak/` is evidence
that a previous run crashed mid-rename; deciding whether to keep
or delete it is the operator's call, not ours. The same applies
to a stale `DIR/<slug>.tmp/`: refuse to overwrite, surface the
path. Both checks happen *before* any rename — no destructive
action without a clean predecessor state.
Always copy the upstream `README.txt` verbatim into the extracted
tree. The manifest also records the attribution string for
convenience, but the file is the source of truth — lossless
preservation beats parser-derived fields.
6. **Write `DIR/<slug>/manifest.json`** with:
- `source_url`
- `fetched_at` (ISO 8601 UTC)
- `http_last_modified` (verbatim from the server)
- `zip_sha256`
- `zip_size`
- `json_filename` (the actual `fragenkatalog*.json` we found)
- the full upstream `metadata` block
- the verbatim attribution string from `README.txt`
- `missing_pictures` (array of `{question_number, field, file}`,
empty when the archive is clean)
7. **Atomically update `DIR/manifest-latest.json`** to point at the
current slug. Precise sequence:
1. Write the new content to a sibling `manifest-latest.json.tmp`
in the same directory as the target.
2. `os.fsync(tmp_fd)` to flush the tmp file's contents to disk.
3. `os.replace(tmp_path, final_path)` to atomically swap.
4. On POSIX, open the containing directory and `os.fsync` its
file descriptor so the rename itself is durable. Wrap in a
`try/except OSError` and ignore on platforms where directory
fsync is not supported (e.g. Windows) — the swap is still
atomic, only its on-disk persistence guarantee differs.
A symlink at `manifest-latest` would be nicer on POSIX but a
small pointer file is portable.
8. **Clean up** the temp ZIP (unless `--keep-zip`).
### Idempotency and safety
- Never extract directly into the final directory — always into a
sibling `*.tmp` that is renamed on success. A crash mid-extract leaves
the previous good edition untouched.
- Network calls go through `urllib.request` with a sane User-Agent
(`amateurfunk-fetch/<version> (+contact)`) and a timeout. Single
retry on transient errors (no exponential-backoff library needed for
a once-a-quarter download).
- No telemetry. No mutation outside `--out`.
### Testing focus
The most valuable tests are behavioral, not network-bound. The real
BNetzA fetch stays as an opt-in integration test (skipped by default);
everything else runs against fixture ZIPs.
- Slug generation from sample `metadata` (covers normal, missing
ordinal, unusual edition strings).
- ZIP path rejection: absolute paths, `..` segments, symlinks.
- Uncompressed-size cap triggers cleanly.
- Validation failures: missing JSON, multiple `fragenkatalog*.json`,
missing `svgs/`, malformed JSON, missing required top-level keys,
missing required metadata/question keys, malformed section nodes.
- `missing_pictures` is populated but extraction still succeeds when
a `picture_*` reference doesn't resolve (soft check).
- Manifest is well-formed and contains the upstream attribution and
`metadata` block verbatim.
- **`Last-Modified` round-trip**: after a successful extraction, a
rerun with the same `Last-Modified` header on the HEAD response
skips the GET entirely. This is the actual contract idempotency
hangs on now that the ZIP is deleted by default.
- `--force` path: existing non-matching `DIR/<slug>/` is replaced via
the temp/bak rename dance, and a simulated crash between the two
renames leaves a recoverable `.bak/`.
- Atomic write of `manifest-latest.json`: a write that fails partway
does not corrupt the existing pointer file.
### What we deliberately do NOT do in v1
- No PDF mirroring.
- No question rendering (LaTeX, SVG).
- No splitting by class / chapter on disk — the JSON tree already
carries that structure and consumers can slice it themselves.
- No diffing between editions. Useful, but a separate tool that reads
two manifest dirs.
- No mirror to S3 / a release artifact. Out of scope unless asked.
---
## 5. Resolved during review
These were open in the first draft and have been settled:
1. **Edition slug format** — resolved: derive from `issued_on` +
numeric edition ordinal, e.g. `2024-03-20-3-auflage`. Avoids
parsing German month names; preserves day-precision; sortable.
2. **`manifest-latest.json` location** — at `DIR/` root (one stable
pointer for consumers); per-edition `manifest.json` is the
immutable record.
3. **Idempotency without a retained ZIP** — resolved: the manifest is
the trusted record after extraction. `Last-Modified` match is
sufficient to skip the download; `zip_sha256` is recorded for
provenance only.
4. **README.txt preservation** — copy verbatim into the extracted
tree alongside any derived attribution field in the manifest.
5. **Missing picture references** — soft-validate (record in
`missing_pictures`, do not fail). Rationale: avoid bricking the
tool on an upstream typo.
## 6. Open questions
These do not block writing v1:
1. **PNG fallbacks** — the archive ships PNGs for ~a handful of
figures alongside the SVGs. v1 just extracts everything as-is. A
future renderer can prefer SVG and fall back to PNG transparently.
2. **Schema drift safety** — if a future edition adds or renames
fields, the validator should warn but not fail. v1 fails closed
on missing required top-level/metadata/question keys but tolerates
extra keys. We can soften this further if BNetzA evolves the
format.
3. **Packaging** — v1 stays a single-file script with `argparse`
(`amateurfunk_fetch.py`). Considered and declined: a
`pyproject.toml` package with a console-script entry point. For a
tool that runs at most quarterly, the packaging ceremony does not
pay for itself. Revisit if the scope grows (e.g. importable from
another project, distributed on PyPI).