Tudor Sitaru
62 lines
3.0 KiB
Markdown
62 lines
3.0 KiB
Markdown
# Incremental Snapshot Downloader Design
|
||
|
||
**Date:** 2026-05-15
|
||
**Status:** Approved
|
||
|
||
## Problem
|
||
|
||
The snapshot downloader currently produces a new HTML file on every daily run, named `snapshots_{date_from}_to_{date_to}.html`. Because `date_to` changes each day, files accumulate with near-identical content. The user wants a single file updated in place each day.
|
||
|
||
## Approach
|
||
|
||
Incremental fetch with a JSON data cache and a state file. On each run, only new snapshots are fetched from the API; they are merged into a persistent cache of all snapshot objects; the single output HTML is re-rendered from the full cache.
|
||
|
||
## Data Files
|
||
|
||
All written to `output_dir`:
|
||
|
||
| File | Purpose |
|
||
|---|---|
|
||
| `snapshots_cache.json` | Array of every snapshot object fetched so far |
|
||
| `last_run.json` | `{"last_date_to": "YYYY-MM-DD"}` — upper bound of the last successful fetch |
|
||
| `snapshots.html` | Single output file, fixed name, overwritten each run |
|
||
|
||
## Run Logic
|
||
|
||
1. Read `last_run.json` → use `last_date_to` as `date_from` for this run. Fall back to the `date_from` argument (from config) if no state file exists yet (first run).
|
||
2. Load `snapshots_cache.json` if it exists → existing snapshot list. Empty list if absent.
|
||
3. Fetch new snapshots from the API with `date_from` (from step 1) and `date_to = today`.
|
||
4. Merge: append new snapshots to the cache list, deduplicating by snapshot `id`. The boundary date may be returned by the API again on subsequent runs.
|
||
5. Save the updated `snapshots_cache.json`.
|
||
6. Re-render `snapshots.html` from the full merged list (sorted newest-first, existing logic unchanged).
|
||
7. Write `last_run.json` with today's date.
|
||
|
||
**On fetch failure:** steps 5–7 are skipped. The cache and state file remain unchanged, so the next run retries from the same `date_from`.
|
||
|
||
## Code Changes
|
||
|
||
All changes are in `src/snapshot_downloader.py`. `ConfigSnapshotDownloader` is unchanged.
|
||
|
||
### New methods on `SnapshotDownloader`
|
||
|
||
- `load_snapshot_cache() -> List[Dict]` — reads `snapshots_cache.json`; returns `[]` if absent or malformed
|
||
- `save_snapshot_cache(snapshots: List[Dict])` — writes `snapshots_cache.json`
|
||
- `load_last_run_date() -> Optional[str]` — reads `last_run.json`; returns `None` if absent
|
||
- `save_last_run_date(date: str)` — writes `last_run.json`
|
||
|
||
### Modified methods
|
||
|
||
**`download_snapshots(type_ids, date_from, date_to, max_pages)`**
|
||
|
||
`date_from` becomes the "initial history start" fallback (used only when `last_run.json` does not exist). `date_to` is always set to today's date, regardless of what is passed. The method orchestrates the 7-step incremental flow above.
|
||
|
||
**`generate_html_file(snapshots, date_from, date_to)`**
|
||
|
||
Output filename changes from `snapshots_{date_from}_to_{date_to}.html` to the fixed name `snapshots.html`.
|
||
|
||
## Out of Scope
|
||
|
||
- Detecting and applying updates to existing snapshot objects (notes edits, etc.) — snapshots are treated as append-only
|
||
- Pruning the cache (old entries are kept indefinitely)
|
||
- Changes to `ConfigSnapshotDownloader`, the web server, or any other module
|