Found while verifying the staging sitemap after #110 deployed.
_school_sitemap_rows tested only the latest year's row for publishable data, which quietly dropped every school with results in its history but a null row for the most recent year — schools that stopped reporting, or whose figures were suppressed for small-cohort disclosure.
The Mallard Academy (150367) is the case that caught it:
year
rwm_expected_pct
2015-16
36.0
2016-17
48.0
2017-18
44.0
2018-19
67.0
2022-23
null
2023-24
null
2024-25
null
Its detail page shows all four years of results. The sitemap omitted it.
Scale
I diffed the 22,979 sitemap URNs against the 25,185-school corpus, took the 2,206 excluded, and sampled 40 at random. Four had performance data — Mallard Academy, Catcote Academy, Downe Primary, Broadwood School. Extrapolating, roughly 220 real pages were being withheld from search.
The other 36 were correctly excluded: independent schools, FE colleges and post-16 institutions with no KS2/KS4 data and no Ofsted grade.
Fix
Publishable is now a property of the school, computed across every row, rather than of its latest row. lastmod still comes from the latest row so the most recent Ofsted date wins. The field list moved to a module constant shared with _has_publishable_data so the two checks cannot drift apart.
Two regression tests: one asserting Mallard's shape is listed, and one asserting the fix has not become "list everything" — a school with no results in any year and no Ofsted is still omitted.
Backend suite: 70 passed.
Expect the count to rise
Staging currently reports 22,983. After this, expect roughly 23,200.
Note on my earlier estimate
I predicted ~21,300 URLs for #110 and it came in at 22,983. That gap was my arithmetic, not this bug: I estimated exclusions using "no current-year data" when the actual criterion is "no data in any year and no Ofsted", which is far narrower. The bug above is a separate, real problem that the gap prompted me to go looking for.
Found while verifying the staging sitemap after #110 deployed.
`_school_sitemap_rows` tested only the **latest year's row** for publishable data, which quietly dropped every school with results in its history but a null row for the most recent year — schools that stopped reporting, or whose figures were suppressed for small-cohort disclosure.
**The Mallard Academy (150367)** is the case that caught it:
| year | rwm_expected_pct |
|---|---|
| 2015-16 | 36.0 |
| 2016-17 | 48.0 |
| 2017-18 | 44.0 |
| 2018-19 | 67.0 |
| 2022-23 | null |
| 2023-24 | null |
| 2024-25 | null |
Its detail page shows all four years of results. The sitemap omitted it.
## Scale
I diffed the 22,979 sitemap URNs against the 25,185-school corpus, took the 2,206 excluded, and sampled 40 at random. **Four had performance data** — Mallard Academy, Catcote Academy, Downe Primary, Broadwood School. Extrapolating, roughly **220 real pages** were being withheld from search.
The other 36 were correctly excluded: independent schools, FE colleges and post-16 institutions with no KS2/KS4 data and no Ofsted grade.
## Fix
Publishable is now a property of the *school*, computed across every row, rather than of its latest row. `lastmod` still comes from the latest row so the most recent Ofsted date wins. The field list moved to a module constant shared with `_has_publishable_data` so the two checks cannot drift apart.
Two regression tests: one asserting Mallard's shape is listed, and one asserting the fix has not become "list everything" — a school with no results in any year and no Ofsted is still omitted.
Backend suite: 70 passed.
## Expect the count to rise
Staging currently reports 22,983. After this, expect roughly **23,200**.
## Note on my earlier estimate
I predicted ~21,300 URLs for #110 and it came in at 22,983. That gap was my arithmetic, not this bug: I estimated exclusions using "no current-year data" when the actual criterion is "no data in any year **and** no Ofsted", which is far narrower. The bug above is a separate, real problem that the gap prompted me to go looking for.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
_school_sitemap_rows tested only the latest year's row, which quietly dropped
every school with results in its history but a null row for the most recent
year — a school that stopped reporting, or whose figures were suppressed for
small-cohort disclosure.
The Mallard Academy (150367) is the case that caught it: real KS2 results for
2015-16 through 2018-19, then null rows from 2022-23 on. Its detail page shows
all four years; the sitemap omitted it. Sampling 40 of the 2,206 excluded
schools found 4 like this, so roughly 220 real pages were being withheld.
Publishable is now a property of the school, computed across every row, while
lastmod still comes from the latest row so the most recent Ofsted date wins.
The field list is a module constant shared with _has_publishable_data so the
two checks cannot drift.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
The diff fixes a sitemap generation bug where publishability was determined only from a school's latest-year row, causing schools with valid historical results but null current-year rows (e.g. schools that stopped reporting) to be dropped from the sitemap. The fix vectorizes the check across all rows per URN and adds solid regression tests for both the fixed case and the still-must-exclude case. The change is small, well-tested, and correctness looks sound.
🟡 Minor
backend/app.py: _has_publishable_data is no longer called anywhere (the per-school check in _school_sitemap_rows now uses the precomputed publishable set instead), but its docstring/comment still claims it's 'shared' with that check 'so the two can never drift.' This is now dead code with a misleading comment and should be removed or the comment updated.
## 🤖 AI Code Review (Claude Code)
The diff fixes a sitemap generation bug where publishability was determined only from a school's latest-year row, causing schools with valid historical results but null current-year rows (e.g. schools that stopped reporting) to be dropped from the sitemap. The fix vectorizes the check across all rows per URN and adds solid regression tests for both the fixed case and the still-must-exclude case. The change is small, well-tested, and correctness looks sound.
### 🟡 Minor
- **backend/app.py**: _has_publishable_data is no longer called anywhere (the per-school check in _school_sitemap_rows now uses the precomputed `publishable` set instead), but its docstring/comment still claims it's 'shared' with that check 'so the two can never drift.' This is now dead code with a misleading comment and should be removed or the comment updated.
tudor
merged commit 4a3928df9f into main2026-08-20 23:15:19 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Found while verifying the staging sitemap after #110 deployed.
_school_sitemap_rowstested only the latest year's row for publishable data, which quietly dropped every school with results in its history but a null row for the most recent year — schools that stopped reporting, or whose figures were suppressed for small-cohort disclosure.The Mallard Academy (150367) is the case that caught it:
Its detail page shows all four years of results. The sitemap omitted it.
Scale
I diffed the 22,979 sitemap URNs against the 25,185-school corpus, took the 2,206 excluded, and sampled 40 at random. Four had performance data — Mallard Academy, Catcote Academy, Downe Primary, Broadwood School. Extrapolating, roughly 220 real pages were being withheld from search.
The other 36 were correctly excluded: independent schools, FE colleges and post-16 institutions with no KS2/KS4 data and no Ofsted grade.
Fix
Publishable is now a property of the school, computed across every row, rather than of its latest row.
lastmodstill comes from the latest row so the most recent Ofsted date wins. The field list moved to a module constant shared with_has_publishable_dataso the two checks cannot drift apart.Two regression tests: one asserting Mallard's shape is listed, and one asserting the fix has not become "list everything" — a school with no results in any year and no Ofsted is still omitted.
Backend suite: 70 passed.
Expect the count to rise
Staging currently reports 22,983. After this, expect roughly 23,200.
Note on my earlier estimate
I predicted ~21,300 URLs for #110 and it came in at 22,983. That gap was my arithmetic, not this bug: I estimated exclusions using "no current-year data" when the actual criterion is "no data in any year and no Ofsted", which is far narrower. The bug above is a separate, real problem that the gap prompted me to go looking for.
🤖 Generated with Claude Code
https://claude.ai/code/session_015mWQnpye9F299NVRCCSRvj
🤖 AI Code Review (Claude Code)
The diff fixes a sitemap generation bug where publishability was determined only from a school's latest-year row, causing schools with valid historical results but null current-year rows (e.g. schools that stopped reporting) to be dropped from the sitemap. The fix vectorizes the check across all rows per URN and adds solid regression tests for both the fixed case and the still-must-exclude case. The change is small, well-tested, and correctness looks sound.
🟡 Minor
publishableset instead), but its docstring/comment still claims it's 'shared' with that check 'so the two can never drift.' This is now dead code with a misleading comment and should be removed or the comment updated.