pipeline/scripts/diagnose_compare_gaps.py

@@ -190,3 +190,68 @@ if __name__ == "__main__":
 #     effectiveness of leadership and management'). Task 7 should map the 7
 #     report-card columns above (grade + date pairs, 6 of them, plus the
 #     safeguarding-effective flag) rather than inventing new column names.
+
+# TASK 6 VERIFICATION 2026-07-12: 2021/22 legacy KS2 school-level archive
+#
+# RESULT: BLOCKED at the source-data level. School-level KS2 attainment for
+# academic year 2021/22 was never published anywhere publicly by DfE -- not
+# in EES (confirmed by Task 1's finding (b) above), not in the legacy
+# "Compare School Performance" download wizard, and not as a standalone
+# performance-tables archive/ODS on assets.publishing.service.gov.uk. This
+# is a deliberate DfE decision, not a gap in our extraction logic.
+#
+# Confirming quote (Key stage 2 attainment 2021/22 release notes, via
+# https://explore-education-statistics.service.gov.uk/find-statistics/
+# key-stage-2-attainment/2021-22):
+#   "We will not publish key stage 2 data for academic year 2021/22 in
+#   performance tables (also known as Compare School and College
+#   Performance)." ... "The Department will, however, still produce the
+#   normal suite of key stage 2 accountability measures at school and
+#   multi-academy trust level and share these securely with primary
+#   schools, academy trusts and local authorities to inform school
+#   improvement discussions."
+# (i.e. school-level 202122 KS2 results exist internally at DfE but were
+# withheld from every public channel: performance tables/CSCP, EES, and by
+# extension the legacy DfE archives the current legacy_ks2_urls entries in
+# meltano.yml were sourced from.)
+#
+# What was tried:
+#  1. Direct download URL pattern from the task brief:
+#       https://www.compare-school-performance.service.gov.uk/download-data?download=true&regions=0&filters=KS2&fileformat=csv&year=2021-2022&meta=false
+#     -> HTTP 404, HTML error page (not a CSV/ZIP). Saved response inspected;
+#     confirmed 404 via response headers (`content-type: text/html`).
+#  2. Walked the actual multi-step download wizard at
+#     https://www.compare-school-performance.service.gov.uk/download-data
+#     with a browser User-Agent and a cookie jar, replicating the GET-based
+#     form steps: currentstep=year (downloadYear=2021-2022) -> currentstep=
+#     region (regiontype=all&la=0) -> currentstep=datatypes. On the final
+#     "datatypes" step, the checkbox list for 2021-2022 has NO "ks2" (or
+#     "ks2mats") option at all -- only ks4/ks4prov/ks4underlying/ks5* /
+#     pupil-destination/absence/census/mats checkboxes are present.
+#     Control check: repeating the same wizard walk for downloadYear=
+#     2018-2019, 2022-2023 and 2023-2024 shows a "ks2" (and "ks2mats")
+#     checkbox present in all three; downloadYear=2020-2021 (COVID-cancelled
+#     KS2 SATs year) also has NO ks2 checkbox, matching the pattern for a
+#     year where school-level KS2 genuinely isn't published. 2021-2022
+#     behaves identically to the cancelled 2020-2021 year, not like the
+#     normal 2018-2019/2022-2023/2023-2024 years.
+#  3. Web search for a standalone KS2 2022 performance-tables archive
+#     (e.g. "england_ks2final" for 2022) on assets.publishing.service.gov.uk
+#     found no such file; only unrelated 2022/2023-dated documents.
+#
+#  No ZIP was ever obtained -- /tmp/dfe-2021-2022-ks2.zip contains the 404
+#  HTML error page from attempt (1) above, not a real archive. It contains
+#  no england_ks2final.csv (there is no ZIP to look inside).
+#
+# Column-map check (brief's Step 1): NOT RUN -- there is no 2021/22
+# england_ks2final.csv to check headers against. This is moot until/unless
+# a non-public source (e.g. a manual/internal DfE extract) becomes
+# available; _LEGACY_KS2_COLUMN_MAP itself is unchanged and untested here.
+#
+# Recommendation: mark 202122 school-level KS2 as a genuine, permanent
+# source-data gap (not a backfill candidate) unless the project can obtain
+# the internal DfE extract DfE says it shared "securely with primary
+# schools, academy trusts and local authorities" -- that is not a route
+# available to this pipeline. Task 6's meltano.yml change (Step 2) and the
+# filebrowser upload should NOT proceed for 202122; there is nothing to
+# upload.