tudor/school_compare
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(pipeline): add Meltano + dbt + Airflow ELT pipeline scaffold
All checks were successful
Build and Push Docker Images / Build Backend (FastAPI) (push) Successful in 35s
Details
Build and Push Docker Images / Build Frontend (Next.js) (push) Successful in 1m9s
Details
Build and Push Docker Images / Build Integrator (push) Successful in 56s
Details
Build and Push Docker Images / Build Kestra Init (push) Successful in 32s
Details
Build and Push Docker Images / Trigger Portainer Update (push) Successful in 1s
Details

Browse Source 
Replaces the hand-rolled integrator with a production-grade ELT pipeline
using Meltano (Singer taps), dbt Core (medallion architecture), and
Apache Airflow (orchestration). Adds Typesense for search and PostGIS
for geospatial queries.

- 6 custom Singer taps (GIAS, EES, Ofsted, Parent View, FBIT, IDACI)
- dbt project: 12 staging, 5 intermediate, 12 mart models
- 3 Airflow DAGs (daily/monthly/annual schedules)
- Typesense sync + batch geocoding scripts
- docker-compose: add Airflow, Typesense; upgrade to PostGIS
- Portainer stack definition matching live deployment topology

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Tudor
2026-03-26 08:37:53 +00:00
parent 8aca0a7a53
commit 8f02b5125e
65 changed files with 2822 additions and 72 deletions
Download Patch File Download Diff File Expand all files Collapse all files

66
pipeline/transform/seeds/la_code_names.csv Normal file
View File

@@ -0,0 +1,66 @@
la_code,la_name
201,City of London
202,Camden
203,Greenwich
204,Hackney
205,Hammersmith and Fulham
206,Islington
207,Kensington and Chelsea
208,Lambeth
209,Lewisham
210,Merton
211,Newham
212,Tower Hamlets
213,Wandsworth
214,Westminster
301,Barking and Dagenham
302,Barnet
303,Bexley
304,Brent
305,Bromley
306,Croydon
307,Ealing
308,Enfield
309,Haringey
310,Harrow
311,Havering
312,Hillingdon
313,Hounslow
314,Kingston upon Thames
315,Redbridge
316,Richmond upon Thames
317,Sutton
318,Waltham Forest
319,City of London
320,City of London
330,Birmingham
331,Coventry
332,Dudley
333,Sandwell
334,Solihull
335,Walsall
336,Wolverhampton
340,Knowsley
341,Liverpool
342,St Helens
343,Sefton
344,Wirral
350,Bolton
351,Bury
352,Manchester
353,Oldham
354,Rochdale
355,Salford
356,Stockport
357,Tameside
358,Trafford
359,Wigan
370,Barnsley
371,Doncaster
372,Rotherham
373,Sheffield
380,Bradford
381,Calderdale
382,Kirklees
383,Leeds
384,Wakefield
1 la_code la_name
2 201 City of London
3 202 Camden
4 203 Greenwich
5 204 Hackney
6 205 Hammersmith and Fulham
7 206 Islington
8 207 Kensington and Chelsea
9 208 Lambeth
10 209 Lewisham
11 210 Merton
12 211 Newham
13 212 Tower Hamlets
14 213 Wandsworth
15 214 Westminster
16 301 Barking and Dagenham
17 302 Barnet
18 303 Bexley
19 304 Brent
20 305 Bromley
21 306 Croydon
22 307 Ealing
23 308 Enfield
24 309 Haringey
25 310 Harrow
26 311 Havering
27 312 Hillingdon
28 313 Hounslow
29 314 Kingston upon Thames
30 315 Redbridge
31 316 Richmond upon Thames
32 317 Sutton
33 318 Waltham Forest
34 319 City of London
35 320 City of London
36 330 Birmingham
37 331 Coventry
38 332 Dudley
39 333 Sandwell
40 334 Solihull
41 335 Walsall
42 336 Wolverhampton
43 340 Knowsley
44 341 Liverpool
45 342 St Helens
46 343 Sefton
47 344 Wirral
48 350 Bolton
49 351 Bury
50 352 Manchester
51 353 Oldham
52 354 Rochdale
53 355 Salford
54 356 Stockport
55 357 Tameside
56 358 Trafford
57 359 Wigan
58 370 Barnsley
59 371 Doncaster
60 372 Rotherham
61 373 Sheffield
62 380 Bradford
63 381 Calderdale
64 382 Kirklees
65 383 Leeds
66 384 Wakefield

30
pipeline/transform/seeds/school_type_codes.csv Normal file
View File

@@ -0,0 +1,30 @@
type_code,type_name,type_group
1,Community school,Maintained
2,Voluntary aided school,Maintained
3,Voluntary controlled school,Maintained
5,Foundation school,Maintained
6,City technology college,Independent
7,Community special school,Special
8,Non-maintained special school,Special
10,Other independent school,Independent
11,Other independent special school,Independent
12,Foundation special school,Special
14,Pupil referral unit,PRU
24,Secure unit,Other
25,Offshore school,Other
26,Service children's education,Other
28,Academy sponsor led,Academy
33,Academy special sponsor led,Academy
34,Academy converter,Academy
35,Free schools,Academy
36,Free schools special,Academy
37,British schools overseas,Other
38,Free schools - alternative provision,Academy
39,Free schools - 16-19,Academy
40,University technical college,Academy
41,Studio school,Academy
42,Academy alternative provision converter,Academy
43,Academy alternative provision sponsor led,Academy
44,Academy special converter,Academy
46,Academy 16-19 converter,Academy
47,Academy 16-19 sponsor led,Academy
1 type_code type_name type_group
2 1 Community school Maintained
3 2 Voluntary aided school Maintained
4 3 Voluntary controlled school Maintained
5 5 Foundation school Maintained
6 6 City technology college Independent
7 7 Community special school Special
8 8 Non-maintained special school Special
9 10 Other independent school Independent
10 11 Other independent special school Independent
11 12 Foundation special school Special
12 14 Pupil referral unit PRU
13 24 Secure unit Other
14 25 Offshore school Other
15 26 Service children's education Other
16 28 Academy sponsor led Academy
17 33 Academy special sponsor led Academy
18 34 Academy converter Academy
19 35 Free schools Academy
20 36 Free schools special Academy
21 37 British schools overseas Other
22 38 Free schools - alternative provision Academy
23 39 Free schools - 16-19 Academy
24 40 University technical college Academy
25 41 Studio school Academy
26 42 Academy alternative provision converter Academy
27 43 Academy alternative provision sponsor led Academy
28 44 Academy special converter Academy
29 46 Academy 16-19 converter Academy
30 47 Academy 16-19 sponsor led Academy