Senior Data Engineer (active SC clearance required)Location: UK (Remote, occasional London travel) | Permanent: £60-85KRole OverviewHands-on Data Engineer working with public sector clients to design, build, and operate production data platforms. Lead technical architecture across cloud environments, mentor delivery teams, and embed analytics/AI capabilities in agile, client-facing projects.Key ResponsibilitiesTranslate complex business challenges into scalable data pipelines, models, and analytics solutionsBuild/operate production-grade pipelines using SQL/Python/Spark across lakehouses and event-driven systemsArchitect semantic models and data visualisations for business intelligence platformsDeploy cloud data/AI solutions (AWS/Azure) with robust CI/CD, testing, and security practicesIntegrate ML models, experimentation frameworks, and governance workflowsDeliver client-ready dashboards, insight narratives, and consulting materialsCollaborate across multidisciplinary teams in agile, iterative delivery cyclesTechnical EnvironmentCore: Advanced SQL/Python, PySpark/Pandas, Spark orchestration toolsCloud: AWS (Databricks/Redshift equivalents), Azure analytics platforms, Git/CI-CDArchitecture: Data lakehouses, event streaming, semantic modelling for BI toolsAI/ML: Model APIs, prompt engineering, experimentation frameworksGovernance: Data catalogues, lineage tracking, security/access controls, GDPREssential RequirementsExpert SQL/Python for data modelling, transformation, and automationProduction data engineering: Spark/PySpark, cloud platforms (AWS or Azure)Data architecture experience: lakehouses, event-driven systems, BI semantic modelsSoftware engineering: Git workflows, CI/CD pipelines, testing frameworksAnalytics: experimentation design, statistical analysis, insight storytellingConsulting mindset: stakeholder collaboration, client deliverables, agile deliveryACTIVE SC clearance requiredDesirablePublic sector/government project deliveryAdvanced BI semantic modelling (Power BI/Tableau equivalents)Operational intelligence platforms or data mesh implementationsData governance frameworks (lineage/catalogues/access management)