Data Engineer
Role Overview
We are looking for a Senior Data Engineer to take a defined platform architecture from design to a running, production-grade data pipeline. You will own the path from raw client data to a unified data model and the analytics layer that machine-learning models read from, and the path that moves those models into production.
The architecture, data dictionary and schema are being defined by a hands-on CTO. Your job is to refine and harden that design, implement it, and make the technical calls along the way — there is no larger engineering team to defer to, so independence and clear written communication matter as much as the engineering.
Project Description
The client is a US healthcare-data company building a platform for pharmacy benefit managers. The platform takes historical patient, condition and prescription data and produces drug and treatment recommendations for new patients. Today this runs as a manual, proof-of-concept process — data prepared by hand, models deployed by hand — and the goal is a true SaaS platform with an automated pipeline behind it, with a first MVP targeted for the end of 2026.
The target architecture is a conventional layered platform: ingest client data into a raw zone, clean and standardise it through ETL, load it into a unified data model, and expose an analytics layer for modelling. The stack is Python on AWS, with Docker and Kubernetes as the runtime, and a strong preference for open-source tooling over managed proprietary services.
Key Responsibilities
- Turn the target architecture into a working data platform: raw zone, ETL, unified data model and analytics layer.
- Review and harden the platform design before building, and document the decisions.
- Build the ingestion and transformation pipelines that take client data dumps through to the unified data model.
- Implement the database schema from the data dictionary and keep the two aligned as the model evolves.
- Establish the model deployment path — versioning, packaging, promotion to production and rollback — so models are no longer pushed by hand.
- Introduce standard data formats, validation and quality checks where none exist today.
- Work with the infrastructure engineer to run pipelines and model services on Kubernetes in AWS.
- Extract requirements directly from client stakeholders, write them down, and get them confirmed.
- Evaluate the existing proof-of-concept code and document what is reused and what is rebuilt.
- Propose an approach and defend it rather than waiting for a fully formed requirement.
- Keep cost in view when choosing services and designing storage and compute.
Required Qualifications
- Senior-level commercial data engineering experience, including ownership of a production data pipeline.
- Strong Python.
- Hands-on experience designing and implementing ETL pipelines and data models from raw source data to a curated layer.
- Solid SQL and relational database schema design.
- Experience building on AWS with open-source tooling.
- Ability to work independently in an environment without an established engineering process, and to set that process where it is missing.
- Direct, clear communication with non-technical stakeholders, including gathering and writing requirements.
- English at a level that supports direct client meetings without an intermediary.
Preferred Qualifications
- Experience with Docker and Kubernetes as a deployment target for data and model workloads.
- Experience with an orchestration tool such as Airflow, Dagster or Prefect.
- Experience with a data lakehouse layout (raw / cleaned / curated zones) and columnar formats.
- Healthcare, pharmacy or insurance data experience, including handling of sensitive patient data.
- Experience setting up CI/CD for data and model pipelines from scratch.
- Experience taking over a proof of concept and rebuilding it for production.
Your First 90 Days
- First 30 days — You have reviewed the architecture, data dictionary and existing proof of concept, documented the hardened design and the open questions, and agreed the MLOps toolchain with the CTO.
- By 60 days — Client data flows through an automated pipeline from raw zone to the unified data model in a non-production environment, with validation in place, and the first model is deployed through the new deployment path rather than by hand.
- By 90 days — The pipeline and model deployment run in production against the availability target, requirements for the MVP are documented and confirmed with the client, and you are the person the client goes to for data-platform decisions.