Data Engineer

We're looking for a mid-to-senior Python data engineer to join the data team of a fast-growing, privacy-focused fintech (Series A). The company is backed by a global credit bureau and a major retailer, and has teams across South Africa, the UK, the US, the UAE and Brazil.

Its platform lets banks and lenders work with retailer and telco data without raw personal data ever changing hands. The main goal is to help people with no credit history get access to credit.

Your purpose in this role is to lead the move of legacy SQL Server / Azure Synapse pipelines onto open-source Apache Spark and Delta Lake. You'll write production software, not notebook scripts.

Logistics

• Location: fully remote, Europe 
• Working hours: South Africa time zone (UTC+2, the same as Kyiv and one hour ahead of Warsaw). Senior leadership is on US Eastern time.
• Employment: full-time. Contract type (B2B or permanent) and salary band to be confirmed.

• Level: Mid-Senior, about 4–8 years of experience

   

What you'll do

• Rebuild legacy T-SQL stored-procedure logic as typed, tested PySpark pipelines
• Prove that the new pipelines match the legacy outputs
• Build FastAPI services that run and report on pipeline jobs
• Add provider-neutral LLM features (local and hosted models) to the data workflows
• Work in a spec-first codebase: design docs and ADRs, with tests and documentation shipped alongside every change

Core stack

Python, PySpark, Spark SQL, Delta Lake, FastAPI, pytest, Docker, T-SQL, LLM integration

Requirements

• Python as your primary production language: modern typed Python (3.10+, ideally 3.12), with type hints that pass pyright or mypy, and Pydantic models
• 2+ years running production Apache Spark / PySpark pipelines, using the DataFrame API and Spark SQL, partitioning and performance tuning, and with a solid grasp of the driver/executor architecture
• Maintainable, tested code (pytest), with tooling such as uv or Poetry and a CLI framework such as Click
• Based in Europe or South Africa and happy to work UTC+2 hours

Strong advantages

• Experience migrating SQL Server / Synapse / stored-procedure logic to Spark and proving output parity with the legacy system
• Open-source or self-managed Spark (Spark on Kubernetes, EMR, Dataproc, Spark Connect), not only Databricks notebooks; Hive Metastore, Trino, Apache Ranger
• FastAPI services with request validation, HMAC-signed auth, idempotency and job-status semantics
• LLM features: prompt construction, structured-output validation, local models (Ollama, vLLM, llama.cpp), evaluation and guardrails
• Privacy engineering: PII tokenisation and hashing, k-anonymity, GDPR/POPIA, multi-tenant isolation
• Fintech, banking, credit-bureau, adtech-measurement or retail-data background
• Observability: structlog, OpenTelemetry, OpenLineage
• Azure (ADLS, Synapse) and a Databricks Certified Spark Developer credential