Data Engineer

Senior Databricks Engineer
The role requires significant daily overlap with the US team during US Eastern Time (EST) working hours. Candidates must be able to work within the required US EST schedule.

TechBar is a software services company working with clients across the US and Europe. We are investor-backed, growing steadily, and known for putting senior, thoughtful engineers in front of our clients.

We are looking for an experienced Senior Databricks Engineer to join an enterprise-level data modernization initiative for a US-based client.

The project focuses on modernizing a large-scale enterprise data platform as part of a broader cloud transformation journey. You will work with the Databricks Lakehouse Platform to build scalable data solutions supporting advanced analytics, AI/ML initiatives, and enterprise-wide data products.

What you’ll do

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Spark SQL
• Build robust ETL/ELT processes for large-scale structured and unstructured data
• Implement Medallion Architecture (Bronze, Silver, Gold) and Delta Lake solutions
• Develop batch and near real-time data processing solutions using Spark Streaming, Kafka, and Delta Live Tables (DLT)
• Integrate data from enterprise sources, including SAP, Teradata, Mainframe, DB2, SQL Server, Oracle, Tibco, Kafka, and REST APIs
• Work across both Microsoft Azure and AWS environments as part of the client’s cloud data modernization program
• Develop and maintain Databricks notebooks, workflows, jobs, and clusters
• Implement Unity Catalog, access controls, data governance, and security standards
• Support deployments and environment promotion across Dev, QA, UAT, and Production
• Optimize workloads for performance, scalability, reliability, and cost efficiency
• Implement data validation, reconciliation, monitoring, and automated data quality frameworks
• Contribute to data lineage, metadata management, compliance, and enterprise data governance
• Build reusable frameworks and automation for data ingestion, orchestration, monitoring, and quality management
• Work closely with Data Scientists, BI developers, architects, and business stakeholders to translate requirements into scalable technical solutions
• Participate in architecture and solution design discussions
• Mentor junior engineers and contribute to engineering best practices
• Participate in Agile ceremonies and collaborate with distributed engineering teams

Must-have requirements

• 7+ years of professional experience in Data Engineering
• Strong hands-on experience with Databricks
• Databricks certification is mandatory: Databricks Certified Data Engineer Associate or Professional
• Strong practical experience with both Microsoft Azure and AWS. Experience in only one cloud platform is not sufficient.
• Solid hands-on experience with:
  
  • Databricks Workflows / Jobs
  • Delta Lake
  • Delta Live Tables (DLT)
  • Unity Catalog
  • Databricks SQL
• Strong programming skills in Python, PySpark, Spark SQL, and SQL
• Strong understanding of ETL/ELT, data modeling, data warehousing, and distributed data processing
• Experience with Apache Kafka, Spark Streaming, and event-driven architectures
• Hands-on experience with Azure, particularly:
  
  • Azure Databricks
  • Azure Data Factory
  • ADLS Gen2
  • Azure Key Vault
• Hands-on experience with AWS, particularly:
  
  • Amazon S3
  • IAM
• Experience working with enterprise databases such as:
  
  • Teradata
  • SQL Server
  • Oracle
  • DB2
• Experience with Git and CI/CD, preferably Azure DevOps or GitHub Actions
• Strong communication skills and the ability to work effectively with technical and non-technical stakeholders
• Ability to work independently, take ownership, and contribute to architectural decisions

Nice to have

• Azure Data Engineer Associate certification
• Experience working on large-scale enterprise data modernization programs
• Experience migrating workloads from Teradata or other legacy platforms to Databricks
• Experience with SAP Datasphere, SAP BW, or SAP Business Data Cloud (BDC)
• Experience with Databricks Asset Bundles
• Experience with BigQuery or Snowflake
• Exposure to AI/ML workloads and feature engineering on Databricks
• Experience working with highly governed or regulated enterprise data environments

What makes this project interesting

This is not a project where you simply maintain existing pipelines.

You will be involved in a broader enterprise data transformation, helping move legacy data workloads toward a modern cloud-based architecture. The platform combines Databricks, Delta Lake, Azure, AWS, streaming technologies, enterprise databases, and modern data governance practices.

You will have the opportunity to work on problems involving large data volumes, complex integrations, real-time processing, data quality, governance, and the enablement of analytics and AI/ML use cases.

Working model

• Remote
• Collaboration with a US-based team
• US Eastern Time (EST) working hours / strong overlap is required
• Long-term project
• English: Upper-Intermediate / Advanced
• Senior-level ownership and direct communication with the client

Відгукнутись на вакансію