Backend Engineer

About the Role

We are looking for a Python Data Pipeline Engineer to transform existing Python-based calculation processes into a stable, production-ready automated data pipeline.

The role focuses on building a reliable end-to-end processing flow that includes:
data ingestion, normalization, validation, calculations, orchestration, monitoring, and automated result delivery — without changing the underlying business logic.

You will work on eliminating manual operational processes and creating a fully controllable, observable, and recoverable execution environment.

Responsibilities:

• Transform existing Python calculations into automated production-grade data pipelines.
• Build and maintain end-to-end processing flows:
  data ingestion → transformation → validation → calculations → result delivery.
• Integrate and process data from multiple internal and external sources.
• Implement data cleaning, mappings, normalization, and merging logic.
• Develop validation and data quality control mechanisms.
• Build incremental and idempotent processing logic to avoid duplication.
• Configure automated scheduled execution of pipelines.
• Implement logging, monitoring, alerting, and error handling.
• Design retry and recovery mechanisms for failed jobs.
• Split processing into stable and recoverable stages.
• Optimize reliability, maintainability, and operational transparency of pipelines.
• Collaborate with business and technical stakeholders to support stable production execution.

Requirements:

• Strong Python development skills.
• Experience with pandas and data processing workflows.
• Understanding of clean project structure and modular architecture.
• Experience with logging, exception handling, and production-ready code practices.
• Strong SQL knowledge including:
  
  • JOINs
  • aggregations
  • CTEs
  • window functions
• Understanding of incremental processing and deduplication approaches.
• Ability to work with large datasets and optimize queries.
• Experience building automated ETL / ELT pipelines.
• Understanding of idempotent processing principles.
• Experience implementing incremental data loading.
• Understanding of data quality validation and control mechanisms.
• Experience designing fault-tolerant and recoverable workflows.

Nice to Have

• Experience with Airflow, Prefect, Dagster, or similar orchestration tools.
• Experience with AWS / Azure / GCP.
• Experience with CI/CD pipelines.
• Understanding of monitoring and observability practices.

What We Offer

🚀 Opportunity to build production-grade data infrastructure from existing business processes.

🛠 High ownership and direct impact on system reliability, automation, and architecture decisions.

🌍 Flexible working environment with remote-friendly collaboration.

🤝 Engineering-focused culture with direct communication and minimal bureaucracy.

💰 Competitive compensation package and long-term growth opportunities.

Відгукнутись на вакансію