Data Engineer

We are looking for a skilled and passionate Data Engineer to design, build, and maintain scalable data infrastructure that powers our analytics, machine learning, and operational systems. You will work closely with software engineers, and business stakeholders to turn raw, complex data into reliable, production-ready pipelines and datasets. This is a hands-on, high-impact role for someone who thrives in a fast-paced environment and takes pride in data quality, reliability, and engineering excellence.

Essential Duties and Responsibilities:

Pipeline Design & Data Ingestion

• Design and build robust ETL/ELT pipelines for both batch and real-time processing, ensuring high throughput and fault tolerance.

• Ingest and process high-frequency data from diverse sources including REST/GraphQL APIs, relational and NoSQL databases, IoT/sensor streams, and event queues.

• Transform raw, messy, and heterogeneous data into clean, validated, and production-ready datasets for downstream consumers.

• Architect and maintain data lake and data warehouse structures, including partitioning strategies, schema evolution, and data versioning.

• Evaluate, select, and integrate appropriate data tools and frameworks based on project requirements and scale.

ML & Analytics Support

• Support and deploy machine learning-ready pipelines, including feature engineering, data preprocessing, and preparation of model inputs and training datasets.

• Collaborate with data scientists to productionize ML workflows, ensuring reproducibility and scalability of pipelines.

• Build and maintain feature stores and data marts that support analytical dashboards, reporting, and model serving.

• Develop and maintain data catalogues and metadata management systems to improve data discoverability and lineage.

Data Quality & Reliability

• Implement comprehensive data quality checks, validation frameworks, and alerting mechanisms across all pipelines.

• Monitor pipeline health, throughput, and latency; proactively diagnose and resolve data issues before they impact downstream systems.

• Design for scalability and reliability in production, applying best practices for idempotency, retry logic, and graceful failure handling.

• Conduct root cause analysis on data incidents and implement corrective and preventive measures.

Infrastructure & Orchestration

• Deploy and manage pipeline orchestration using tools such as Apache Airflow, Prefect, or Dagster.

• Work with cloud platforms (AWS, GCP, or Azure) to provision and manage data infrastructure including storage, compute, and streaming services.

• Implement CI/CD practices for data pipelines, including automated testing, version control, and deployment pipelines.

• Collaborate with DevOps and platform teams to containerize and deploy data workloads using Docker and Kubernetes.

Documentation & Collaboration

• Maintain thorough documentation of pipeline architectures, data models, data dictionaries, and operational runbooks.

• Participate in code reviews, architecture discussions, and cross-functional planning sessions.

• Mentor junior data engineers and contribute to internal knowledge sharing on data engineering best practices.

• Communicate technical concepts clearly to non-technical stakeholders, translating business requirements into engineering solutions.

Required Experience

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, Statistics, or a related discipline.

• 3–5 years of hands-on experience building and maintaining data pipelines in production environments.

• Proven track record working with imperfect, noisy, and real-world data at scale.

• Experience with time-series or IoT data ingestion and processing is highly valued.

• Experience in a fintech, banking, SaaS, or high-availability production environment is strongly preferred.

• Exposure to agile development practices and cross-functional team collaboration.

Required Skills

Core Engineering

• Strong proficiency in Python for data engineering tasks including data manipulation, pipeline scripting, and automation.

• Advanced SQL skills: query optimisation, window functions, CTEs, indexing strategies, and working with large datasets.

• Solid understanding of data modelling concepts including star/snowflake schemas, dimensional modelling, and normalisation.

• Experience with data serialisation formats: JSON, Avro, Parquet, ORC, and Protobuf.

Pipeline & Streaming Tools

• Proven experience with orchestration tools such as Apache Airflow, Prefect, or Dagster for scheduling and dependency management.

• Exposure to streaming platforms such as Apache Kafka, Apache Flink, or AWS Kinesis for real-time data processing.

• Familiarity with ETL/ELT frameworks such as dbt, Spark, or Beam.

• Experience with change data capture (CDC) patterns and tools (e.g., Debezium) is an advantage.

Cloud & Infrastructure

• Hands-on experience with at least one major cloud platform: Amazon Web Services (S3, Glue, Redshift, Lambda), Google Cloud Platform (BigQuery, Dataflow, Pub/Sub), or Microsoft Azure (Data Factory, Synapse, Event Hubs).

• Familiarity with infrastructure-as-code tools such as Terraform or CloudFormation.

• Working knowledge of containerisation with Docker; experience with Kubernetes is a plus.

• Understanding of data lake architectures (Delta Lake, Apache Iceberg, or Apache Hudi).

Machine Learning Integration

• Experience with machine learning workflows including feature engineering, data preprocessing pipelines, and integration with ML frameworks (scikit-learn, XGBoost, or similar).

• Familiarity with MLflow, Kubeflow, or similar ML lifecycle management tools is an advantage. • Ability to work closely with data scientists to translate model requirements into scalable, production-grade data pipelines.

Soft Skills

• Strong analytical mindset with the ability to diagnose complex data issues in distributed systems under pressure.

• Excellent written and verbal communication in English; ability to produce clear technical documentation and present findings to stakeholders.

• Collaborative and proactive team player with experience working across data science, engineering, and product functions.

• Detail-oriented with a strong sense of ownership over data quality and pipeline reliability.

• Comfortable working with ambiguity and rapidly changing requirements in a fast-paced environment.

Working Environment

• Join a young and dynamic team in an international, professional, English-speaking environment.

• Work with cutting-edge data technologies and modern cloud platforms.

• Collaborate with data scientists, engineers, and product teams on high-impact, large-scale projects.

• Open culture where people are valued, trusted, and empowered to do great things