Job description for Data Engineer at Bvarta
Responsibilities :
- Design, develop, and maintain scalable data pipelines.
- Build data ingestion processes using Change Data Capture and ETL / ELT jobs.
- Build data pipeline to cleanse and transform raw data into clean data in the targeted format to be stored in Data lake.
- Build and manage jobs to preprocess data into an optimized and highly structured data format into Data mart.
- Write clean, efficient, and well-tested code for data processing and transformation.
- Optimize data storage and retrieval processes to ensure high performance and scalability.
- Collaborate with cross-functional teams to understand data requirements and deliver solutions.
- Participate in code reviews and contribute to technical discussions.
- Monitor and optimize data pipeline performance.
- Document technical specifications and data pipeline documentation.
- Troubleshoot and resolve complex technical issues related to data processing.
Preferred Qualifications:
- Bachelor’s degree in Computer Science, Software Engineering, or equivalent practical experience.
- 3+ years of experience in data engineering.
- Strong proficiency in building data pipelines using Debezium, Airflow, Spark, Hive Metastore, Trino (Presto).
- Solid understanding of data lake and data mart architectures.
- Experience with ETL processes and data transformation techniques.
- Proficiency in SQL and experience with relational and NoSQL databases.
- Strong knowledge of software design patterns and principles.
- Experience with containerization (Docker) and version control (Git).
- Proven track record of writing clean, maintainable code.
- Proficiency in unit testing and test-driven development.
- Excellent problem-solving, communication, and collaboration abilities.
- Experience in event-driven architecture is a plus.
