Chi tiết công việc Data Engineer (Data Lakehouse architectures) tại SotaTek
JOB DESCRIPTION:
- Design, develop, and maintain data pipelines to support business operations.
- Process and transform data to meet both functional and non-functional business requirements.
- Identify, design, and implement process improvements to automate manual tasks and optimize data delivery.
- Design and develop data warehouse (DW) solutions, including data modeling and ETL processes.
- Develop real-time data ingestion and streaming solutions.
- Design and implement cloud-based data platforms and architectures on AWS or GCP.
- Design and develop container-based services using Kubernetes (Amazon EKS/Google GKE).
- Design, develop, and maintain MLOps platforms and machine learning deployment pipelines..
JOB REQUIREMENTS:
- No spoken English proficiency required.
- 3–5 years of hands-on experience designing and building large-scale data pipelines (processing over 100GB/day) in enterprise environments.
- Proven experience with AWS Data Services, including Amazon S3, AWS Glue (Crawlers & Jobs), Amazon EMR, Amazon MSK (Kafka), AWS Step Functions, AWS Lake Formation, and Amazon Redshift Spectrum.
- Strong expertise in Apache Spark (PySpark or Scala) for developing batch and streaming ETL/ELT workloads on Amazon EMR or AWS Glue.
- Solid understanding of and hands-on experience implementing Data Lakehouse architectures using open table formats such as Apache Iceberg or Delta Lake, including schema evolution, time travel, and ACID transactions.
- Strong Python programming skills with experience developing AWS Glue Jobs, AWS Lambda functions, pipeline automation, and unit testing for data transformations.
- Experience building real-time streaming pipelines using Apache Kafka or Amazon MSK, with a solid understanding of Kafka Consumer Groups, offset management, and Schema Registry.
- Hands-on experience designing and implementing data pipeline orchestration using Apache Airflow or AWS Step Functions, including DAG design, dependency management, retry mechanisms, and SLA monitoring.
- Experience integrating data from multiple sources, including relational databases (Oracle, MySQL, PostgreSQL), NoSQL databases (MongoDB, Redis), SaaS APIs (REST/SOAP), and SFTP-based file ingestion.
- Good understanding of Data Quality concepts, with experience implementing data quality checks (completeness, uniqueness, timeliness) using Oracle EDQ, Great Expectations, or Deequ.
- Familiarity with Data Governance practices, including metadata management, data lineage, and data tagging, with hands-on experience using Apache Atlas or AWS Glue Data Catalog.
- Experience with Oracle GoldenGate CDC is a strong advantage.
- Experience working in Agile/Scrum environments and collaborating closely with Data Architects, Cloud Engineers, DBAs, and Analytics Engineers.
- Proactive in creating and maintaining technical documentation.
Note: Open to level Fresher. Must have: Good at English
COMPENSATION & BENEFITS:
🎯Flexible working regime and health care:
- Flexible timekeeping (from 8:00 - 9:00 to 17:30 - 18:30)
- Minimum 14 paid leaves per annum for all employees after probation
- 01-day remote work per month
- A flexitime allowance of 90-180 minutes per month for employees
- 01 hour paid leave per day for women having children under 12 months
- Social insurance, health insurance, unemployment insurance and Bao Viet care insurance
🎯Transparent and fair benefits:
- Saturday & Sunday OFF, Overtime pay is 150%, 200%, 300% as per labor law;
- Work performance review 2 times/ year (in April and October)
- Bonus Policy: 13th-month salary and Performance-based bonus
- Men’s Day, Women’s Day, Children’s Day, Mid-Autumn Festival and other benefits under the provisions of the company
🎯Dynamic environment and open culture:
- Year-end party, sports day, yearly company trip and quarterly team building,...with a generous budget
- Socialize with colleagues through monthly Happy Hour
- Monthly allowance when joining clubs: Soccer, Swimming, Yoga, Music,...
- Nice & modern working space with young, dynamic & friendly colleagues and free coffee, tea, drinks,...
- Flat, open and sharing culture with friendly management team; outsourcing company with product mindset
🎯Strong learning culture:
- Free training courses for technical and soft skills (presentation skills, communication skills, foreign language courses,...)
- Account to log in to our online learning system, which contains thousands of valuable lectures (LMS)
- Participate in workshops, seminars, tech talk,... with sharing from experts inside and outside the company
- Working opportunities with technical gurus who built and operated world-class applications with millions of users.
