Job description for Data Engineer at Skyshi Digital Indonesia
Data Engineer
Key Responsibilities
- Design and develop microservices for ingesting data from multiple sources, both internal and external.
- Build and manage end-to-end data pipelines, covering data extraction, cleaning, transformation, and loading (ETL/ELT).
- Implement and maintain data quality monitoring processes to ensure data used by AI systems is accurate, consistent, and reliable.
- Develop data pipelines that serve as input for AI systems, enabling AI-generated outputs to be integrated into Dialog application features.
- Assess business requirements related to data warehousing, including evaluating whether the existing PostgreSQL infrastructure remains sufficient or whether migration to a dedicated data warehouse (e.g., BigQuery, Snowflake, Amazon Redshift, or similar platforms) is required.
- Delegate tasks to other team members when necessary while maintaining full ownership of the overall data aggregation roadmap, architecture, and quality.
- Collaborate closely with AI and Product teams to ensure the data architecture and pipelines align with AI model requirements and product feature objectives.
Qualifications
- Proven experience as a Data Engineer (this is not an entry-level role; candidates are expected to have hands-on experience designing and building data pipelines independently).
- Strong proficiency in Python for data processing and pipeline development.
- Hands-on experience with at least one of the following technologies: SQL, Apache Spark, Apache Airflow, Apache Kafka, or equivalent tools such as dbt, Dagster, Apache Flink, or similar.
- Solid understanding of data warehousing concepts, with the ability to analyze business requirements and provide technical recommendations, including strategic decisions around warehouse architecture and migration.
- Experience implementing and monitoring data quality checks, including data validation, anomaly detection, and other data integrity practices.
- Ability to work independently while also leading initiatives and delegating tasks to other team members when required.
- Good command of English, both written and spoken.
Nice to have: Experience building and maintaining data pipelines that serve as input for AI/ML systems
