Job Requirements
5 - 10 years of experience
This job post is managed by
jit
Last active 2 years ago
Skills
Job description for Senior Big Data Engineer at One North Consulting Pte Ltd
One North Consulting is currently hiring Singapore Citizens / Singapore Permanent Residents for Senior Big Data Engineer position with about 7 - 10 years of experience as per details given below.
Role:-
As Senior Big Data Engineer, you will focus on managing the Hadoop cluster, Implementing data ingestion framework & designing data models.
Your primary role will be to implement data lake & transform the data for business use.
• Experience with at least one Cloud Infra provider (Azure/AWS)
• Experience in building data pipelines using batch processing with Apache Spark (Spark SQL, DataSet / Dataframe API) or Hive query language (HQL)
• Knowledge of Big data ETL processing tools
• Experience with Hive and Hadoop file formats (Avro / Parquet / ORC)
• Basic knowledge of scripting (shell / bash)
• Experience with CI CD tools such as Jenkins, JIRA, Bitbucket, Artifactory, Bamboo and Azure Dev-ops.
• understanding of DevOps practices using Git version control
• Debug, fine tune and optimize large scale data processing jobs
• Deliver big data solutions based on premise Hadoop or cloud-based systems like AWS.
• Manage Hadoop cluster, participate in scale out planning & implementation.
• Design ingestion layer for structured & unstructured data (text, voice, xml etc)
• Implement specific data model for business & analytics use.
• Deliver ELT solution including data extraction, transformation, cleansing, data integration and data management.
• Augment with new sources of data including internal/external untapped data.
• Contribute to the establishment and maintenance of cloud computing platform and big data services.
• Provide support for analytics tools & environment like RServer etc & debug performance issues.
COMPETENCIES
• Databases: RDBMS, SQL programming
• ETL and Data Integration Tools: Azure Data Factory (ADF), Microsoft SQL Server Integration Service(SSIS), SAS Data Integration
• Big Data: Hadoop (Hortonworks), Hive, Spark, Sqoop, etc
• Programming and Scripting: Linux/Unix Shell Scripting, Java, Scala, Hive QL
• BI/Dashboarding: SAS Visual Analytics, Qliksense
Working Experience:
• 5~8 years in data engineering and modelling.
• Hands-on in managing data mapping, data quality and integrity, performance in data processing.
• Experience in Agile software development.