Job description for Data Engineer at Emtrade
Responsibilities:
- Create, maintain and improve optimal data pipeline architecture
- Compiling large and complex datasets to fulfill business requirements
- Strong analytic skills related to working with unstructured datasets.
- Build processes supporting data transformation, data structures, metadata, dependency and workload management.
- A successful history of manipulating, processing and extracting value from large disconnected datasets.
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and GCP ‘big data’ technologies.
- Build analytics tools that utilize the data pipeline to provide actionable business insights.
- Work with stakeholders including the Senior Management, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs
Requirements:
- Bachelor or equivalent experience in a quantitative field (Computer Science)
- Minimum 1+ years of relevant work experience
- Experience in building and optimizing ‘Big Data’ data pipelines, architectures and datasets (including Batch and Streaming pipeline)
- Experience with big data tools: Spark, Kafka, Beam, Flink, Storm, etc.
- Experience with relational SQL or NoSQL databases, including Postgres or Cassandra.
- Experience with cloud services such as GCP or AWS and Linux environment and storage architecture.
- Experience with stream-processing systems: Flink, Spark-Streaming, Storm Bolt, etc.
- Experience with multiple programming languages (preferably at least one of Python, Go, Java)
- Have experience with R and Scala is a plus
- Willingness to learn new languages and technologies.
