Job description for Data Engineer at Insignia
About the Role
You’ll be part of a hands-on consulting team building data platforms that clients actually run in production. This isn’t a support role, you’ll own workstreams, collaborate with client engineers, and be expected to ship. You’ll work across the full data lifecycle: ingestion, modeling, orchestration, and serving, on real enterprise-scale data.
Responsibilities :
▪️ Design, develop, and maintain end-to-end data pipelines for large-scale processing on cloud platforms (primarily Databricks on AWS)
▪️ Build and optimize pipelines using Databricks — Delta Lake, Auto Loader, Workflows, and SQL Warehouses
▪️ Implement medallion architecture (bronze/silver/gold) with proper data modeling practices
▪️ Collaborate with client stakeholders and internal team to ensure data quality and deliver actionable insights via BI tools
▪️ Monitor and troubleshoot data workflows in production, resolving issues in a structured and timely manner
▪️ Write clean, maintainable PySpark and SQL code that other engineers can read and extend
▪️ Support technical scoping and solution design during pre-sales or project kickoff phases
Requirements :
Must Have [ ✔️ ]
✔️ Min. 2 years of experience as a Data Engineer, preferably in a consulting or multi-client environment
✔️ Strong proficiency in Python (PySpark, Pandas) and SQL — CTAS, Window Functions, CTEs
✔️ Hands-on Databricks experience: notebooks, Delta Lake, Workflows, and SQL Warehouses
✔️ Experience with AWS data services: S3, Glue, RDS/Aurora, or Redshift
✔️ Solid understanding of data modeling — star schema, medallion architecture, normalization
✔️ Version control with GitHub or GitLab in a team environment
✔️ Strong communication skills — ability to explain technical decisions to non-technical stakeholders
Nice to Have [ ➕ ]
➕ Databricks Unity Catalog, Auto Loader, Delta Sharing, or lakehouse concepts
➕ Snowflake experience — schema design, data sharing, cost optimization
SAP data extraction via OData, PyRFC, or similar connectors
➕ BI tools: Power BI, QuickSight, Tableau, or Databricks AI/BI
➕ Databricks Certified Data Engineer Associate or above
API-based data ingestion patterns
➕ Hands-on experience with MySQL, PostgreSQL, or MariaDB
