Job description for Data Engineer at Insignia
You’ll be part of a hands-on consulting team building data platforms that clients actually run in production. This isn’t a support role — you’ll own workstreams, collaborate with client engineers, and be expected to ship. You’ll work across the full data lifecycle: ingestion, modeling, orchestration, and serving — on real enterprise-scale data.
Responsibilities
Design, develop, and maintain end-to-end data pipelines for large-scale processing on cloud platforms (primarily Databricks on AWS)
Build and optimize pipelines using Databricks — Delta Lake, Auto Loader, Workflows, and SQL Warehouses
Implement medallion architecture (bronze/silver/gold) with proper data modeling practices
Collaborate with client stakeholders and internal team to ensure data quality and deliver actionable insights via BI tools
Monitor and troubleshoot data workflows in production, resolving issues in a structured and timely manner
Write clean, maintainable PySpark and SQL code that other engineers can read and extend
Support technical scoping and solution design during pre-sales or project kickoff phases
Requirements
Must Have
Min. 2 years of experience as a Data Engineer, preferably in a consulting or multi-client environment
Strong proficiency in Python (PySpark, Pandas) and SQL — CTAS, Window Functions, CTEs
Hands-on Databricks experience: notebooks, Delta Lake, Workflows, and SQL Warehouses
Experience with AWS data services: S3, Glue, RDS/Aurora, or Redshift
Solid understanding of data modeling — star schema, medallion architecture, normalization
Version control with GitHub or GitLab in a team environment
Strong communication skills — ability to explain technical decisions to non-technical stakeholders
Nice to Have
Databricks Unity Catalog, Auto Loader, Delta Sharing, or lakehouse concepts
Snowflake experience — schema design, data sharing, cost optimization
SAP data extraction via OData, PyRFC, or similar connectors
BI tools: Power BI, QuickSight, Tableau, or Databricks AI/BI
Databricks Certified Data Engineer Associate or above
API-based data ingestion patterns
Hands-on experience with MySQL, PostgreSQL, or MariaDB
