Senior Data Engineer

The Senior Data Engineer owns that substrate end to end: ingestion from messy third-party APIs, modeling and governance in Unity Catalog, the serving layer agents query at runtime, and the health monitoring that tells us something broke before a CSM finds out in a client meeting. Infrastructure is code, deploys go through CI/CD, and quality is measured - not assumed. You write the Terraform, you own the alerts, you get paged by your own monitors.

Requirements

Strong software engineering fundamentals in Python and advanced SQL - this role builds tested, deployed, version-controlled pipelines, not one-off notebooks

Deep hands-on experience with Databricks: Spark/PySpark, Delta Lake, Unity Catalog, Workflows, and SQL warehouses, including performance tuning of real workloads

Proven experience designing data pipelines and data models in production: medallion or equivalent layered architectures, incremental processing, CDC, and dimensional modeling

Experience integrating third-party APIs at scale, with the operational maturity that implies (auth, pagination, throttling, partial failure, backfills, idempotency, schema evolution)

Familiarity with cloud platforms (we run on Azure - ADLS, Container Apps, Key Vault, API Management, Entra ID)

Experience with Infrastructure as Code (Terraform) and CI/CD pipelines, plus solid git and code-review habits

A working definition of data quality: tests, expectations, monitoring, and the discipline to catch problems upstream rather than explaining them downstream

Practical experience using AI coding assistants/agents in real engineering work, and the critical eye to know when their output is wrong

Excellent problem-solving and analytical skills, and the autonomy expected of a senior engineer: you own a problem end to end - from framing to shipped, monitored outcome - and are accountable for the result, not just the merge

Bachelor's degree or higher in Computer Science, Information Technology, or a related field - or equivalent practical experience

Preferred Qualifications

Experience serving data to LLM-powered applications, including retrieval and context design

Experience with operational stores alongside the lakehouse (Databricks Lakebase, PostgreSQL, MongoDB)

Observability stacks: OpenTelemetry, Grafana, Loki, structured logging

Delta Sharing for secure data exchange with clients and partners

Domain background in retail media, digital advertising, or marketplace data

Databricks or Azure certifications

What You'll Do

Design, build, and operate ingestion pipelines from advertising and business platforms (Amazon Ads, Google Ads, Walmart Connect, Criteo, Salesforce, and internal services) via REST APIs, Delta Sharing, object storage, and event streams

Model curated data layers in Delta Lake under Unity Catalog: medallion architecture, incremental and CDC patterns, clear contracts between layers, lineage, and row/column-level governance

Build the serving layer for AI: SQL views, Unity Catalog functions, and low-latency stores exposed to agents as tools through the Model Context Protocol (MCP) behind Azure API Management - designing for what an agent needs to reason well, not just for what a BI tool needs to render

Own data quality and health monitoring as code: freshness, volume, schema-drift and business-rule expectations, with alerting wired into Grafana and a clear triage path when something fires

Ship everything through Terraform and CI/CD - Databricks and Azure resources, jobs, permissions, and environment promotion from dev to prod, with no manual clicking in the console

Own performance and cost: warehouse and cluster sizing, partitioning and liquid clustering, job orchestration in Databricks Workflows / Lakeflow, and continuous scrutiny of what each pipeline actually costs to run

Work fluently with AI coding agents as part of your daily engineering practice - writing precise specs, keeping repository context and documentation in a state where agents produce good output, and reviewing what they generate with real judgment

Collaborate with cross-functional teams - AI engineers, software engineers, and product - turning product requirements into data the platform can actually serve, and treating internal consumers as customers with contracts and expectations

Instrument what you ship with OpenTelemetry and structured logging, and improve reliability and latency in production

Document and maintain the codebase, ensuring code quality and adherence to best practices