Job Title

Databricks Data Engineer

Job Description:

Position Summary

This role is ideal for someone who enjoys building well-architected data solutions, thrives in a modern cloud analytics environment, and wants to influence a next-generation Lakehouse platform. The engineer will play a central role in shaping data quality, architecture, and analytic capabilities across multiple domains.

We are seeking a skilled Databricks Data Engineer to help build and optimize a modern data platform leveraging the Databricks Lakehouse architecture. This role focuses on designing reliable ETL/ELT pipelines, implementing data governance within Unity Catalog, and enabling high-quality analytics products for internal and client-facing use.

Key Responsibilities

• • Design and develop data pipelines using Databricks (Python, SQL, Delta Live Tables, Workflows) to ingest, transform, and curate data across bronze, silver, and gold layers.
  • Implement and enforce governance standards within Unity Catalog, including catalog/schema/volume organization, permissions, lineage, and metadata management.
  • Build and optimize Delta Lake tables, including performance tuning, schema evolution strategies, audit patterns, and CDC approaches (e.g., Change Tracking, timestamp deltas).
  • Collaborate closely with architects and data teams to translate business requirements into scalable data models and operational pipelines.
  • Develop reusable frameworks and best practices for ingestion, orchestration, DevOps, monitoring, and quality validation.
  • Support CI/CD workflows for Databricks assets via Git-based development and automated deployment pipelines.
  • Ensure reliability and observability, including logging, testing, and performance diagnostics across the platform.
  • Contribute to cloud architecture decisions, such as storage design, identity management, and compute strategies (jobs, clusters, SQL warehouses).

Required Skills & Experience

• • 3+ years of experience in data engineering, preferably in a cloud environment (Azure, AWS, or GCP).
  • Hands-on experience with Databricks (Delta Lake, Workflows, DLT, Unity Catalog).
  • Strong proficiency in SQL and Python for data transformation and pipeline development.
  • Solid understanding of ETL/ELT patterns, medallion architecture, data modeling, and schema design.
  • Experience with CI/CD using Git-based workflows.
  • Working knowledge of cloud storage technologies (ADLS, S3, or GCS).
  • Familiarity with table-level governance, permissions, and security models in modern data platforms.

Preferred Qualifications

• • Experience integrating SQL Server Change Tracking, timestamp-based incremental ingestion, or similar CDC patterns.
  • Exposure to Terraform or IaC for Databricks governance and infrastructure configuration.
  • Understanding of analytics ecosystems such as Power BI, Tableau, or Business Objects.
  • Background working with domain-driven data models (e.g., CEDS, education data, ERP/operational domains).
  • Experience building multi-tenant data products or securely partitioned data architectures.

Soft Skills

• • Strong problem-solving mindset and ability to work in fast-moving, collaborative environments.
  • Clear, proactive communication that supports cross-team alignment.
  • A commitment to documentation, quality, and repeatable processes.