Databricks Engineer Jobs

Databricks engineer jobs on carbon.talent, for data engineers and platform specialists who build on the Databricks lakehouse: Spark, Delta Lake, Unity Catalog and the surrounding tooling. Databricks roles are common in large enterprises, financial services and organisations moving off on-premise Hadoop, and many are contract positions tied to migrations. This guide covers the platform skills employers test and how to show Databricks experience that goes beyond notebooks.

Open roles

Databricks work sits on Apache Spark, so Spark fundamentals come first: how transformations and actions work, partitioning, shuffles, joins and the causes of skew. Interviewers often ask why a job is slow and expect answers involving partition sizes, broadcast joins, caching choices and avoiding unnecessary shuffles. PySpark is the most common language in listings, with SQL close behind and Scala in some older or performance-sensitive estates.

Delta Lake is the storage layer. Expect questions on ACID transactions, time travel, MERGE for upserts, schema enforcement and evolution, and maintenance operations such as OPTIMIZE and VACUUM. The medallion pattern of bronze, silver and gold layers is widely used; be ready to explain how you would design it for a specific use case and where data quality checks belong.

Governance and platform features increasingly decide senior hires. Unity Catalog for permissions, lineage and data discovery; cluster policies and cost controls; job orchestration with Databricks Workflows; Delta Live Tables for declarative pipelines; and deployment through Databricks Asset Bundles or Terraform all appear in listings. Experience setting up a workspace securely for multiple teams is particularly valuable.

Migration work is a large share of Databricks roles. Common projects include moving from on-premise Hadoop or legacy warehouses, consolidating several workspaces, or adopting Unity Catalog in an existing estate. Contract roles often focus on these migrations. Describe migrations you have done in terms of volumes, downtime, the order of work and how you validated that the new platform produced the same results.

Machine learning features overlap with data engineering on Databricks. Feature tables, MLflow for experiment tracking and model registry, and model serving show up in ML-adjacent listings. Data engineers who understand how their tables feed models, and ML engineers who understand Delta and Spark performance, are both in demand.

Cost management is a recurring interview theme. Explain how you choose cluster types and sizes, use autoscaling and auto-termination, schedule jobs on job clusters rather than all-purpose ones, and monitor spend. Concrete examples of reducing cost without slowing pipelines are persuasive.

Databricks offers certifications such as Data Engineer Associate and Professional, and listings sometimes request them. They help with screening; a public project or detailed case study of a real pipeline helps more in interview.

Common mistakes on Databricks include running production jobs on all-purpose clusters, leaving clusters running when idle, writing many small files that slow reads, and skipping table maintenance. Interviewers often ask how you would diagnose a slow job; walking through the Spark UI, partition sizes and file layout shows hands-on experience.

Streaming is another area where Databricks roles test depth. Structured Streaming, Auto Loader for incremental file ingestion, checkpointing and handling late data all appear in listings for real-time or near-real-time pipelines. If you have built streaming pipelines, describe throughput, latency targets and how you handled failures.

Testing and code quality are increasingly expected in Databricks roles. Moving logic out of notebooks into packaged modules, writing unit tests for transformations and running them in CI before deployment show maturity. Interviewers may ask how you structure a Databricks project for a team rather than for one person.

Security questions also come up: how secrets are stored, how service principals are used for jobs, how network access is restricted and how sensitive columns are masked. Clear answers show you can be trusted with production data.

Questions to ask: whether Unity Catalog is in place, how workspaces are organised, how code is deployed, and what the monthly spend looks like. The answers show whether you are joining a mature platform or helping build one. carbon.talent lists Databricks roles with their employment type and locations, and you apply from a free profile.

How carbon.talent works for candidates

  • Free signup, profile and applying.
  • Contract, fractional and full-time data and AI roles.
  • carbon.talent takes no cut of a contractor's rate.
Create your free profile

Questions

What skills do Databricks engineer jobs require?

Spark and PySpark, Delta Lake, SQL, the medallion architecture, Unity Catalog, Workflows and deployment tooling, plus cost management.

Are Databricks certifications worth it?

They can help pass screening. Detailed examples of real pipelines and migrations usually matter more in interviews.

Are there contract Databricks jobs?

Yes, many, often tied to migrations from Hadoop or legacy warehouses and to Unity Catalog adoption.

Do Databricks roles involve machine learning?

Some do, through feature tables, MLflow and model serving. Data engineering roles often feed ML teams on the same platform.