How do I schedule and run Spark jobs in production?

Summary

• Lakeflow Jobs is Databricks' workflow automation for orchestrating data processing workloads. A job coordinates tasks such as notebooks, Python scripts, Python wheels, JARs, SQL and pipelines as one workflow.
• Attach a trigger to the job. Trigger types include a schedule (Quartz cron syntax plus a time zone), a table update, a file arrival, and continuous mode. You can also start runs manually or through the Jobs API.
• Production settings are explicit. Serverless compute for jobs is the default for tasks that support it, retries are set per task and are not on by default for most configurations, and failure notifications can go to email, Slack, PagerDuty or any webhook.
• Define the job as code with Declarative Automation Bundles, deploy it with the Databricks CLI, and set Run as to a service principal so runs don't depend on one person's permissions.
• Monitor runs in the Jobs UI, query the system.lakeflow system tables, or list runs with GET /api/2.2/jobs/runs/list.

How do I schedule and run Spark jobs in production?

Create a job in Lakeflow Jobs, add your Spark code as a task, attach a trigger, and set retries, notifications and permissions before it goes live. Databricks describes Lakeflow Jobs as "workflow automation for Databricks, providing orchestration for data processing workloads so that you can coordinate and run multiple tasks as part of a larger workflow." Keep the job definition in a Declarative Automation Bundle so you can validate, deploy and run it from the CLI.

What is a job in Lakeflow Jobs?

Apache Spark defines a job as "a parallel computation consisting of multiple tasks that gets spawned in response to a Spark action". A Lakeflow job is a larger unit: a workflow of tasks, where each task runs your code and can spawn many Spark jobs. Pick the task type that matches your code:

• Notebook, Python script, Python wheel: run on serverless compute for jobs, classic jobs compute, or all-purpose compute.
• JAR: deploys "Scala or Java code compiled into a JAR".
• SQL and dbt: SQL tasks run on a SQL warehouse, and the SQL that dbt generates runs on one too.
• Pipeline: runs a Lakeflow pipeline.
• Spark Submit: deprecated. Databricks recommends migrating JVM workloads to JAR tasks.

What should I configure before a job goes live?

Concern	What to set	What the docs say
Trigger	A trigger type such as scheduled, table update, file arrival or continuous	Scheduled triggers take a simple interval or an advanced schedule with a time zone and optional Quartz cron syntax. Databricks enforces a minimum interval of 10 seconds between schedule-triggered runs.
Compute	Serverless compute for jobs, or classic jobs compute	Serverless is the default for all tasks that support it. Databricks does not recommend all-purpose compute for production jobs.
Retries	max_retries, min_retry_interval_millis, retry_on_timeout on each task	For most configurations the default does not retry. A max_retries value of -1 retries indefinitely. Continuous jobs use an exponential backoff retry policy.
Dependencies	depends_on and Run if	Run if options are All succeeded (the default), At least one succeeded, None failed, All done, At least one failed and All failed.
Parameters	Job parameters	Notebook tasks read them with dbutils.widgets. Python scripts receive them as command-line arguments.
Notifications	Email, Slack, PagerDuty or a webhook	Events include start, success, failure, and a run duration that exceeds a threshold.
Access	CAN VIEW, CAN MANAGE RUN, CAN MANAGE, IS OWNER, and Run as	A service principal as Run as means runs "do not change when users leave or have changed privileges."
Monitoring	Jobs UI, system.lakeflow tables, Jobs API	The matrix view shows run history for each task. Task logs are in the run details.

How do I define the job as code?

This bundle runs a notebook on a Quartz schedule, retries the task, and emails someone on failure.

resources:
  jobs:
    my_job:
      name: my_job
      tasks:
        - task_key: my_task
          notebook_task:
            notebook_path: /Users/someone@example.com/my_notebook
          max_retries: 3
          min_retry_interval_millis: 60000
          retry_on_timeout: true
      schedule:
        quartz_cron_expression: '0 0 10 * * ?'
        timezone_id: America/Los_Angeles
      email_notifications:
        on_failure:
          - someone@example.com

In a bundle that defines a dev target, run databricks bundle validate, then databricks bundle deploy --target dev, then databricks bundle run --target dev my_job.

What happens when a run fails?

Retries are set per task, and when you set both a timeout and retries, the timeout applies to each retry. For a job with two or more tasks, Repair run re-runs "only the subset of unsuccessful tasks and any dependent tasks." Repair starts each unsuccessful task from the beginning, and "Lakeflow Jobs doesn't make tasks idempotent," so a task that wrote partial output can duplicate data when it runs again. Write your outputs so a repeat run is safe.

FAQs

Does the schedule handle time zones and daylight saving time?

Yes. An advanced schedule takes a time zone, and you can pick UTC or a zone that observes daylight saving time. With a daylight saving zone, an hourly job is skipped or might appear delayed by an hour or two when daylight saving time begins or ends.

How is Lakeflow Jobs different from Lakeflow Pipelines?

Lakeflow Jobs orchestrates tasks. Lakeflow Pipelines is "a declarative framework for building batch and streaming data pipelines in SQL and Python." A job can run a pipeline through a pipeline task.

How do I pass values between tasks?

Put dynamic value references in the parameters or fields that pass context into a task. Use {{job.parameters.<name>}} for a job parameter and {{tasks.<task_name>.values.<value_name>}} for a value an earlier task produced. The braces are not evaluated as expressions.

Can I start a job from outside Databricks?

Yes. Call POST /api/2.2/jobs/run-now or use the Databricks CLI, for example databricks jobs run-now.

Who can edit or run a job?

CAN MANAGE can edit the job definition and pause or resume a schedule. CAN MANAGE RUN can trigger and cancel runs. CAN VIEW can see run results.