Apache Livy

Definition

Apache Livy is a REST interface that lets applications submit and manage Apache Spark jobs over HTTP without needing a Spark client installed locally or direct network access to the cluster. It supports both long-running interactive Spark sessions, useful for notebooks, and batch job submission, managing the underlying SparkContext lifecycle so multiple clients can share Spark resources through a single service.

Overview

Running Spark jobs traditionally requires a client with the correct Spark libraries, cluster configuration, and network access to submit work, which is awkward for web applications, notebooks, or services that want to trigger Spark computation without embedding a full Spark installation. Apache Livy addresses this by exposing Spark job submission and interaction as a plain REST API, so any HTTP client, in any language, can start a Spark job or query its results. Underneath the API, Livy manages one or more SparkContexts on the cluster, launching them through the cluster's resource manager, typically YARN, and keeping them alive across multiple requests when operating in interactive session mode. A client opens a session, submits code snippets in Scala, Python, or R as statements against that session, and receives results back over REST, which is the mechanism notebook tools use to give users an interactive Spark experience without installing Spark on their own machine. For non-interactive use, Livy also accepts complete batch job submissions, packaging and launching them on the cluster and exposing status and log endpoints so a caller can poll for completion. Livy differs from submitting jobs directly through `spark-submit` in that it decouples the client from the cluster's Spark installation entirely and adds multi-tenancy: many users or applications can share a Livy service, each with isolated sessions, rather than each needing their own gateway host with Spark installed. It is often deployed alongside Apache Oozie or general workflow schedulers, providing the piece those tools need to trigger Spark work over a simple network call instead of shelling out to a local Spark binary. In practice, organizations use Livy to let a web application or scheduling service kick off Spark ETL jobs on demand, to back notebook environments where data scientists write exploratory Spark code without a local Spark setup, and to expose Spark execution to services written in languages that have no native Spark client library, since any HTTP client can talk to Livy. The main trade-offs are operational: a Livy service becomes another stateful component to secure, monitor, and scale, session management can consume cluster resources if sessions are left idle rather than closed, and its REST abstraction adds a layer of indirection compared to submitting jobs directly, which can complicate debugging low-level Spark configuration issues, since problems sometimes have to be traced through Livy's own logs before reaching the underlying Spark driver output, adding one more layer a team must understand when something goes wrong under load.

Key Features

• REST API for submitting and managing Spark jobs over HTTP
• Interactive session mode for Scala, Python, and R code snippets
• Batch job submission with polling endpoints for status and logs
• Manages SparkContext lifecycle on behalf of multiple clients
• Multi-tenant design allowing many users to share one Livy service
• Decouples clients from needing a local Spark installation

Use Cases

Alternatives

Frequently Asked Questions

Related in Database