Running Apache Spark Applications
Also available as:
PDF

Livy API Reference for Batch Jobs

GET /batches returns all active batch sessions.

Request Parameters Description Type
from Starting index used to fetch sessions int
size Number of sessions to fetch int
Response Description Type
from Starting index of fetched sessions int
total Number of sessions fetched int
sessions List of active batch sessions list

GET /batches/{batchId} returns the batch session information as a batch object.

GET /batches/{batchId}/state returns the state of batch session:

Response Description Type
id A non-negative integer that represents a specific batch session int
state The current state of batch session string

GET /batches/{batchId}/log retrieves log records for the specified batch session.

Request Parameters Description Type
from Offset into log file int
size Max number of log lines to return int
Response Description Type
id A non-negative integer that represents a specific batch session int
from Offset from start of the log file int
size Number of log records returned int
log Log records list of strings

POST /batches creates a new batch environment and runs a specified application:

Request Body Description Type
file File containing the application to run (required) path
proxyUser User ID to impersonate when running the job string
className Application Java or Spark main class string
args Command line arguments for the application list of strings
jars Jar files to be used in this session list of strings
pyFiles Python files to be used in this session list of strings
files Other files to be used in this session list of strings
driverMemory Amount of memory to use for the driver process string
driverCores Number of cores to use for the driver process int
executorMemory Amount of memory to use for each executor process string
executorCores Number of cores to use for each executor int
numExecutors Number of executors to launch for this session int
archives Archives to be used in this session list of strings
queue The name of the YARN queue to which the job should be submitted string
name Name of this session string
conf Spark configuration properties Map of key=val
Response Description Type
batch object (for more information see "Livy Batch Object") The created batch object batch object

DELETE /batches/{batchId} terminates the Batch job.