Running Apache Spark Applications
Also available as:
PDF

Livy API Reference for Interactive Sessions

GET

GET /sessions returns all active interactive sessions.

Request Parameter Description Type
from Starting index for fetching sessions int
size Number of sessions to fetch int
Response Description Type
from Starting index of fetched sessions int
total Number of sessions fetched int
sessions Session list list

The following response shows zero active sessions:

      {"from":0,"total":0,"sessions":[]}

GET /sessions/{sessionId} returns information about the specified session.

GET /sessions/{sessionId}/state returns the state of the specified session:

Response Description Type
id A non-negative integer that represents a specific session int
state Current state of the session string

GET /sessions/{sessionId}/logs retrieves log records for the specified session.

Request Parameters Description Type
from Offset int
size Maximum number of log records to retrieve int
Response Description Type
id A non-negative integer that represents a specific session int
from Offset from the start of the log file int
size Number of log records retrieved int
log Log records list of strings

GET /sessions/{sessionId}/statements returns all the statements in a session.

Response Description Type
statements List of statements in the specified session list

GET /sessions/{sessionId}/statements/{statementId} returns a specified statement in a session.

Response Description Type
statement object (for more information see "Livy Objects for Interactive Sessions") Statement statement object

POST

POST /sessions creates a new interactive Scala, Python, or R shell in the cluster.

Request Parameter Description Type
kind Session kind (required) session kind
proxyUser User ID to impersonate when starting the session string
jars Jar files to be used in this session list of strings
pyFiles Python files to be used in this session list of strings
files Other files to be used in this session list of strings
driverMemory Amount of memory to use for the driver process string
driverCores Number of cores to use for the driver process int
executorMemory Amount of memory to use for each executor process string
executorCores Number of cores to use for each executor process int
numExecutors Number of executors to launch for this session int
archives Archives to be used in this session list of strings
queue The name of the YARN queue to which the job should be submitted string
name Name of this session string
conf Spark configuration properties Map of key=value
heartbeatTimeoutInSecond Timeout in second to which session be orphaned int
Response Description Type
session object (for more information see "Livy Objects for Interactive Sessions") The created session session object

The following response shows a PySpark session in the process of starting:

      {"id":0,"state":"starting","kind":"pyspark","log":[]}

POST /sessions/{sessionId}/statements runs a statement in a session.

Request Parameter Description Type
code The code to execute string
Response Description Type
statement object (for more information see "Livy Objects for Interactive Sessions") Result of an execution statement statement object

POST /sessions/{sessionId}/statements/{statementId}/cancel cancels the specified statement in the session.

Response Description Type
cancellation message Reports "cancelled" string

DELETE

DELETE /sessions/{sessionId} terminates the session.