Monitor and troubleshoot interactive sessions

This document describes how to monitor and troubleshoot Serverless for Apache Spark interactive sessions using the Spark UI.

Spark UI

The Spark UI collects Apache Spark execution details from Serverless for Apache Spark interactive sessions. There is no charge for the Spark UI feature, which is enabled by default. Data collected by the Spark UI feature is retained for 90 days.

Required Identity and Access Management permissions and roles

The following permissions are required to use the Spark UI feature with interactive sessions:

• Data collection permission: dataproc.sessions.sparkApplicationWrite . This permission must be granted to the service account that runs interactive sessions. It is included in the Dataproc Worker role, which is automatically granted to the Compute Engine default service account that Serverless for Apache Spark uses by default. However, if you specify a custom service account for your session or session runtime template , you must add this dataproc.sessions.sparkApplicationWrite permission to that service account (typically, by granting the service account the Dataproc Worker role).

• Spark UI access permission: dataproc.sessions.sparkApplicationRead . This permission must be granted to a user to access the Spark UI in the Google Cloud console. It is included in the Dataproc Viewer , Dataproc Editor and Dataproc Administrator roles. To open the Spark UI in the Google Cloud console, you must have one of these roles or have a custom role that includes this permission.

Open the Spark UI

The Spark UI page is available in the Google Cloud console.

1. Go to Dataproc Interactive Sessions

2. Click an Interactive Session IDto open the Interactive Session Detailspage.
3. Click View Spark UIin the top menu.

Is the View Spark UI link disabled?If a required permission is not granted , or if you clear the Enable Spark UIcheckbox on the interactive session details page or set the spark.dataproc.appContext.enabled property to false (default: true ) when you create a session or create a session runtime template , the View Spark UIbutton will be disabled.