This page lists services that Managed Service for Apache Spark image versions run on Managed Service for Apache Spark cluster nodes.
All nodes
The following services run on all nodes in a cluster.
Node type
Service
Image versions
Description
All nodes
all
Receives jobs from Managed Service for Apache Spark and launches job drivers
Standard clusters
The following services run on standard clusters.
Node type
Service
Image versions
Description
all
Manages Hive table metadata. As a default, uses the local
mariadb
(image versions < 1.5) or mysql
(image versions 1.5+) database
on the master node as the Hive table metadata store.
Using the default database is not recommended because these databases
are tied to the cluster's lifecycle. Instead, use either of the following as
the Hive metastore database (in recommendation order):
all
< 1.5
A relational database used as the default underlying database for Hive
metastore in Managed Service for Apache Spark < 1.5 images
1.5+
A relational database used as the default underlying database for Hive metastore
in Managed Service for Apache Spark 1.5+ images
HA Clusters
In Managed Service for Apache Spark High Availability (HA) clusters , different services run on different master nodes, as show below. HA cluster worker node services are the same as those listed for standard clusters .
Node type
Service
Image versions
Description
All masters
all
A quorum of journal nodes maintains an edit log of HDFS namespace modifications.
If a failover occurs, the Standby NameNode reads the edit log
and takes control from the Active NameNode.
all
Manages Hive table metadata. As a default, uses the local
mariadb
(image versions < 1.5) or mysql
(image versions 1.5+) database
on the master node as the Hive table metadata store.
Using the default database is not recommended because these databases
are tied to the cluster's lifecycle. Instead, use either of the following as
the Hive metastore database (in recommendation order):
all
all
A ZooKeeper quorum is used for distributed coordination. In High Availability (HA) clusters
,
it is used for HDFS NameNodes
and YARN
resource managers
leader election.
all
ZKFC is the
ZKFailoverController
process, which runs
with the HDFS NameNode. It monitors the health of the NameNode, and manages leader
election via ZooKeeper in the event of a failover.< 1.5
A relational database used as the default underlying database for Hive
metastore in Managed Service for Apache Spark < 1.5 images
1.5+
A relational database used as the default underlying database for Hive metastore
in Managed Service for Apache Spark 1.5+ images
