Reference documentation and code samples for the Google Cloud Dataproc V1 Client class ClusterConfig.
The cluster config.
Generated from protobuf message google.cloud.dataproc.v1.ClusterConfig
Methods
__construct
Constructor.
data
array
Optional. Data for populating the Message object.
↳ config_bucket
string
Optional. A Cloud Storage bucket used to stage job dependencies, config files, and job driver console output. If you do not specify a staging bucket, Cloud Dataproc will determine a Cloud Storage location (US, ASIA, or EU) for your cluster's staging bucket according to the Compute Engine zone where your cluster is deployed, and then create and manage this project-level, per-location bucket (see Dataproc staging and temp buckets
). This field requires a Cloud Storage bucket name, not a gs://...
URI to a Cloud Storage bucket.
↳ temp_bucket
string
Optional. A Cloud Storage bucket used to store ephemeral cluster and jobs data, such as Spark and MapReduce history files. If you do not specify a temp bucket, Dataproc will determine a Cloud Storage location (US, ASIA, or EU) for your cluster's temp bucket according to the Compute Engine zone where your cluster is deployed, and then create and manage this project-level, per-location bucket. The default bucket has a TTL of 90 days, but you can use any TTL (or none) if you specify a bucket (see Dataproc staging and temp buckets
). This field requires a Cloud Storage bucket name, not a gs://...
URI to a Cloud Storage bucket.
↳ gce_cluster_config
Google\Cloud\Dataproc\V1\GceClusterConfig
Optional. The shared Compute Engine config settings for all instances in a cluster.
↳ master_config
Google\Cloud\Dataproc\V1\InstanceGroupConfig
Optional. The Compute Engine config settings for the cluster's master instance.
↳ worker_config
Google\Cloud\Dataproc\V1\InstanceGroupConfig
Optional. The Compute Engine config settings for the cluster's worker instances.
↳ secondary_worker_config
Google\Cloud\Dataproc\V1\InstanceGroupConfig
Optional. The Compute Engine config settings for a cluster's secondary worker instances
↳ software_config
↳ initialization_actions
array< Google\Cloud\Dataproc\V1\NodeInitializationAction
>
Optional. Commands to execute on each node after config is completed. By default, executables are run on master and all worker nodes. You can test a node's role
metadata to run an executable on a master or worker node, as shown below using curl
(you can also use wget
): ROLE=$(curl -H Metadata-Flavor:Google http://metadata/computeMetadata/v1/instance/attributes/dataproc-role
) if [[ "${ROLE}" == 'Master' ]]; then ... master specific actions ... else ... worker specific actions ... fi
↳ encryption_config
↳ autoscaling_config
Google\Cloud\Dataproc\V1\AutoscalingConfig
Optional. Autoscaling config for the policy associated with the cluster. Cluster does not autoscale if this field is unset.
↳ security_config
↳ lifecycle_config
↳ endpoint_config
↳ metastore_config
↳ dataproc_metric_config
getConfigBucket
Optional. A Cloud Storage bucket used to stage job dependencies, config files, and job driver console output.
If you do not specify a staging bucket, Cloud
Dataproc will determine a Cloud Storage location (US,
ASIA, or EU) for your cluster's staging bucket according to the
Compute Engine zone where your cluster is deployed, and then create
and manage this project-level, per-location bucket (see Dataproc staging and temp
buckets
). This field requires a Cloud Storage bucket name, not a gs://...
URI to
a Cloud Storage bucket.
Generated from protobuf field string config_bucket = 1 [(.google.api.field_behavior) = OPTIONAL];
string
setConfigBucket
Optional. A Cloud Storage bucket used to stage job dependencies, config files, and job driver console output.
If you do not specify a staging bucket, Cloud
Dataproc will determine a Cloud Storage location (US,
ASIA, or EU) for your cluster's staging bucket according to the
Compute Engine zone where your cluster is deployed, and then create
and manage this project-level, per-location bucket (see Dataproc staging and temp
buckets
). This field requires a Cloud Storage bucket name, not a gs://...
URI to
a Cloud Storage bucket.
Generated from protobuf field string config_bucket = 1 [(.google.api.field_behavior) = OPTIONAL];
var
string
$this
getTempBucket
Optional. A Cloud Storage bucket used to store ephemeral cluster and jobs data, such as Spark and MapReduce history files.
If you do not specify a temp bucket,
Dataproc will determine a Cloud Storage location (US,
ASIA, or EU) for your cluster's temp bucket according to the
Compute Engine zone where your cluster is deployed, and then create
and manage this project-level, per-location bucket. The default bucket has
a TTL of 90 days, but you can use any TTL (or none) if you specify a
bucket (see Dataproc staging and temp
buckets
). This field requires a Cloud Storage bucket name, not a gs://...
URI to
a Cloud Storage bucket.
Generated from protobuf field string temp_bucket = 2 [(.google.api.field_behavior) = OPTIONAL];
string
setTempBucket
Optional. A Cloud Storage bucket used to store ephemeral cluster and jobs data, such as Spark and MapReduce history files.
If you do not specify a temp bucket,
Dataproc will determine a Cloud Storage location (US,
ASIA, or EU) for your cluster's temp bucket according to the
Compute Engine zone where your cluster is deployed, and then create
and manage this project-level, per-location bucket. The default bucket has
a TTL of 90 days, but you can use any TTL (or none) if you specify a
bucket (see Dataproc staging and temp
buckets
). This field requires a Cloud Storage bucket name, not a gs://...
URI to
a Cloud Storage bucket.
Generated from protobuf field string temp_bucket = 2 [(.google.api.field_behavior) = OPTIONAL];
var
string
$this
getGceClusterConfig
Optional. The shared Compute Engine config settings for all instances in a cluster.
Generated from protobuf field .google.cloud.dataproc.v1.GceClusterConfig gce_cluster_config = 8 [(.google.api.field_behavior) = OPTIONAL];
hasGceClusterConfig
clearGceClusterConfig
setGceClusterConfig
Optional. The shared Compute Engine config settings for all instances in a cluster.
Generated from protobuf field .google.cloud.dataproc.v1.GceClusterConfig gce_cluster_config = 8 [(.google.api.field_behavior) = OPTIONAL];
$this
getMasterConfig
Optional. The Compute Engine config settings for the cluster's master instance.
Generated from protobuf field .google.cloud.dataproc.v1.InstanceGroupConfig master_config = 9 [(.google.api.field_behavior) = OPTIONAL];
hasMasterConfig
clearMasterConfig
setMasterConfig
Optional. The Compute Engine config settings for the cluster's master instance.
Generated from protobuf field .google.cloud.dataproc.v1.InstanceGroupConfig master_config = 9 [(.google.api.field_behavior) = OPTIONAL];
$this
getWorkerConfig
Optional. The Compute Engine config settings for the cluster's worker instances.
Generated from protobuf field .google.cloud.dataproc.v1.InstanceGroupConfig worker_config = 10 [(.google.api.field_behavior) = OPTIONAL];
hasWorkerConfig
clearWorkerConfig
setWorkerConfig
Optional. The Compute Engine config settings for the cluster's worker instances.
Generated from protobuf field .google.cloud.dataproc.v1.InstanceGroupConfig worker_config = 10 [(.google.api.field_behavior) = OPTIONAL];
$this
getSecondaryWorkerConfig
Optional. The Compute Engine config settings for a cluster's secondary worker instances
Generated from protobuf field .google.cloud.dataproc.v1.InstanceGroupConfig secondary_worker_config = 12 [(.google.api.field_behavior) = OPTIONAL];
hasSecondaryWorkerConfig
clearSecondaryWorkerConfig
setSecondaryWorkerConfig
Optional. The Compute Engine config settings for a cluster's secondary worker instances
Generated from protobuf field .google.cloud.dataproc.v1.InstanceGroupConfig secondary_worker_config = 12 [(.google.api.field_behavior) = OPTIONAL];
$this
getSoftwareConfig
Optional. The config settings for cluster software.
Generated from protobuf field .google.cloud.dataproc.v1.SoftwareConfig software_config = 13 [(.google.api.field_behavior) = OPTIONAL];
hasSoftwareConfig
clearSoftwareConfig
setSoftwareConfig
Optional. The config settings for cluster software.
Generated from protobuf field .google.cloud.dataproc.v1.SoftwareConfig software_config = 13 [(.google.api.field_behavior) = OPTIONAL];
$this
getInitializationActions
Optional. Commands to execute on each node after config is completed. By default, executables are run on master and all worker nodes.
You can test a node's role
metadata to run an executable on
a master or worker node, as shown below using curl
(you can also use wget
):
ROLE=$(curl -H Metadata-Flavor:Google http://metadata/computeMetadata/v1/instance/attributes/dataproc-role
)
if [[ "${ROLE}" == 'Master' ]]; then
... master specific actions ...
else
... worker specific actions ...
fi
Generated from protobuf field repeated .google.cloud.dataproc.v1.NodeInitializationAction initialization_actions = 11 [(.google.api.field_behavior) = OPTIONAL];
Google\Protobuf\Internal\RepeatedField
setInitializationActions
Optional. Commands to execute on each node after config is completed. By default, executables are run on master and all worker nodes.
You can test a node's role
metadata to run an executable on
a master or worker node, as shown below using curl
(you can also use wget
):
ROLE=$(curl -H Metadata-Flavor:Google http://metadata/computeMetadata/v1/instance/attributes/dataproc-role
)
if [[ "${ROLE}" == 'Master' ]]; then
... master specific actions ...
else
... worker specific actions ...
fi
Generated from protobuf field repeated .google.cloud.dataproc.v1.NodeInitializationAction initialization_actions = 11 [(.google.api.field_behavior) = OPTIONAL];
$this
getEncryptionConfig
Optional. Encryption settings for the cluster.
Generated from protobuf field .google.cloud.dataproc.v1.EncryptionConfig encryption_config = 15 [(.google.api.field_behavior) = OPTIONAL];
hasEncryptionConfig
clearEncryptionConfig
setEncryptionConfig
Optional. Encryption settings for the cluster.
Generated from protobuf field .google.cloud.dataproc.v1.EncryptionConfig encryption_config = 15 [(.google.api.field_behavior) = OPTIONAL];
$this
getAutoscalingConfig
Optional. Autoscaling config for the policy associated with the cluster.
Cluster does not autoscale if this field is unset.
Generated from protobuf field .google.cloud.dataproc.v1.AutoscalingConfig autoscaling_config = 18 [(.google.api.field_behavior) = OPTIONAL];
hasAutoscalingConfig
clearAutoscalingConfig
setAutoscalingConfig
Optional. Autoscaling config for the policy associated with the cluster.
Cluster does not autoscale if this field is unset.
Generated from protobuf field .google.cloud.dataproc.v1.AutoscalingConfig autoscaling_config = 18 [(.google.api.field_behavior) = OPTIONAL];
$this
getSecurityConfig
Optional. Security settings for the cluster.
Generated from protobuf field .google.cloud.dataproc.v1.SecurityConfig security_config = 16 [(.google.api.field_behavior) = OPTIONAL];
hasSecurityConfig
clearSecurityConfig
setSecurityConfig
Optional. Security settings for the cluster.
Generated from protobuf field .google.cloud.dataproc.v1.SecurityConfig security_config = 16 [(.google.api.field_behavior) = OPTIONAL];
$this
getLifecycleConfig
Optional. Lifecycle setting for the cluster.
Generated from protobuf field .google.cloud.dataproc.v1.LifecycleConfig lifecycle_config = 17 [(.google.api.field_behavior) = OPTIONAL];
hasLifecycleConfig
clearLifecycleConfig
setLifecycleConfig
Optional. Lifecycle setting for the cluster.
Generated from protobuf field .google.cloud.dataproc.v1.LifecycleConfig lifecycle_config = 17 [(.google.api.field_behavior) = OPTIONAL];
$this
getEndpointConfig
Optional. Port/endpoint configuration for this cluster
Generated from protobuf field .google.cloud.dataproc.v1.EndpointConfig endpoint_config = 19 [(.google.api.field_behavior) = OPTIONAL];
hasEndpointConfig
clearEndpointConfig
setEndpointConfig
Optional. Port/endpoint configuration for this cluster
Generated from protobuf field .google.cloud.dataproc.v1.EndpointConfig endpoint_config = 19 [(.google.api.field_behavior) = OPTIONAL];
$this
getMetastoreConfig
Optional. Metastore configuration.
Generated from protobuf field .google.cloud.dataproc.v1.MetastoreConfig metastore_config = 20 [(.google.api.field_behavior) = OPTIONAL];
hasMetastoreConfig
clearMetastoreConfig
setMetastoreConfig
Optional. Metastore configuration.
Generated from protobuf field .google.cloud.dataproc.v1.MetastoreConfig metastore_config = 20 [(.google.api.field_behavior) = OPTIONAL];
$this
getDataprocMetricConfig
Optional. The config for Dataproc metrics.
Generated from protobuf field .google.cloud.dataproc.v1.DataprocMetricConfig dataproc_metric_config = 23 [(.google.api.field_behavior) = OPTIONAL];
hasDataprocMetricConfig
clearDataprocMetricConfig
setDataprocMetricConfig
Optional. The config for Dataproc metrics.
Generated from protobuf field .google.cloud.dataproc.v1.DataprocMetricConfig dataproc_metric_config = 23 [(.google.api.field_behavior) = OPTIONAL];
$this
