Reference documentation and code samples for the Google Cloud Dataproc V1 Client class PySparkBatch.
A configuration for running an Apache PySpark batch workload.
Generated from protobuf message google.cloud.dataproc.v1.PySparkBatch
Namespace
Google \ Cloud \ Dataproc \ V1Methods
__construct
Constructor.
data
array
Optional. Data for populating the Message object.
↳ main_python_file_uri
string
Required. The HCFS URI of the main Python file to use as the Spark driver. Must be a .py file.
↳ args
array
Optional. The arguments to pass to the driver. Do not include arguments that can be set as batch properties, such as --conf
, since a collision can occur that causes an incorrect batch submission.
↳ python_file_uris
array
Optional. HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py
, .egg
, and .zip
.
↳ jar_file_uris
array
Optional. HCFS URIs of jar files to add to the classpath of the Spark driver and tasks.
↳ file_uris
array
Optional. HCFS URIs of files to be placed in the working directory of each executor.
↳ archive_uris
array
Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar
, .tar
, .tar.gz
, .tgz
, and .zip
.
getMainPythonFileUri
Required. The HCFS URI of the main Python file to use as the Spark driver.
Must be a .py file.
string
setMainPythonFileUri
Required. The HCFS URI of the main Python file to use as the Spark driver.
Must be a .py file.
var
string
$this
getArgs
Optional. The arguments to pass to the driver. Do not include arguments
that can be set as batch properties, such as --conf
, since a collision
can occur that causes an incorrect batch submission.
setArgs
Optional. The arguments to pass to the driver. Do not include arguments
that can be set as batch properties, such as --conf
, since a collision
can occur that causes an incorrect batch submission.
var
string[]
$this
getPythonFileUris
Optional. HCFS file URIs of Python files to pass to the PySpark
framework. Supported file types: .py
, .egg
, and .zip
.
setPythonFileUris
Optional. HCFS file URIs of Python files to pass to the PySpark
framework. Supported file types: .py
, .egg
, and .zip
.
var
string[]
$this
getJarFileUris
Optional. HCFS URIs of jar files to add to the classpath of the Spark driver and tasks.
setJarFileUris
Optional. HCFS URIs of jar files to add to the classpath of the Spark driver and tasks.
var
string[]
$this
getFileUris
Optional. HCFS URIs of files to be placed in the working directory of each executor.
setFileUris
Optional. HCFS URIs of files to be placed in the working directory of each executor.
var
string[]
$this
getArchiveUris
Optional. HCFS URIs of archives to be extracted into the working directory
of each executor. Supported file types: .jar
, .tar
, .tar.gz
, .tgz
, and .zip
.
setArchiveUris
Optional. HCFS URIs of archives to be extracted into the working directory
of each executor. Supported file types: .jar
, .tar
, .tar.gz
, .tgz
, and .zip
.
var
string[]
$this
