Managed Service for Apache Spark components

The Managed Service for Apache Spark components let you run Apache Spark batch workloads from a pipeline within Vertex AI Pipelines. Managed Service for Apache Spark runs the batch workloads on a managed compute infrastructure, autoscaling resources as needed.

Learn more about Managed Service for Apache Spark and supported Spark workloads .

In Managed Service for Apache Spark, a Batch resource represents a batch workload. The Google Cloud SDK includes the following operators to create Batch resources and monitor their execution:

• DataprocPySparkBatchOp
• DataprocSparkBatchOp
• DataprocSparkRBatchOp
• DataprocSparkSqlBatchOp

API reference

• For component reference, see the Google Cloud SDK reference for Managed Service for Apache Spark components .

• For Managed Service for Apache Spark resource reference, see the following API reference page:

  • Batch resource

Tutorials

• Get started with Managed Service for Apache Spark pipeline components

Version history and release notes

To learn more about the version history and changes to the Google Cloud Pipeline Components SDK, see the Google Cloud Pipeline Components SDK Release Notes .

Technical support contacts

If you have any questions, reach out to kfp-dataproc-components@google.com .