Console
    Contact Us Start free

    Create and view a Managed Service for Apache Kafka cluster

    Learn how to create a Managed Service for Apache Kafka cluster with a topic, and view details about the cluster. You can use the Google Cloud console or the Google Cloud CLI to complete this tutorial.

    Before you begin

    Console

    1. Sign in to your Google Account.

      If you don't already have one, sign up for a new account .

    2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

      Go to project selector

    3. Verify that billing is enabled for your Google Cloud project .

    4. Enable the Managed Kafka API.

      Enable the API

    5. Make sure that you have the following role or roles on the project: Managed Kafka Cluster Editor, Managed Kafka Topic Editor

      Check for the roles

      1. In the Google Cloud console, go to the IAM page.

        Go to IAM
      2. Select the project.

      3. In the Principal column, find all rows that identify you or a group that you're included in. To learn which groups you're included in, contact your administrator.

      4. For all rows that specify or include you, check the Role column to see whether the list of roles includes the required roles.

      Grant the roles

      1. In the Google Cloud console, go to the IAM page.

        Go to IAM
      2. Select the project.
      3. Click Grant access .

      4. In the New principals field, enter your user identifier. This is typically the email address for a Google Account.

      5. In the Select a role list, select a role.
      6. To grant additional roles, click Add another role and add each additional role.
      7. Click Save .

    6. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

      Go to project selector

    7. Verify that billing is enabled for your Google Cloud project .

    8. Enable the Managed Kafka API.

      Enable the API

    9. Make sure that you have the following role or roles on the project: Managed Kafka Cluster Editor, Managed Kafka Topic Editor

      Check for the roles

      1. In the Google Cloud console, go to the IAM page.

        Go to IAM
      2. Select the project.

      3. In the Principal column, find all rows that identify you or a group that you're included in. To learn which groups you're included in, contact your administrator.

      4. For all rows that specify or include you, check the Role column to see whether the list of roles includes the required roles.

      Grant the roles

      1. In the Google Cloud console, go to the IAM page.

        Go to IAM
      2. Select the project.
      3. Click Grant access .

      4. In the New principals field, enter your user identifier. This is typically the email address for a Google Account.

      5. In the Select a role list, select a role.
      6. To grant additional roles, click Add another role and add each additional role.
      7. Click Save .

    gcloud

    1. Sign in to your Google Account.

      If you don't already have one, sign up for a new account .

    2. Install the Google Cloud CLI.

    3. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity .

    4. To initialize the gcloud CLI, run the following command:

      gcloud  
init

    5. Create or select a Google Cloud project .

      • Create a Google Cloud project:

        gcloud projects create PROJECT_ID

        Replace PROJECT_ID with a name for the Google Cloud project you are creating.

      • Select the Google Cloud project that you created:

        gcloud config set project PROJECT_ID

        Replace PROJECT_ID with your Google Cloud project name.

    6. Verify that billing is enabled for your Google Cloud project .

    7. Enable the Managed Kafka API:

      gcloud  
services  
 enable 
  
managedkafka.googleapis.com

    8. Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/managedkafka.clusterEditor, roles/managedkafka.topicEditor 

      gcloud  
projects  
add-iam-policy-binding  
 PROJECT_ID 
  
--member = 
 "user: USER_IDENTIFIER 
" 
  
--role = 
 ROLE

      Replace the following:

      • PROJECT_ID : your project ID.
      • USER_IDENTIFIER : the identifier for your user account—for example, myemail@example.com .
      • ROLE : the IAM role that you grant to your user account.

    9. Install the Google Cloud CLI.

    10. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity .

    11. To initialize the gcloud CLI, run the following command:

      gcloud  
init

    12. Create or select a Google Cloud project .

      • Create a Google Cloud project:

        gcloud projects create PROJECT_ID

        Replace PROJECT_ID with a name for the Google Cloud project you are creating.

      • Select the Google Cloud project that you created:

        gcloud config set project PROJECT_ID

        Replace PROJECT_ID with your Google Cloud project name.

    13. Verify that billing is enabled for your Google Cloud project .

    14. Enable the Managed Kafka API:

      gcloud  
services  
 enable 
  
managedkafka.googleapis.com

    15. Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/managedkafka.clusterEditor, roles/managedkafka.topicEditor 

      gcloud  
projects  
add-iam-policy-binding  
 PROJECT_ID 
  
--member = 
 "user: USER_IDENTIFIER 
" 
  
--role = 
 ROLE

      Replace the following:

      • PROJECT_ID : your project ID.
      • USER_IDENTIFIER : the identifier for your user account—for example, myemail@example.com .
      • ROLE : the IAM role that you grant to your user account.
      16.

    Create a cluster

    Console

    1. Go to the Managed Service for Apache Kafka > Clusterspage.

      Go to Clusters

    2. Click Create.

    3. In the Cluster namebox, enter a name for the cluster.

    4. In the Regionlist, select a location for the cluster.

    5. For Network configuration, configure the subnet where the cluster is accessible:

      1. For Project, select your project.
      2. For Network, select the VPC network.
      3. For Subnet, select the subnet.
      4. Click Done.

    6. Click Create.

    After you click Create, the cluster state is Creating . When the cluster is ready, the state is Active .

    gcloud

    To create a Kafka cluster, run the managed-kafka clusters create command.

     gcloud  
managed-kafka  
clusters  
create  
 KAFKA_CLUSTER 
  
 \ 
--location = 
 REGION 
  
 \ 
--cpu = 
 3 
  
 \ 
--memory = 
3GiB  
 \ 
--subnets = 
projects/ PROJECT_ID 
/regions/ REGION 
/subnetworks/ SUBNET_NAME 
  
 \ 
--async

    Replace the following:

    • KAFKA_CLUSTER : a name for the Kafka cluster
    • REGION : the location of the cluster
    • PROJECT_ID : your project ID
    • SUBNET_NAME : the subnet where you want to create the cluster, for example default

    For information about supported locations, see Managed Service for Apache Kafka locations .

    The command runs asynchronously and returns an operation ID:

     Check  
operation  
 [ 
projects/ PROJECT_ID 
/locations/ REGION 
/operations/ OPERATION_ID 
 ] 
  
 for 
  
status.

    To track the progress of the create operation, use the gcloud managed-kafka operations describe command:

     gcloud  
managed-kafka  
operations  
describe  
 OPERATION_ID 
  
 \ 
  
--location = 
 REGION

    When the cluster is ready, the output from this command includes the entry state: ACTIVE . For more information, see Monitor the cluster creation operation .

    After you follow these steps, it can take up to 30 minutes for the cluster to be ready.

    Optionally, while you wait for this operation to complete, look at the tutorials listed in What's next . They show how to produce messages for the cluster using various methods. Once the cluster is ready, you can complete any of those tutorials.

    Create a topic

    When the Managed Service for Apache Kafka cluster is ready, create a topic.

    Console

    1. Go to the Managed Service for Apache Kafka > Clusterspage.

      Go to Clusters

    2. Click the name of the cluster.

    3. In the cluster details page, click Create Topic.

    4. In the Topic namebox, enter a name for the topic.

    5. Click Create.

    gcloud

    To create a Kafka topic, run the managed-kafka topics create command.

     gcloud  
managed-kafka  
topics  
create  
 KAFKA_TOPIC_NAME 
  
 \ 
--cluster = 
 KAFKA_CLUSTER 
  
 \ 
--location = 
 REGION 
  
 \ 
--partitions = 
 10 
  
 \ 
--replication-factor = 
 3

    Replace the following:

    • KAFKA_TOPIC_NAME : the name of the Kafka topic to create
    • KAFKA_CLUSTER : the name of the Kafka cluster
    • REGION : the region where you created the Kafka cluster

    View the cluster

    Once the cluster is ready, you can view information about the cluster, including the topics, consumer groups, networks, and other details.

    Console

    1. Go to the Managed Service for Apache Kafka > Clusterspage.

      Go to Clusters

    2. Click the name of the cluster. The Cluster detailspage shows information about the cluster. The Resourcestab shows the resources associated with the cluster, including topics.

    3. To view details about a topic, select the Resourcestab and click the topic name. The Topic detailspage shows information about the topic, such as the number of partitions and replicas.

    4. To return to the Cluster detailspage, click Back to parent page.

    gcloud

    1. To view information about the cluster, run the gcloud managed-kafka clusters describe command.

       gcloud  
managed-kafka  
clusters  
describe  
 KAFKA_CLUSTER 
  
 \ 
  
--location = 
 REGION

      Replace the following:

      • KAFKA_CLUSTER : the name of the cluster
      • REGION : the region where you created the cluster

      The output of this command is a list of cluster properties, similar to the following:

       bootstrapAddress: BOOTSTRAP_ADDRESS 
capacityConfig:
  memoryBytes: '3221225472'
  vcpuCount: '3'
createTime: ' TIMESTAMP 
'

    2. To get a list of topics in the cluster, run the gcloud managed-kafka topics list command.

       gcloud  
managed-kafka  
topics  
list  
 KAFKA_CLUSTER 
  
 \ 
  
--location = 
 REGION 
  
 \ 
  
--format = 
 "value(name)"

      The output includes the topics that you created, as well as topics that Managed Service for Apache Kafka creates for internal use.

    3. To view information about a topic, run the gcloud managed-kafka topics describe command.

       gcloud  
managed-kafka  
topics  
describe  
 TOPIC_NAME 
  
 \ 
  
--cluster = 
 KAFKA_CLUSTER 
  
 \ 
  
--location = 
 REGION

      Replace TOPIC_NAME with the name of the topic.

      The output of this command is a list of topic properties, similar to the following:

       name: projects/ PROJECT_ID 
/locations/  
/clusters/ KAFKA_CLUSTER 
/topics/ TOPIC_NAME 
partitionCount: 10
 replicationFactor: 3

    Get the bootstrap address

    Kafka clients use the cluster's bootstrap address to establish a connection with the cluster. To get the bootstrap address, perform the following steps:

    Console

    1. Go to the Managed Service for Apache Kafka > Clusterspage.

      Go to Clusters

    2. Click the name of the cluster.

    3. Select the Configurationstab.

    4. The bootstrap URL is listed under Bootstrap URL. Click Copyto copy the value.

    gcloud

    To get the bootstrap URL, run the managed-kafka clusters describe command.

     gcloud  
managed-kafka  
clusters  
describe  
 KAFKA_CLUSTER 
  
 \ 
  
--location = 
 REGION 
  
 \ 
  
--format = 
 "value(bootstrapAddress)"

    Replace the following:

    • KAFKA_CLUSTER : the name of the cluster
    • REGION : the region where you created the cluster

    You are now ready to send messages to the cluster. For more information, see What's next in this document.

    Clean up

    To avoid incurring charges to your Google Cloud account for the resources used on this page, delete the Google Cloud project with the resources.

    Console

    Delete the cluster.

    1. Go to the Managed Service for Apache Kafka > Clusterspage.

      Go to Clusters

    2. Select the Kafka cluster and click Delete.

    gcloud

    1. To delete the cluster, use the gcloud managed-kafka clusters delete command.

       gcloud  
managed-kafka  
clusters  
delete  
 KAFKA_CLUSTER 
  
 \ 
  
--location = 
 REGION 
  
--async

    What's next

    Produce messages for Kafka, either by using the Kafka command-line tools, or by creating a client application:
    Design a Mobile Site
    View Site in Mobile | Classic
    Share by: