Version 1.16. This version is no longer supported. For information about how to upgrade to version 1.28, seeUpgrade clustersin the latest documentation. For more information about supported and unsupported versions, see theVersioningpage in the latest documentation.
This pages shows you how to resolve issues with etcd for Google Distributed Cloud.
If you need additional assistance, reach out toCloud Customer Care.
You can also seeGetting supportfor more information about support resources, including the following:
To resolve this issue, resize the data disk for etcd using the appropriate
procedure for your storage provider and operating system. Add enough additional
space to account for future etcd growth.
After the disk is resized, check if there's still a warning on disk space:
ETCDCTL_API=3etcdctlalarmlist
If the last column reportsNOSPACE, disarm the alarm as follows:
ETCDCTL_API=3etcdctlalarmdisarm
What's next
If you need additional assistance, reach out toCloud Customer Care.
You can also seeGetting supportfor more information about support resources, including the following:
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-09-04 UTC."],[[["\u003cp\u003eThis document addresses issues related to a full etcd data disk in Google Distributed Cloud, which can cause timeout errors in etcd logs and prevent the etcd server from starting.\u003c/p\u003e\n"],["\u003cp\u003eYou can determine the size of your etcd instance via SSH by using the \u003ccode\u003eETCDCTL_API=3 etcdctl --write-out=table endpoint status\u003c/code\u003e command, or by accessing the \u003ccode\u003emetric etcd_mvcc_db_total_size_in_bytes\u003c/code\u003e metric in the Cloud Monitoring console.\u003c/p\u003e\n"],["\u003cp\u003eTo resolve the issue of a full etcd data disk, you must resize the data disk using the appropriate method for your storage provider and operating system, adding enough space for future etcd growth.\u003c/p\u003e\n"],["\u003cp\u003eAfter resizing, verify if a disk space warning is present by running \u003ccode\u003eETCDCTL_API=3 etcdctl alarm list\u003c/code\u003e, and if \u003ccode\u003eNOSPACE\u003c/code\u003e is reported, disarm the alarm using \u003ccode\u003eETCDCTL_API=3 etcdctl alarm disarm\u003c/code\u003e.\u003c/p\u003e\n"]]],[],null,["# Troubleshoot etcd\n\n\u003cbr /\u003e\n\nThis pages shows you how to resolve issues with etcd for Google Distributed Cloud.\nIf you need additional assistance, reach out to [Cloud Customer Care](/support-hub). You can also see [Getting support](/anthos/clusters/docs/bare-metal/1.16/getting-support) for more information about support resources, including the following:\n\n- [Requirements](/kubernetes-engine/distributed-cloud/bare-metal/docs/getting-support#intro-support) for opening a support case.\n- [Tools](/kubernetes-engine/distributed-cloud/bare-metal/docs/getting-support#support-tools) to help you troubleshoot, such as your environment configuration, logs, and metrics.\n- Supported [components](/kubernetes-engine/distributed-cloud/bare-metal/docs/getting-support#what-we-support).\n\netcd data disk is full\n----------------------\n\nThe following error occurs when the etcd data disk is full, and might be\nobserved in a few different ways:\n\n- The etcd logs might show timeout errors for writing requests:\n\n rafthttp: failed to save KV snapshot (write /var/etcd/data/member/snap/tmp720030520: no space left on device)\n\n You might also see timeout errors for connections to peers: \n\n rafthttp: health check for peer [peer-id] could not connect: dial tcp [peer-ip]:2380: i/o timeout\n etcd server doesn't start:\n\n- The serial port logs might indicate that etcd can't start due to lack of space:\n\n failed on file /dev/stdout (No space left on device)\n\nTo determine the size of your etcd instance, use one of the following methods: \n\n### SSH\n\n1. Connect to one of the master nodes using SSH and run the following command:\n\n ETCDCTL_API=3 etcdctl --write-out=table endpoint status\n\n The `DB_SIZE` column indicates the size used, as shown in the following\n condensed example output: \n\n +------------------+------------------+---------+---------+\n | ENDPOINT | ID | VERSION | DB SIZE |\n +------------------+------------------+---------+---------+\n | 10.240.0.17:2379 | 4917a7ab173fabe7 | 3.5.0 | 45 kB |\n | 10.240.0.18:2379 | 59796ba9cd1bcd72 | 3.5.0 | 45 kB |\n | 10.240.0.19:2379 | 94df724b66343e6c | 3.5.0 | 45 kB |\n +------------------+------------------+---------+---------+\n\n### Console\n\n1. In the console, go to the Cloud Monitoring page.\n\n [Go to the Cloud Monitoring page](https://console.cloud.google.com/monitoring/)\n2. Select **Metrics explorer**.\n\n3. Select the `metric etcd_mvcc_db_total_size_in_bytes` metric.\n\nTo resolve this issue, resize the data disk for etcd using the appropriate\nprocedure for your storage provider and operating system. Add enough additional\nspace to account for future etcd growth.\n\n1. After the disk is resized, check if there's still a warning on disk space:\n\n ETCDCTL_API=3 etcdctl alarm list\n\n2. If the last column reports `NOSPACE`, disarm the alarm as follows:\n\n ETCDCTL_API=3 etcdctl alarm disarm\n\nWhat's next\n-----------\n\nIf you need additional assistance, reach out to [Cloud Customer Care](/support-hub). You can also see [Getting support](/anthos/clusters/docs/bare-metal/1.16/getting-support) for more information about support resources, including the following:\n\n- [Requirements](/kubernetes-engine/distributed-cloud/bare-metal/docs/getting-support#intro-support) for opening a support case.\n- [Tools](/kubernetes-engine/distributed-cloud/bare-metal/docs/getting-support#support-tools) to help you troubleshoot, such as your environment configuration, logs, and metrics.\n- Supported [components](/kubernetes-engine/distributed-cloud/bare-metal/docs/getting-support#what-we-support)."]]
