Stay organized with collectionsSave and categorize content based on your preferences.
Distributed Cloud connected clusters use a local control plane deployed on your
Distributed Cloud connected hardware. When the connection to Google Cloud
is lost, your clusters enter survivability mode and your workloads continue to run for up to 7 days.
If a workload goes down while the cluster is in survivability mode, local image caching ensures
that workload comes back up when the connection Google Cloud is restored.
A cluster can enter survivability mode due to a hardware or software fault outside of Google's control,
or due to a fault in the Distributed Cloud connected hardware or software.
Examples of faults beyond Google's control:
Internet connectivity failure at the deployment site.
Firewall or network misconfiguration or hardware fault at the deployment site.
Boundary proxy instance serving the cluster is taken down or misconfigured.
If your Distributed Cloud connected cluster operates in survivability mode
for 7 days or fewer due to a hardware or software failure beyond Google's control,
Google Support works with you to restore it to normal operation up to the 7-day mark.
Past the 7-day mark, further support is not guaranteed.
Examples of faults in the Distributed Cloud connected hardware or software:
A faulty Distributed Cloud connected software update.
A failure in the Distributed Cloud connected machine or networking hardware.
An undiagnosed fault in the Distributed Cloud connected software.
If your Distributed Cloud connected cluster enters survivability mode due to a fault
in Distributed Cloud connected software or hardware, Google Support works with you
until the cluster is restored to normal operation.
What happens when a cluster enters survivability mode
When a Distributed Cloud connected cluster enters survivability mode,
the following happens:
Google automatically creates a support case against your Distributed Cloud connected
deployment using the contact information you provided when you ordered
Distributed Cloud connected hardware.
You are notified through email that the cluster has entered survivability mode and that a support
case has been created.
Google Supportworks with you to restore
your cluster to normal operation.
Cluster operation in survivability mode
When in survivability mode, a Distributed Cloud connected cluster
operates as follows:
When the connection to Google Cloud is lost, Distributed Cloud connected
continually attempts to reconnect to Google Cloud until the connection is re-established.
Control over workloads through the Google Cloud CLI, thekubectlCLI, and the
Distributed Cloud Edge Container API is disabled. You can, however, generate offline credentials
to access your clusters over an alternative internet connection as described inObtain credentials for a cluster.
Distributed Cloud software updates, SLOs, and hardware
repair are unavailable.
Limited logs and metrics are synchronized with Google Cloud after the
connection to Google Cloud is re-established:
System metrics are limited to 6 GB or 22 hours, whichever limit is reached first.
Workload logs are limited to 4 hours.
Workload metrics are limited to 1 GB.
Audit logs are limited to 10 GB.
By default, if a node reboots while the cluster is disconnected from Google Cloud,
it cannot rejoin its cluster until the connection to Google Cloud is
re-established because its authentication key cannot be refreshed. You have the option
to specify an offline reboot window during which a node can rejoin a cluster after
rebooting while the cluster is running in survivability mode. For more information,
seeCreate a cluster.
What to do when a cluster exits survivability mode
When a Distributed Cloud connected cluster exits survivability mode,
you might to address the following:
Distributed Cloud connected software version.You might need to update the affected
cluster to the latest version of Distributed Cloud connected software unless you deliberately
pinned the cluster to a specific software version. For more information,
seeUpgrade the software version of a cluster.
Fleet management certificates.You might need to refresh your expired fleet management LOAS certificates.
To address this, contactGoogle Support.
Check the connection state of a cluster
You can check the state of your Distributed Cloud cluster's to Google Cloud
by completing the steps inGet information about a cluster.
The command returns the value for theconnectionStatefield. This field can have one of the
following values:
CONNECTED: The cluster is connected to and fully synchronized with Google Cloud.
DISCONNECTED: The cluster is not connected to Google Cloud.
CONNECTED_AND_SYNCING: The cluster has reconnected to Google Cloud and is synchronizing
offline data with Google Cloud. Do not disconnect this cluster from Google Cloud until
synchronization has completed.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-09-04 UTC."],[[["\u003cp\u003eDistributed Cloud connected clusters utilize a local control plane on your hardware, and workloads continue running for up to 7 days in survivability mode when the connection to Google Cloud is lost.\u003c/p\u003e\n"],["\u003cp\u003eThese clusters must reside in their own Google Cloud project, separate from any other cluster types, to prevent data loss.\u003c/p\u003e\n"],["\u003cp\u003eThe local control plane operates in high availability mode with three replicas spread across three nodes, unless fewer nodes are present or a single-node configuration is specified.\u003c/p\u003e\n"],["\u003cp\u003eWhile in survivability mode, control of workloads via the Google Cloud CLI, \u003ccode\u003ekubectl\u003c/code\u003e, and the Distributed Cloud Edge Container API is disabled, and updates, SLOs, and hardware repair are unavailable.\u003c/p\u003e\n"],["\u003cp\u003eA cluster's connection status can be checked using a command that returns \u003ccode\u003eCONNECTED\u003c/code\u003e, \u003ccode\u003eDISCONNECTED\u003c/code\u003e, or \u003ccode\u003eCONNECTED_AND_SYNCING\u003c/code\u003e to indicate its state with Google Cloud.\u003c/p\u003e\n"]]],[],null,["# Survivability mode\n\nDistributed Cloud connected clusters use a local control plane deployed on your\nDistributed Cloud connected hardware. When the connection to Google Cloud\nis lost, your clusters enter survivability mode and your workloads continue to run for up to 7 days.\nIf a workload goes down while the cluster is in survivability mode, local image caching ensures\nthat workload comes back up when the connection Google Cloud is restored.\n\nA cluster can enter survivability mode due to a hardware or software fault outside of Google's control,\nor due to a fault in the Distributed Cloud connected hardware or software.\n\nExamples of faults beyond Google's control:\n\n- Internet connectivity failure at the deployment site.\n- Firewall or network misconfiguration or hardware fault at the deployment site.\n- Boundary proxy instance serving the cluster is taken down or misconfigured.\n\nIf your Distributed Cloud connected cluster operates in survivability mode\nfor 7 days or fewer due to a hardware or software failure beyond Google's control,\nGoogle Support works with you to restore it to normal operation up to the 7-day mark.\nPast the 7-day mark, further support is not guaranteed.\n\nExamples of faults in the Distributed Cloud connected hardware or software:\n\n- A faulty Distributed Cloud connected software update.\n- A failure in the Distributed Cloud connected machine or networking hardware.\n- An undiagnosed fault in the Distributed Cloud connected software.\n\nIf your Distributed Cloud connected cluster enters survivability mode due to a fault\nin Distributed Cloud connected software or hardware, Google Support works with you\nuntil the cluster is restored to normal operation.\n\nWhat happens when a cluster enters survivability mode\n-----------------------------------------------------\n\nWhen a Distributed Cloud connected cluster enters survivability mode,\nthe following happens:\n\n- Google automatically creates a support case against your Distributed Cloud connected deployment using the contact information you provided when you ordered Distributed Cloud connected hardware.\n- You are notified through email that the cluster has entered survivability mode and that a support case has been created.\n- [Google Support](/distributed-cloud/edge/latest/docs/getting-support) works with you to restore your cluster to normal operation.\n\nCluster operation in survivability mode\n---------------------------------------\n\nWhen in survivability mode, a Distributed Cloud connected cluster\noperates as follows:\n\n- When the connection to Google Cloud is lost, Distributed Cloud connected continually attempts to reconnect to Google Cloud until the connection is re-established.\n- Control over workloads through the Google Cloud CLI, the `kubectl` CLI, and the Distributed Cloud Edge Container API is disabled. You can, however, generate offline credentials to access your clusters over an alternative internet connection as described in [Obtain credentials for a cluster](/distributed-cloud/edge/latest/docs/clusters#credentials).\n- Distributed Cloud software updates, SLOs, and hardware repair are unavailable.\n- Limited logs and metrics are synchronized with Google Cloud after the connection to Google Cloud is re-established:\n - System metrics are limited to 6 GB or 22 hours, whichever limit is reached first.\n - Workload logs are limited to 4 hours.\n - Workload metrics are limited to 1 GB.\n - Audit logs are limited to 10 GB.\n- By default, if a node reboots while the cluster is disconnected from Google Cloud, it cannot rejoin its cluster until the connection to Google Cloud is re-established because its authentication key cannot be refreshed. You have the option to specify an offline reboot window during which a node can rejoin a cluster after rebooting while the cluster is running in survivability mode. For more information, see [Create a cluster](/distributed-cloud/edge/latest/docs/clusters#create).\n\nWhat to do when a cluster exits survivability mode\n--------------------------------------------------\n\nWhen a Distributed Cloud connected cluster exits survivability mode,\nyou might to address the following:\n\n- **Distributed Cloud connected software version.** You might need to update the affected cluster to the latest version of Distributed Cloud connected software unless you deliberately pinned the cluster to a specific software version. For more information, see [Upgrade the software version of a cluster](/distributed-cloud/edge/latest/docs/clusters#software-upgrade).\n- **Fleet management certificates.** You might need to refresh your expired fleet management LOAS certificates. To address this, contact [Google Support](/distributed-cloud/edge/latest/docs/getting-support).\n\nCheck the connection state of a cluster\n---------------------------------------\n\nYou can check the state of your Distributed Cloud cluster's to Google Cloud\nby completing the steps in [Get information about a cluster](/distributed-cloud/edge/latest/docs/clusters#get-cluster-info).\nThe command returns the value for the `connectionState` field. This field can have one of the\nfollowing values:\n\n- **`CONNECTED`**: The cluster is connected to and fully synchronized with Google Cloud.\n- **`DISCONNECTED`**: The cluster is not connected to Google Cloud.\n- **`CONNECTED_AND_SYNCING`**: The cluster has reconnected to Google Cloud and is synchronizing offline data with Google Cloud. Do not disconnect this cluster from Google Cloud until synchronization has completed.\n\nWhat's next\n-----------\n\n- [How Distributed Cloud connected works](/distributed-cloud/edge/latest/docs/how-it-works)"]]
