Stay organized with collectionsSave and categorize content based on your preferences.
Google remotely monitors and maintains the Google Distributed Cloud connected
hardware. For this purpose, Google engineers have Secure Shell (SSH) access to
the Distributed Cloud connected hardware. If Google detects an issue,
a Google engineer contacts you to troubleshoot and resolve it. If you have
identified an issue yourself, contact Google Support immediately to diagnose
and resolve it.
Network connectivity loss
If the Distributed Cloud connected hardware loses its connection to
Google Cloud and remains disconnected for 120 seconds, the
Distributed Cloud connected control plane marks the affected Pods as
"Not Ready" and initiates Pod eviction.
To mitigate this, you must plan your Distributed Cloud connected
configuration and architect your workloads for your chosen level of availability.
For more information, seeAvailability best practices.
Corrupt BGP sessions in Cloud Router resources used by VPN connections
Distributed Cloud VPN connections rely on BGP sessions
established and managed by their corresponding Cloud Router resources to
advertise routes between the Distributed Cloud connected
cluster and Google Cloud. If you modify the configuration of a
Cloud Router resource associated with a Distributed Cloud
VPN connection, that connection can stop functioning.
To recover the corrupt BGP session configuration in the affected Cloud Router,
complete the following steps:
In the Google Cloud console, get the name of the corrupt BGP session. For example:
INTERFACE=anthos-mcc-34987234
Get the peer BGP and the Cloud Router BGP IP addresses
for the corrupted BGP session, as well as the peer ASN used by the affected Distributed Cloud VPN connection. For example:
INTERFACE_NAME: the name of the interface that you
created in the previous step.
TUNNEL_NAME: the name of the VPN tunnel that you used
to create the interface in the previous step.
REGION: the Google Cloud region in which the
target VPC Google Cloud project is created.
VPC_PROJECT_ID: the ID of the target VPC
Google Cloud project.
GDCE_BGP_IP: the Distributed Cloud
peer BGP IP address that you obtained earlier in this procedure.
GDCE_BGP_ASN: the Distributed Cloud
peer BGP ASN that you obtained earlier in this procedure.
At this point, the BGP session is back up and operational.
Node stuck in theReady,SchedulingDisabledstate
When you apply or delete theNodeSystemConfigUpdateorSriovNetworkNodePolicyresource,
the target node might reboot. When a node reboots, its status changes toNotReadyorScheduling Disabled.
If a node remains in aReady,SchedulingDisabledstate for more than 30 minutes, do the following:
Check the configuration and status of the correspondingNodeSystemConfigUpdateorSriovNetworkNodePolicyresource. If theSriovNetworkNodePolicyresource does not exist.
the node is not capable of SR-IOV.
If the resource status isSucceeded, enable scheduling on the node using the following command:
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-09-04 UTC."],[[["\u003cp\u003eGoogle engineers monitor and maintain the Google Distributed Cloud connected hardware, and will contact you to help resolve any detected issues.\u003c/p\u003e\n"],["\u003cp\u003eIf network connectivity to Google Cloud is lost for 120 seconds, affected Pods are marked as "Not Ready" and eviction is initiated, requiring planning and workload architecture for availability.\u003c/p\u003e\n"],["\u003cp\u003eModifying Cloud Router resource configurations for Distributed Cloud VPN connections can disrupt BGP sessions, requiring a specific recovery process to fix any issues.\u003c/p\u003e\n"],["\u003cp\u003eNodes may reboot when updating \u003ccode\u003eNodeSystemConfigUpdate\u003c/code\u003e or \u003ccode\u003eSriovNetworkNodePolicy\u003c/code\u003e resources, and if a node remains in a \u003ccode\u003eReady,SchedulingDisabled\u003c/code\u003e state for over 30 minutes, you will need to verify the configuration of said resource, and uncordon the node if the status is \u003ccode\u003eSucceeded\u003c/code\u003e.\u003c/p\u003e\n"]]],[],null,["# Troubleshoot Distributed Cloud connected\n\nGoogle remotely monitors and maintains the Google Distributed Cloud connected\nhardware. For this purpose, Google engineers have Secure Shell (SSH) access to\nthe Distributed Cloud connected hardware. If Google detects an issue,\na Google engineer contacts you to troubleshoot and resolve it. If you have\nidentified an issue yourself, contact Google Support immediately to diagnose\nand resolve it.\n\nNetwork connectivity loss\n-------------------------\n\nIf the Distributed Cloud connected hardware loses its connection to\nGoogle Cloud and remains disconnected for 120 seconds, the\nDistributed Cloud connected control plane marks the affected Pods as\n\"Not Ready\" and initiates Pod eviction.\n\nTo mitigate this, you must plan your Distributed Cloud connected\nconfiguration and architect your workloads for your chosen level of availability.\nFor more information, see\n[Availability best practices](/distributed-cloud/edge/1.6.1/docs/availability).\n\nCorrupt BGP sessions in Cloud Router resources used by VPN connections\n----------------------------------------------------------------------\n\nDistributed Cloud VPN connections rely on BGP sessions\nestablished and managed by their corresponding Cloud Router resources to\nadvertise routes between the Distributed Cloud connected\ncluster and Google Cloud. If you modify the configuration of a\nCloud Router resource associated with a Distributed Cloud\nVPN connection, that connection can stop functioning.\n\nTo recover the corrupt BGP session configuration in the affected Cloud Router,\ncomplete the following steps:\n\n1. In the Google Cloud console, get the name of the corrupt BGP session. For example:\n\n INTERFACE=anthos-mcc-34987234\n\n2. Get the peer BGP and the Cloud Router BGP IP addresses\n for the corrupted BGP session, as well as the peer ASN used by the affected Distributed Cloud VPN connection. For example:\n\n GDCE_BGP_IP=168.254.208.74\n CLOUD_ROUTER_BGP_IP=168.254.208.73\n PEER_ASN=65506\n\n If you deleted the BGP session, get this information from the Distributed Cloud connected cluster instead:\n 1. Get the cluster credentials:\n\n ```\n gcloud edge-cloud container clusters get-credentials CLUSTER_ID \\\n --location REGION \\\n --project PROJECT_ID\n ```\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003eCLUSTER_ID\u003c/var\u003e: the name of the target cluster.\n - \u003cvar translate=\"no\"\u003eREGION\u003c/var\u003e: the Google Cloud region in which the target cluster is created.\n - \u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e: the ID of the target Google Cloud project.\n 2. Get the configuration of the `MultiClusterConnectivityConfig` resource:\n\n ```\n kubectl get multiclusterconnectivityconfig -A\n ```\n\n The command returns output similar to the following: \n\n NAMESPACE NAME LOCAL ASN PEER ASN\n kube-system MultiClusterConfig1 65505 65506\n ```\n\n 3. Get the peer BGP IP address, the Cloud Router IP address, and the\n BGP session ASN:\n\n ```\n kubectl describe multiclusterconnectivityconfig -n kube-system MCC_CONFIG_NAME \n ```\n\n Replace \u003cvar translate=\"no\"\u003eMCC_CONFIG_NAME\u003c/var\u003e with the name of the\n `MultiClusterConfigResource` that you obtained in the previous step.\n\n The command returns output similar to the following: \n\n Spec:\n Asns:\n Peer: 65505\n Self: 65506 # GDCE ASN\n Tunnels:\n Ike Key:\n Name: MCC_CONFIG_NAME-0\n Namespace: kube-system\n Peer:\n Bgp IP: 169.254.208.73 # Cloud Router BGP IP\n Private IP: 34.157.98.148\n Public IP: 34.157.98.148\n Self:\n Bgp IP: 169.254.208.74 # GDCE BGP IP\n Private IP: 10.100.29.49\n Public IP: 208.117.254.68\n ```\n\n3. In the Google Cloud console, get the name, region, and\n Google Cloud project name for the corrupted VPN tunnel. For example:\n\n VPN_TUNNEL=VPNTunnel1\n REGION=US-East1\n VPC_PROJECT_ID=VPC-Project-1\n\n4. Delete the corrupted BGP session from the Cloud Router configuration.\n\n5. Create a new Cloud Router interface:\n\n ```\n gcloud compute routers add-interface --interface-name=INTERFACE_NAME \\\n --vpn-tunnel=TUNNEL_NAME \\ \n --ip-address=ROUTER_BGP_IP \\\n --project=VPC_PROJECT_ID \\\n --region=REGION \\ \n --mask-length=30\n ```\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003eINTERFACE_NAME\u003c/var\u003e: a descriptive name that uniquely identifies this interface.\n - \u003cvar translate=\"no\"\u003eTUNNEL_NAME\u003c/var\u003e: the name of the VPN tunnel that you obtained in the previous step.\n - \u003cvar translate=\"no\"\u003eROUTER_BGP_IP\u003c/var\u003e: the BGP IP address of the Cloud Router that you obtained earlier in this procedure.\n - \u003cvar translate=\"no\"\u003eVPC_PROJECT_ID\u003c/var\u003e: the ID of the target VPC Google Cloud project.\n - \u003cvar translate=\"no\"\u003eREGION\u003c/var\u003e: the Google Cloud region in which the target VPC Google Cloud project has been created.\n6. Create the BGP peer:\n\n ```\n gcloud compute routers add-bgp-peer --interface=INTERFACE_NAME \\\n --peer-name=TUNNEL_NAME \\\n --region REGION \\\n --project=VPC_PROJECT_ID \\\n --peer-ip-address=GDCE_BGP_IP \\\n --peer-asn=GDCE_BGP_ASN \\\n --advertised-route-priority=100 \\\n --advertisement-mode=DEFAULT\n ```\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003eINTERFACE_NAME\u003c/var\u003e: the name of the interface that you created in the previous step.\n - \u003cvar translate=\"no\"\u003eTUNNEL_NAME\u003c/var\u003e: the name of the VPN tunnel that you used to create the interface in the previous step.\n - \u003cvar translate=\"no\"\u003eREGION\u003c/var\u003e: the Google Cloud region in which the target VPC Google Cloud project is created.\n - \u003cvar translate=\"no\"\u003eVPC_PROJECT_ID\u003c/var\u003e: the ID of the target VPC Google Cloud project.\n - \u003cvar translate=\"no\"\u003eGDCE_BGP_IP\u003c/var\u003e: the Distributed Cloud peer BGP IP address that you obtained earlier in this procedure.\n - \u003cvar translate=\"no\"\u003eGDCE_BGP_ASN\u003c/var\u003e: the Distributed Cloud peer BGP ASN that you obtained earlier in this procedure.\n\nAt this point, the BGP session is back up and operational.\n\nNode stuck in the `Ready,SchedulingDisabled` state\n--------------------------------------------------\n\nWhen you apply or delete the `NodeSystemConfigUpdate` or `SriovNetworkNodePolicy` resource,\nthe target node might reboot. When a node reboots, its status changes to `NotReady` or `Scheduling Disabled`.\nIf a node remains in a `Ready,SchedulingDisabled` state for more than 30 minutes, do the following:\n\n1. Check the configuration and status of the corresponding `NodeSystemConfigUpdate` or\n `SriovNetworkNodePolicy` resource. If the `SriovNetworkNodePolicy` resource does not exist.\n the node is not capable of SR-IOV.\n\n2. If the resource status is `Succeeded`, enable scheduling on the node using the following command:\n\n ```\n kubectl uncordon NODE_NAME.\n ```\n\n Replace \u003cvar translate=\"no\"\u003eNODE_NAME\u003c/var\u003e with the name of the target node.\n3. If the issue persists, contact [Google Support](/distributed-cloud/edge/1.6.1/docs/getting-support)."]]
