You're viewing Apigee
and Apigee hybrid
documentation.
View Apigee Edge
documentation.
The act of troubleshooting is both an art and a science. The constant effort of Apigee technical support teams has been to demystify the art and expose the science behind problem identification and resolution.
What are playbooks?
Developed in collaboration with ApigeeTechnical Support teams, Apigee troubleshooting playbooks are designed to provide quick and effective solutions to errors or other issues that you may encounter when working with Apigee products.
Audience
Troubleshooting playbooks are intended for readers with a high-level understanding of Apigee and its architecture, as well as some understanding of basic concepts such as policies and analytics.
Some problems can be diagnosed and solved only by Apigee hybrid users and may require knowledge of internal components such such as Cassandra and Postgres datastores, Message Processors, and Routers.
If you are on Apigee, then we clearly specify when you can perform the indicated troubleshooting steps and when you need to contact Google Cloud Customer Care for assistance.
Playbooks
This section describes the current playbooks.
To filter this table, do one or more of the following: select a category, select a product, type a search term, or click a column heading to sort.
Category
Playbook/Problem description
Error message
Playbook applicable for
Cassandra
If the Apigee CA certificate does not match across clusters, then
TLS certificate verification in Cassandra could fail.
Apigee hybrid only
Cassandra
During the Cassandra restoration in Apigee hybrid, you may encounter
errors in the restore logs.
Apigee hybrid only
Automated issue surfacing
Apigee API management requests fail:
- API products, Developers, Apps do not get populated on the Apigee UI.
- API proxy deployments do not complete.
- Apigee API management requests fail.
Apigee hybrid only
Automated issue surfacing
After running
kubectl -n apigee get apigeeissues
, the
AIS_VIRTUALHOST_MISSING_ENVGROUP error is displayed.Apigee hybrid only
Automated issue surfacing
After running
kubectl -n apigee get apigeeissues
, the
AIS_VIRTUALHOST_MISSING_SELECTOR error is displayed.Apigee hybrid only
Automated issue surfacing
After running
kubectl -n apigee get apigeeissues
, the
AIS_INGRESS_CERT_MISMATCH error is displayed.Apigee hybrid only
Automated issue surfacing
After running
kubectl -n apigee get apigeeissues
, the
AIS_INGRESS_CERT_EXPIREY error is displayed.Apigee hybrid only
Automated issue surfacing
After running
kubectl -n apigee get apigeeissues
, the
AIS_INGRESS_MTLS_CA_CERT_EXPIREY error is displayed.Apigee hybrid only
Automated issue surfacing
After running
kubectl -n apigee get apigeeissues
, the
AIS_INGRESS_MTLS_CA_CERT_INVALID error is displayed.Apigee hybrid only
Cassandra
When replicating data during a multi-region expansion, the
CassandraDataReplication
status may show an error state and data
replication may fail.Apigee hybrid only
Cassandra
Cassandra heap issues may cause slowness in the Apigee hybrid proxy
execution or even
Datastore
errors. Sometimes logs are an early
indicator, even before the onset of symptoms.Apigee hybrid only
Cassandra
Cassandra pods fail to start in one of the regions in a multi-region Hybrid setup.
You may see a
node already exists
error message in the Cassandra pod logs, or
a FailedPreStopHook
warning in the Cassandra pod status.Apigee hybrid only
Cassandra
When you use
kubectl
to view the pod states, you see that
one or more Cassandra pods are stuck. This guide describes the diagnosis
and resolution for problems with the Cassandra datastore.Apigee hybrid only
Deployment
The No active runtime pods
warning is displayed in the Details
dialog next to the error message Deployment issues on ENVIRONMENT
: REVISION_NUMBER
on the API proxy page.
Apigee hybrid only
Ingressgateway
curl: (7) Failed to connect to example.apis.com port 443: Operation timed out
Apigee hybrid only
Ingressgateway
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to example.apis.com:443
Apigee hybrid only
Logging
No error messages are known to be shown in this scenario.
Apigee and Apigee hybrid
Management/UI
No error messages are known to be shown in this scenario.
Apigee hybrid only
Network configuration
External clients are not able to access/connect to Apigee in a
desired manner. These include either network connectivity failures
(TLS handshake fails) or
4xx/5xx
responses from Apigee.Apigee and Apigee hybrid
Network configuration
A network connection issue or a TCP timeout between Apigee and the
target service would show up as a
503
error response and
would show an error similar to below if you create a debug session. {"fault":{"faultstring":"The Service is temporarily unavailable","detail":{"errorcode":"messaging.adaptors.http.flow.ServiceUnavailable","reason":"TARGET_CONNECT_TIMEOUT"}}}
Apigee and Apigee hybrid
Other
When draining Istio pods some nodes may not drain because they have a
replica count of 1, while 3 or more replicas are required. In order to
avoid this, you should set the minimum replica count for each property
to at least 3.
Apigee hybrid only
Other
One or more apigee-runtime pods are not in the
Ready
state.
When you use kubectl
to describe a failed apigee-runtime
pod, you see the error: Readiness probe failed: HTTP probe failed with statuscode: 500
Apigee hybrid only
Other
The
buildinfo
API returns information about the current
build for a runtime component. This information may be useful if you
need to contact support.Apigee hybrid only
Other
If you see in your metrics dashboard that the method
google.pubsub.vl.Subscriber.StreamingPull
is failing with
100% errors, you can safely ignore the issue. This is expected
behavior.Apigee hybrid only
Deployment
Deployments of API proxies fail with Instance INSTANCE_NAME
is not reporting status for environment group ENV_GROUP_NAME
error in the Apigee hybrid UI.
Apigee hybrid only
Deployment
API proxy deployments fail with error messages in the
apigee-watcher
logs.Apigee hybrid only
Ingressgateway
When draining Istio pods some nodes may not drain because they have a
replica count of
1
, while 3
or more replicas are
required. In order to avoid this, you should set the minimum replica count
for each property to at least 3
.Apigee hybrid only
Network configuration
During Apigee provisioning, if you select a network CIDR range that is not
completely free, you may see an error message.
Apigee and Apigee hybrid
Network configuration
This document describes how to diagnose and correct "503 Service Unavailable" errors with
TARGET_CONNECT_TIMEOUT when using internet or VPC peering targets.
Apigee
Network configuration
This document describes how to diagnose and correct "504 Gateway Timeout" errors with
a TARGET_READ_TIMEOUT reason.
Apigee and Apigee hybrid
Other
This document describes how to reset Apigee hybrid components when
they are stuck in a
creating
or releasing
state.Apigee hybrid only
Apigee hybrid installation and upgrade
Cassandra pods are stuck in a CrashLoopBackOff state after installation or upgrade.
Apigee hybrid only
Apigee hybrid installation and upgrade
UDCA and Connect Agent pods throw a permission issue error.
Apigee hybrid only
