Apigee known issues

270574696

268104619

364872027

For Apigee and Apigee hybrid versions 1.13 and higher, any deviations in the required PEM format of keys used in Apigee JWS or JWT policies may result in a parsing error. For example, placing any character other an a newline ( /n ) immediately before the "-----END" line (post-encapsulation boundary) is not allowed and will result in an error.

To prevent this error, make sure that no characters other than a newline, such as trailing spaces or slashes, immediately precede the post-encapsulation boundary.

For more information about the encoding used for public or private keys, see IETF RFC 7468 .

310191899

The following endpoints may experience timeouts when used with a high volume of queries per second (QPS):

• organizations.environments.apis.revisions.
  deployments.deploy
• organizations.environments.apis.revisions.
  deployments.undeploy
• organizations.environments.sharedflows.revisions.
  deployments.deploy
• organizations.environments.sharedflows.revisions.
  deployments.undeploy

To reduce the likelihood of timeouts, we recommend setting a target of 1 QPS when using these endpoints or checking the status of a deployment before attempting another deployment.

329304975

Apigee is enforcing a temporary limit of 1000 basepaths per environment to avoid potential failures when deploying API proxy revisions.

While this limit is in place, you can deploy up to 1000 API proxy revisions (each containing a single basepath) per environment. If your API proxies or revisions contain more than one basepath, the total number of basepaths per environment must not exceed 1000.

333791378

For the steps required to install a patch for the workaround, see Troubleshooting .

310384001

289583112

If the OASValidation policy specifies an <OASResource> with security requirements set at a global level, the security requirements are not enforced.

Workaround : To ensure enforcement, all security requirements must be set at the operation level in the OpenAPI specification passed in the <OASResource> element of the OASValidation policy.

205666368

See About setting TLS options in a target endpoint or target server .

295929616

Installing or upgrading to Apigee hybrid 1.10.0 through 1.10.2 could fail on OSE due to out-of-memory issues. Fixed in Apigee hybrid version 1.10.3.

292118812

If the firewall forces all traffic through a forward-proxy, apigee-udca may go into a crash-loop backoff state.

292558790

• The OASValidation Policy fails when the JSON content does not match the anticipated pattern. For example, if a header is expecting a value in the format < text >@< text > and is populated with text missing the @ symbol, the policy will fail with an Unable to parse JSON error.
• If the OASValidation policy specifies an <OASResource> containing a path parameter that utilizes a $ref schema, the policy will fail with an Unable to parse JSON - Unrecognized token error.

  Workaround : Do not use $ref in the path parameters of the OpenAPI spec specified in the <OASResource> element.

297012500

• Apigee deployment will fail for OAS Validation policy when using circular references for OpenAPI 3.0.0 specification as it gets into an infinite loop.

Workaround : Use an OpenAPI specification yaml without circular references.

289254725

Proxy deployments that include the OASValidation policy may fail if:

• The OpenAPI specification used for validation in the OASValidation policy is in YAML format, and
• The YAML-formatted OpenAPI specification contains a floating number. For example:

   schema 
   : 
   type 
   : 
    
   number 
   example 
   : 
    
   2.345 
  

284500460

To avoid increasing latency in responses to the client, the Message Logging policy should be attached to the PostClientFlow . For more information on using policies in PostClientFlows, see Controlling API proxies with flows .

282997216

Use only alphanumeric characters for the Cassandra Jolokia password. Using special characters (including but not limited to "!", "@", "#", "$", "%", "^", "&", & "*") can cause Cassandra startup to fail.

270371160

Apigee Ingress gateway only supports TLS1.2+, and not earlier versions of TLS.

269139342

Apigee organization validation does not follow HTTP Forward proxy rules set in overrides.yaml . Set validateOrg: false to skip this validation.

266452840

In certain circumstances, web sockets are not working for Apigee X and Apigee Hybrid when using Anthos Service Mesh 1.15.3-asm.6 .

242213234

This error might be returned when attempting to load API products: "Products were not loaded successfully. Error: no connections available from the Apigee connect agent(s)."

The problem occurs after enabling VPC service control in the Google Cloud project and adding iamcredentials.googleapis.com as one of the restricted services in the service perimeter.

Workaround: Manually create an egress rule, such as the following:

-egressTo:
    operations:
    -serviceName: "iamcredentials.googleapis.com"
        methodSelectors:
        -method:
    resources:
    -projects/608305225983
  egressFrom:
    identityType: ANY_IDENTITY

247540503

In certain circumstances at very high throughput a race condition with encryption key lookup can cause KVM lookup failures.

258699204

If you see problems with the apigee-telemetry-app or apigee-telemetry-proxy pods not running, change the metrics resources requests and resources limits properties to match the following defaults in Configuration property reference: metrics .

Apply the changes with apigeectl apply with the ‑‑telemetry flag:

apigeectl apply --telemetry -f overrides.yaml 

260324159

API proxies and shared flows could take around 20 to 30 minutes to deploy in the runtime plane in certain circumstances due to a 'socket closed' error in synchronizer.

214447386

This is expected to occur every minute and does not affect your billing cost.

260772383

If installing hybrid on AKS, you may see this error:

envoy config listener '0.0.0.0_443' failed to bind or apply socket options: cannot bind '0.0.0.0:443': Permission denied

Workaround: Add the following svcAnnotations stanza to the overrides file:

ingressGateways:
- name: INGRESS_NAME
...
svcAnnotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"

See Configure the hybrid runtime . See also Use an internal load balancer with AKS .

241786534

When using Org-scoped UDCA, MART is sometimes unable to connect to FluentD. Org-scoped UDCA is the default in Apigee hybrid version 1.8. See orgScopedUDCA in the Configuration property reference.

After migration of apigee-logger from fluend to fluent-bit in Apigee hybrid version 1.6.6, the logger stopped working on Anthos BareMetal with CentOS or RHEL.

Current apigee-logger configurations, including liveness probes, are incompatible with the Kubernetes runtime, resulting in logs not being shipped to Cloud Logging as expected. This problem also causes the apigee-logger pods to crash regularly. This issue affects Apigee hybrid installations on AKS, Anthos Bare Metal, and other platforms. Note that in some cases this issue results in excessive log volume.

In the Apigee UI, you cannot view, confirm deployment status, or manage your archive deployments, as described Deploying an API proxy , or use the Debug UI as described in Using Debug . As a workaround, you can use gcloud or the API to List all archive deployments in an environment and use the Debug API .

Rolling back an archive deployment is not currently supported. To remove a version of an archive deployment you need to either redeploy a previous version of an archive or delete the environment .

421402073

Google authentication in ServiceCallout and ExternalCallout policies, as described in Using Google Authentication , is not supported in Apigee in VS Code.

Invalid HTTP Header error: The Istio ingress switches all incoming target responses to the HTTP2 protocol. Because the hybrid message processor only supports HTTP1, you may see the following error when an API proxy is called:

http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1,

name: [:authority], value: [domain_name]

If you see this error, you can take either of the following actions to correct the problem:

• Modify the target service to omit the Host header in the response.
• Remove the Host header using the AssignMessage policy in your API proxy if necessary.

420985360

• Apigee supports OpenAPI Specification 3.0 when you publish your APIs using SmartDocs on your portal, though a subset of features are not yet supported. For example, allOf properties for combining and extending schemas.

  If an unsupported feature is referenced in your OpenAPI Specification, in some cases the tools will ignore the feature but still render the API reference documentation. In other cases, an unsupported feature will cause errors that prevent the successful rendering of the API reference documentation. In either case, you will need to modify your OpenAPI Specification to avoid use of the unsupported feature until it is supported in a future release.

• When using Try this API in the portal, the Accept header is set to application/json regardless of the value set for consumes in the OpenAPI Specification.

• Simultaneous portal updates (such as page, theme, CSS, or script edits) by multiple users is not supported at this time.
• If you delete an API reference documentation page from the portal, there is no way to recreate it; you'll need to delete and re-add the API product, and regenerate the API reference documentation.
• When customizing your portal theme , it may take up to 5 minutes for changes to fully apply.

Search will be integrated into the integrated portal in a future release.

Single logout (SLO) with the SAML identity provider is not supported for custom domains. To enable a custom domain with a SAML identity provider, leave the Sign-out URL field blank when you configure SAML settings .

• API Proxy request and response counts (for proxy and targets) show abnormal spikes

  Here is a sample showing such a spike:

  ( view larger image )

  
• Due to a bug, the system registers the count incorrectly for a brief period and the count is corrected. This happens where there is a reduction in API traffic (which results in a scale down of API gateways).
• To distinguish actual spikes in requests vs. this issue, please consult the API Analytics page (specifically the Proxy Performance and Target Performance pages)

Affected Metrics:

• apigee.googleapis.com/proxyv2/request_count
• apigee.googleapis.com/proxyv2/response_count
• apigee.googleapis.com/targetv2/request_count
• apigee.googleapis.com/targetv2/response_count

New metrics

You can use the new metrics to avoid this issue.

For Apigee hybrid, see: Metrics collection overview and View metrics .

Workaround: Disable the logging agent on hybrid.

Workaround: To maintain the fire and forget behavior:

1. Add <Response>calloutResponse</Response> to the ServiceCallout.
2. Set continueOnError to true .

Workaround: Avoid using more than one SpikeArrest policy in the proxy to prevent the issue.

For a particular key, if there is continuous traffic, the key may not be rate limited at the updated rate. If there is five minutes of no traffic for a particular key, the rate will be reflected.

Workaround: Redeploy the proxy with a new reference variable if the rate has to take effect immediately. Or use two conditional spike arrests with different flow variables to adjust the rate.

API Monitoring of Configurable API Proxies may display a fault code of '(not set)' for responses with a non-2xx status from the target.

If a proxy sets two or more io.timeout.millis values in two or more flows using the same target host, only one io.timeout.millis value is honored.

During upgrade to Apigee hybrid 1.8.x, after running apigeectl init and confirming that check-ready succeeded, you may notice, if you view the pods, that the Cassandra schema validation job is in an error state. This is a harmless condition , and you can safely move to the next step in the upgrade procedure.

Deploying proxies with the same path to multiple environments that are attached to the same instance and environment group is not allowed and should return a warning message about a base path conflict. Instead, no error is shown and the deployments appear to succeed.

Workaround: When deploying and after deployment, verify that there are not base path conflicts with deployed proxies and correct as needed.

When attempting to save a previously deployed proxy, the deployment might fail with an error stating that the revision is immutable.

Workaround: Click the dropdown arrow next to the Save button and select Save as new revision . Then reattempt the deployment.

When a call is made to a gRPC Target Server, the only trailer that's returned is the "grpc-status" trailer. All other trailers are removed from the response.