Ingress cert mismatch

You're viewing Apigee and Apigee hybrid documentation.
There is no equivalent Apigee Edge documentation for this topic.

Symptom

An ApigeeIssue with Reason AIS_INGRESS_CERT_MISMATCH is shown when viewing resources in the apigee namespace.

Error messages

After running kubectl -n apigee get apigeeissues , the following error is displayed:

NAME                                          SEVERITY   REASON                      DOCUMENTATION                                                                           AGE
ingress-cert-mismatch-my-org-my-virtualhost   ERROR      AIS_INGRESS_CERT_MISMATCH   https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_INGRESS_CERT_MISMATCH   5h18m

Cause: Ingress cert mismatch

The private key and certificate stored in the Kubernetes secret referenced by an ApigeeRouteConfig do not match.

Diagnosis

Run the following command:

kubectl -n apigee describe apigeeissue ISSUE_NAME 

Where ISSUE_NAME is the name of the issue. For example, ingress-cert-mismatch-my-org-my-virtualhost .

Something similar to the following is returned:

 Name 
 : 
  
 ingress 
 - 
 cert 
 - 
 mismatch 
 - 
 my 
 - 
 org 
 - 
 my 
 - 
 virtualhost 
 Namespace 
 : 
  
 apigee 
 Labels 
 : 
  
 ais 
 - 
 reason 
 = 
 AIS_INGRESS_CERT_MISMATCH 
 Annotations 
 : 
  
< none 
 > 
 API 
  
 Version 
 : 
  
 apigee 
 . 
 cloud 
 . 
 google 
 . 
 com 
 / 
 v1alpha1 
 Kind 
 : 
  
 ApigeeIssue 
 Metadata 
 : 
  
 Creation 
  
 Timestamp 
 : 
  
 2023 
 - 
 06 
 - 
 12 
 T17 
 : 
 03 
 : 
 43 
 Z 
  
 Generation 
 : 
  
 1 
  
 Owner 
  
 References 
 : 
  
 API 
  
 Version 
 : 
  
 apigee 
 . 
 cloud 
 . 
 google 
 . 
 com 
 / 
 v1alpha2 
  
 Kind 
 : 
  
 ApigeeOrganization 
  
 Name 
 : 
  
 my 
 - 
 org 
  
 UID 
 : 
  
 7 
 e83a52c 
 - 
 ce00 
 - 
 4 
 bed 
 - 
 98 
 be 
 - 
 55835 
 ada1817 
  
 Resource 
  
 Version 
 : 
  
 3281563 
  
 UID 
 : 
  
 adc775c2 
 - 
 376d 
 - 
 4 
 bf9 
 - 
 9860 
 - 
 500 
 b2b2b8273 
 Spec 
 : 
  
 Details 
 : 
  
 Ingress 
  
 cert 
  
 and 
  
 private 
  
 key 
  
 in 
  
 secret 
  
 "my-org-my-virtualhost" 
  
 for 
  
 ApigeeRouteConfig 
  
 "my-org-my-virtualhost" 
  
 do 
  
 not 
  
 match 
  
 Documentation 
 : 
  
 https 
 : 
 // 
 cloud 
 . 
 google 
 . 
 com 
 / 
 apigee 
 / 
 docs 
 / 
 hybrid 
 / 
 troubleshooting 
 / 
 AIS_INGRESS_CERT_MISMATCH 
  
 Reason 
 : 
  
 AIS_INGRESS_CERT_MISMATCH 
  
 Severity 
 : 
  
 ERROR 
 Events 
 : 
  
< none 
 > 
 Spec 
 : 
  
 Details 
 : 
  
 Ingress 
  
 cert 
  
 and 
  
 private 
  
 key 
  
 in 
  
 secret 
  
 "my-org-my-virtualhost" 
  
 for 
  
 ApigeeRouteConfig 
  
 "my-org-my-virtualhost" 
  
 do 
  
 not 
  
 match 

The contents of Spec.Details explains the name of the ApigeeRouteConfig that refers to the Kubernetes secret containing the ingress certificate and private key.

Resolution

  1. Use kubectl describe to display the name of the ApigeeRouteConfig which has the mismatched ingress cert and key:

    kubectl -n apigee describe apigeeissue ISSUE_NAME 
    

    Where ISSUE_NAME is the name of the issue. For example, ingress-cert-mismatch-my-org-my-virtualhost .

    In this example, the ApigeeRouteConfig my-org-my-virtualhost is referencing the secret my-org-my-virtualhost .

  2. Determine the virtualhost name from the ApigeeRouteConfig .

    The ApigeeRouteConfig 's name is in the format: <Apigee organization>-<virtualhost name> .

    In this example, my-org is the Apigee organization and the virtualhost name is my-virtualhost .

  3. Find the corresponding virtualhost in your overrides.yaml file:

    virtualhosts:
    ...
    - name: my-virtualhost
      selector:
        app: apigee-ingressgateway
      sslCertPath: ./certs/vhost.pem
      sslKeyPath: ./certs/vhost.key
    ...
  4. Validate that the certificate and key files being used match.

    Using the sslCertPath and sslKeyPath from the previous step, validate the MD5 sums match using openssl :

    openssl x509 -noout -modulus -in SSL_CERT_PATH 
    | openssl md5
    openssl rsa -noout -modulus -in SSL_KEY_PATH 
    | openssl md5
    • Where SSL_CERT_PATH is the sslCertPath from the previous step. For example, ./certs/vhost.pem .
    • Where SSL_KEY_PATH is the sslKeyPath from the previous step. For example, ./certs/vhost.key .

    The output of the above two commands should match:

    openssl x509 -noout -modulus -in ./certs/vhost.pem | openssl md5
    MD5(stdin)= 40ba189dbe9d6fbfbdde1a8ef803d2d4
    openssl rsa -noout -modulus -in ./certs/vhost.key | openssl md5
    MD5(stdin)= 40ba189dbe9d6fbfbdde1a8ef803d2d4
  5. If the output from the previous commands did not match, use one of the following methods to correct the problem:

    • Update the virtualhost's sslCertPath and sslKeyPath to point to the correct file paths.
    • Fix the sslCertPath or sslKeyPath file contents to contain the correct matching certificate and key pair.
  6. Apply the changes to the virtualhost using Helm :

    helm upgrade ENV_GROUP_NAME 
    apigee-virtualhost/ \
      --namespace APIGEE_NAMESPACE 
    \
      --atomic \
      --set envgroup= ENV_GROUP_NAME 
    \
      -f OVERRIDES_FILE 
    

    Make sure to include all of the settings shown, including --atomic so that the action rolls back on failure.

Must gather diagnostic information

If the problem persists even after following the above instructions, gather the following diagnostic information and then contact Google Cloud Customer Care :

  1. The Google Cloud Project ID.
  2. The name of the Apigee hybrid organization.
  3. The Spec.Details field of the ApigeeIssue .
  4. (optional) Files referenced by sslCertPath and sslKeyPath for the affected virtualhost.
Create a Mobile Website
View Site in Mobile | Classic
Share by: