Console
    Contact Us Start free

    Ingress mTLS CA cert invalid

    You're viewing Apigee and Apigee hybrid documentation.
    There is no equivalent Apigee Edge documentation for this topic.

    Symptom

    An ApigeeIssue with Reason AIS_INGRESS_MTLS_CA_CERT_INVALID is shown when viewing resources in the apigee namespace.

    Error messages

    After running kubectl -n apigee get apigeeissues , an error like the following is displayed:

    NAME                                    SEVERITY   REASON                             DOCUMENTATION                                                                                  AGE
ca-cert-invalid-my-org-my-virtualhost   ERROR      AIS_INGRESS_MTLS_CA_CERT_INVALID   https://cloud.google.com/apigee/docs/hybrid/troubleshooting/AIS_INGRESS_MTLS_CA_CERT_INVALID   5h18m

    Possible causes

    Cause Description
    Ingress mTLS CA certificate is not PEM-encoded The configured ingress mTLS CA certificate is not PEM-encoded.
    Ingress mTLS CA certificate is malformed The configured ingress mTLS CA certificate is malformed.

    Cause: Ingress mTLS CA certificate is not PEM-encoded

    The ingress mTLS CA certificate stored in the Kubernetes secret referenced by an ApigeeRouteConfig is missing the PEM-encoding text blocks (or they are malformed).

    Diagnosis

    Run the following command:

    kubectl -n apigee describe apigeeissue ISSUE_NAME

    Where ISSUE_NAME is the name of the issue. For example, ca-cert-invalid-my-org-my-virtualhost .

    The output should look similar to the following:

     Name 
 : 
  
 ca 
 - 
 cert 
 - 
 invalid 
 - 
 my 
 - 
 org 
 - 
 my 
 - 
 virtualhost 
 Namespace 
 : 
  
 apigee 
 Labels 
 : 
  
 ais 
 - 
 reason 
 = 
 AIS_INGRESS_MTLS_CA_CERT_INVALID 
 Annotations 
 : 
  
< none 
 > 
 API 
  
 Version 
 : 
  
 apigee 
 . 
 cloud 
 . 
 google 
 . 
 com 
 / 
 v1alpha1 
 Kind 
 : 
  
 ApigeeIssue 
 Metadata 
 : 
  
 Creation 
  
 Timestamp 
 : 
  
 2023 
 - 
 06 
 - 
 12 
 T17 
 : 
 03 
 : 
 43 
 Z 
  
 Generation 
 : 
  
 1 
  
 Owner 
  
 References 
 : 
  
 API 
  
 Version 
 : 
  
 apigee 
 . 
 cloud 
 . 
 google 
 . 
 com 
 / 
 v1alpha2 
  
 Kind 
 : 
  
 ApigeeOrganization 
  
 Name 
 : 
  
 my 
 - 
 org 
  
 UID 
 : 
  
 7 
 e83a52c 
 - 
 ce00 
 - 
 4 
 bed 
 - 
 98 
 be 
 - 
 55835 
 ada1817 
  
 Resource 
  
 Version 
 : 
  
 3281563 
  
 UID 
 : 
  
 adc775c2 
 - 
 376d 
 - 
 4 
 bf9 
 - 
 9860 
 - 
 500 
 b2b2b8273 
 Spec 
 : 
  
 Details 
 : 
  
 CA 
  
 certificate 
  
 in 
  
 secret 
  
 "my-org-my-virtualhost" 
  
 for 
  
 ApigeeRouteConfig 
  
 "my-org-my-virtualhost" 
  
 is 
  
 not 
  
 PEM 
  
 encoded 
  
 Documentation 
 : 
  
 https 
 : 
 // 
 cloud 
 . 
 google 
 . 
 com 
 / 
 apigee 
 / 
 docs 
 / 
 hybrid 
 / 
 troubleshooting 
 / 
 AIS_INGRESS_MTLS_CA_CERT_INVALID 
  
 Reason 
 : 
  
 AIS_INGRESS_MTLS_CA_CERT_INVALID 
  
 Severity 
 : 
  
 ERROR 
 Events 
 : 
  
< none 
 > 

     Spec 
 : 
  
 Details 
 : 
  
 CA 
  
 certificate 
  
 in 
  
 secret 
  
 "my-org-my-virtualhost" 
  
 for 
  
 ApigeeRouteConfig 
  
 "my-org-my-virtualhost" 
  
 is 
  
 not 
  
 PEM 
  
 encoded

    The contents of Spec.Details explains the name of the ApigeeRouteConfig that refers to the Kubernetes secret containing the ingress mTLS CA certificate which is not PEM-encoded.

    Resolution

    1. Use kubectl describe to display the name of the ApigeeRouteConfig which has the ingress mTLS CA cert which is not PEM-encoded:

      kubectl -n apigee describe apigeeissue ISSUE_NAME

      Where ISSUE_NAME is the name of the issue. For example, ca-cert-invalid-my-org-my-virtualhost .

      In this example, the ApigeeRouteConfig my-org-my-virtualhost is referencing the secret my-org-my-virtualhost .

    2. Determine the virtualhost name from the ApigeeRouteConfig .

      The ApigeeRouteConfig 's name is in the format: <Apigee organization>-<virtualhost name> .

      In this example, my-org is the Apigee organization and the virtualhost name is my-virtualhost .

    3. Find the corresponding virtualhost in your overrides.yaml .

      virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  caCertPath: ./certs/ca.pem
  sslCertPath: ./certs/vhost.pem
  sslKeyPath: ./certs/vhost.key
...

    4. Validate the contents of the file provided via the path declared in caCertPath . Ensure the content contains the required starting and ending certificate blocks. For example:

      -----BEGIN CERTIFICATE-----
MIIDYTCCAkmgAwIBAgIUSXeU0pQYRFzYlqZpKhNNJdBLFBIwDQYJKoZIhvcNAQEL
... <contents omitted> ...
eWJyaWQuZTJlLmFwaWdlZWtzLm5ldDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
MMM=
-----END CERTIFICATE-----

    5. Once the contents and format of the file have been fixed, apply the change to the virtualhost:

      helm upgrade ENV_GROUP_NAME 
apigee-virtualhost/ \
  --namespace apigee \
  --set envgroup= ENV_GROUP_NAME 
\
  -f overrides.yaml

    Cause: Ingress mTLS CA certificate is malformed

    The ingress mTLS CA certificate stored in the Kubernetes secret referenced by an ApigeeRouteConfig is malformed.

    Diagnosis

    Run the following command:

    kubectl -n apigee describe apigeeissue ISSUE_NAME

    Where ISSUE_NAME is the name of the issue. For example, ca-cert-invalid-my-org-my-virtualhost .

    The output should look similar to the following:

     Name 
 : 
  
 ca 
 - 
 cert 
 - 
 invalid 
 - 
 my 
 - 
 org 
 - 
 my 
 - 
 virtualhost 
 Namespace 
 : 
  
 apigee 
 Labels 
 : 
  
 ais 
 - 
 reason 
 = 
 AIS_INGRESS_MTLS_CA_CERT_INVALID 
 Annotations 
 : 
  
< none 
 > 
 API 
  
 Version 
 : 
  
 apigee 
 . 
 cloud 
 . 
 google 
 . 
 com 
 / 
 v1alpha1 
 Kind 
 : 
  
 ApigeeIssue 
 Metadata 
 : 
  
 Creation 
  
 Timestamp 
 : 
  
 2023 
 - 
 06 
 - 
 12 
 T17 
 : 
 03 
 : 
 43 
 Z 
  
 Generation 
 : 
  
 1 
  
 Owner 
  
 References 
 : 
  
 API 
  
 Version 
 : 
  
 apigee 
 . 
 cloud 
 . 
 google 
 . 
 com 
 / 
 v1alpha2 
  
 Kind 
 : 
  
 ApigeeOrganization 
  
 Name 
 : 
  
 my 
 - 
 org 
  
 UID 
 : 
  
 7 
 e83a52c 
 - 
 ce00 
 - 
 4 
 bed 
 - 
 98 
 be 
 - 
 55835 
 ada1817 
  
 Resource 
  
 Version 
 : 
  
 3281563 
  
 UID 
 : 
  
 adc775c2 
 - 
 376d 
 - 
 4 
 bf9 
 - 
 9860 
 - 
 500 
 b2b2b8273 
 Spec 
 : 
  
 Details 
 : 
  
 CA 
  
 certificate 
  
 in 
  
 secret 
  
 "my-org-my-virtualhost" 
  
 for 
  
 ApigeeRouteConfig 
  
 "my-org-my-virtualhost" 
  
 is 
  
 invalid 
  
 Documentation 
 : 
  
 https 
 : 
 // 
 cloud 
 . 
 google 
 . 
 com 
 / 
 apigee 
 / 
 docs 
 / 
 hybrid 
 / 
 troubleshooting 
 / 
 AIS_INGRESS_MTLS_CA_CERT_INVALID 
  
 Reason 
 : 
  
 AIS_INGRESS_MTLS_CA_CERT_INVALID 
  
 Severity 
 : 
  
 ERROR 
 Events 
 : 
  
< none 
 > 

     Spec 
 : 
  
 Details 
 : 
  
 CA 
  
 certificate 
  
 in 
  
 secret 
  
 "my-org-my-virtualhost" 
  
 for 
  
 ApigeeRouteConfig 
  
 "my-org-my-virtualhost" 
  
 is 
  
 invalid

    The contents of Spec.Details explains the name of the ApigeeRouteConfig that refers to the Kubernetes secret containing the ingress mTLS CA certificate which is malformed and failed to parse.

    Resolution

    1. Use kubectl describe to display the name of the ApigeeRouteConfig which has the ingress mTLS CA cert which is malformed:

      kubectl -n apigee describe apigeeissue ISSUE_NAME

      Where ISSUE_NAME is the name of the issue. For example, ca-cert-invalid-my-org-my-virtualhost .

      In this example, the ApigeeRouteConfig my-org-my-virtualhost is referencing the secret my-org-my-virtualhost .

    2. Determine the virtualhost name from the ApigeeRouteConfig .

      The ApigeeRouteConfig 's name is in the format: <Apigee organization>-<virtualhost name> .

      In this example, my-org is the Apigee organization and the virtualhost name is my-virtualhost .

    3. Find the corresponding virtualhost in your overrides.yaml .

      virtualhosts:
...
- name: my-virtualhost
  selector:
    app: apigee-ingressgateway
  caCertPath: ./certs/ca.pem
  sslCertPath: ./certs/vhost.pem
  sslKeyPath: ./certs/vhost.key
...

    4. Validate the contents of the file provided via the path declared in caCertPath . Ensure the contents contain the entire certificate. For example:

      -----BEGIN CERTIFICATE-----
MIIDYTCCAkmgAwIBAgIUSXeU0pQYRFzYlqZpKhNNJdBLFBIwDQYJKoZIhvcNAQEL
BQAwQDE+MDwGA1UEAww1bWF4bWlsbGlvbi1oeWJyaWQtc3RhZ2luZy1kZXYuaHli
... <contents omitted> ...
0TPPaik8U9dtFXMGT1DJkjPRPO0Jw5rlU8DKlNA/Kkv52aKJZNwB/cwmvoa/BFji
PIPa9wY=
-----END CERTIFICATE-----

    5. Once the contents and format of the file have been fixed, apply the change to the virtualhost:

      helm upgrade ENV_GROUP_NAME 
apigee-virtualhost/ \
  --namespace apigee \
  --set envgroup= ENV_GROUP_NAME 
\
  -f overrides.yaml

    Must gather diagnostic information

    If the problem persists even after following the above instructions, gather the following diagnostic information and then contact Google Cloud Customer Care :

    1. The Google Cloud Project ID.
    2. The name of the Apigee hybrid organization.
    3. The Spec.Details field of the ApigeeIssue .
    4. (optional) File referenced by caCertPath for the affected virtualhost.
    Create a Mobile Website
    View Site in Mobile | Classic
    Share by: