Set up Elastic Stack on GKE

Prepare the environment

In this tutorial, you use Cloud Shell to manage resources hosted on Google Cloud. Cloud Shell is preinstalled with the software you need for this tutorial, including kubectl , Helm , and the gcloud CLI .

To set up your environment with Cloud Shell, follow these steps:

1. Launch a Cloud Shell session from the Google Cloud console, by clicking Activate Cloud Shellin the Google Cloud console . This launches a session in the bottom pane of the Google Cloud console.

2. Add a Helm chart repository and update it:

    helm  
   repo  
   add  
   elastic  
   https://helm.elastic.co
   helm  
   repo  
   update 
   

3. Clone the GitHub repository:

    git  
   clone  
   https://github.com/GoogleCloudPlatform/kubernetes-engine-samples.git 
   

4. Change to the working directory:

     cd 
     
   kubernetes-engine-samples/observability/elastic-stack-tutorial 
   

Create a GKE cluster

Create a GKE cluster with control plane metrics collection enabled:

 gcloud  
container  
clusters  
create-auto  
elk-stack  
 \ 
  
--location = 
 "us-central1" 
  
 \ 
  
--monitoring = 
 "SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER" 
 

Deploy the ECK operator

Elastic Cloud on Kubernetes (ECK) is a platform for deploying and managing the Elastic Stack on Kubernetes clusters.

ECK automates the deployment and management of Elastic Stack clusters, simplifying the process of setting up and maintaining Elastic Stack on Kubernetes. It provides a set of Kubernetes custom resources that you can use to create and configure Elasticsearch, Kibana, Application Performance Management Server, and other Elastic Stack components in Kubernetes. This lets developers and DevOps teams configure and manage Elastic Stack clusters at scale.

ECK supports multiple Elasticsearch nodes, automatic application failover, seamless upgrades, and SSL encryption. ECK also includes features that let you monitor and troubleshoot Elasticsearch performance.

1. Install the ECK Helm chart:

    helm  
   upgrade  
   --install  
    "elastic-operator" 
     
    "elastic/eck-operator" 
     
    \ 
     
   --version = 
    "2.8.0" 
     
    \ 
     
   --create-namespace  
    \ 
     
   --namespace = 
    "elastic-system" 
     
    \ 
     
   --set = 
    "resources.limits.cpu=250m" 
     
    \ 
     
   --set = 
    "resources.limits.memory=512Mi" 
     
    \ 
     
   --set = 
    "resources.limits.ephemeral-storage=1Gi" 
     
    \ 
     
   --set = 
    "resources.requests.cpu=250m" 
     
    \ 
     
   --set = 
    "resources.requests.memory=512Mi" 
     
    \ 
     
   --set = 
    "resources.requests.ephemeral-storage=1Gi" 
    
   

2. Wait for the operator to be ready:

    watch  
   kubectl  
   get  
   pods  
   -n  
   elastic-system 
   

   The output is similar to the following:

    NAME                 READY   STATUS    RESTARTS   AGE
   elastic-operator-0   1/1     Running   0          31s 
   

   When the operator STATUS is Running , return to the command line by pressing Ctrl+C .

Configure Elastic Stack with ECK

By using Elastic Stack with Elasticsearch, Kibana, and Elastic Agent working in Fleet mode, you can set up a powerful, scalable, and fully-managed solution for managing and visualizing data using Kibana.

Kibana is an open source data analytics and visualization tool that lets you search, analyze and visualize data in Elasticsearch.

Elastic Agent is a lightweight data shipper that collects data from different sources, such as logs or metrics, and automatically sends it to Elasticsearch.

Elastic Fleet is a mode of operation in which Elastic agents report to a central fleet server, which handles their configuration and management. The fleet server simplifies the deployment, configuration, and scaling of Elastic agents, making it easier to manage large and complex deployments.

Elasticsearch autoscaling is a self-monitoring feature that can report when additional resources are needed based on an operator-defined policy. For example, a policy might specify that a certain tier should scale based on available disk space. Elasticsearch can monitor the disk space and suggest scaling if it predicts a shortage, although it is still up to the operator to add the necessary resources. For more information about Elasticsearch autoscaling see Autoscaling in the Elasticsearch documentation.

Configure an Elasticsearch cluster

Elasticsearch provides a distributed, RESTful search and analytics engine designed to store and search large volumes of data quickly and efficiently.

When deploying Elastic Stack on Kubernetes, you should manage the VM settings, specifically the vm.max_map_count setting , which is required by Elasticsearch. vm.max_map_count specifies the number of memory areas that a process can allocate to a file. Elasticsearch must have this value set to at least 262144 to run optimally. For more information, see Virtual memory in the ECK documentation.

1. Review the following manifest:

     # Copyright 2023 Google LLC 
    # 
    # Licensed under the Apache License, Version 2.0 (the "License"); 
    # you may not use this file except in compliance with the License. 
    # You may obtain a copy of the License at 
    # 
    #      https://www.apache.org/licenses/LICENSE-2.0 
    # 
    # Unless required by applicable law or agreed to in writing, software 
    # distributed under the License is distributed on an "AS IS" BASIS, 
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 
    # See the License for the specific language governing permissions and 
    # limitations under the License. 
    apiVersion 
    : 
     
    scheduling.k8s.io/v1 
    kind 
    : 
     
    PriorityClass 
    metadata 
    : 
     
    name 
    : 
     
    user-daemonset-priority 
    value 
    : 
     
    999999999 
    preemptionPolicy 
    : 
     
    PreemptLowerPriority 
    globalDefault 
    : 
     
    false 
    description 
    : 
     
    "User 
     
    DaemonSet 
     
    priority" 
    --- 
    apiVersion 
    : 
     
    apps/v1 
    kind 
    : 
     
    DaemonSet 
    metadata 
    : 
     
    name 
    : 
     
    max-map-count-setter 
     
    namespace 
    : 
     
    elastic-system 
     
    labels 
    : 
     
    k8s-app 
    : 
     
    max-map-count-setter 
    spec 
    : 
     
    selector 
    : 
     
    matchLabels 
    : 
     
    name 
    : 
     
    max-map-count-setter 
     
    template 
    : 
     
    metadata 
    : 
     
    labels 
    : 
     
    name 
    : 
     
    max-map-count-setter 
     
    spec 
    : 
     
    priorityClassName 
    : 
     
    user-daemonset-priority 
     
    nodeSelector 
    : 
     
    cloud.google.com/compute-class 
    : 
     
    "Balanced" 
     
    initContainers 
    : 
     
    - 
     
    name 
    : 
     
    max-map-count-setter 
     
    image 
    : 
     
    docker.io/bash:5.2.15 
     
    resources 
    : 
     
    requests 
    : 
     
    cpu 
    : 
     
    10m 
     
    memory 
    : 
     
    10Mi 
     
    ephemeral-storage 
    : 
     
    10Mi 
     
    limits 
    : 
     
    cpu 
    : 
     
    50m 
     
    memory 
    : 
     
    32Mi 
     
    ephemeral-storage 
    : 
     
    10Mi 
     
    securityContext 
    : 
     
    privileged 
    : 
     
    true 
     
    runAsUser 
    : 
     
    0 
     
    command 
    : 
     
    [ 
    "/usr/local/bin/bash" 
    , 
     
    "-e" 
    , 
     
    "-c" 
    , 
     
    "echo 
     
    262144 
    > 
    /proc/sys/vm/max_map_count" 
    ] 
     
    containers 
    : 
     
    - 
     
    name 
    : 
     
    sleep 
     
    image 
    : 
     
    docker.io/bash:5.2.15 
     
    command 
    : 
     
    [ 
    "sleep" 
    , 
     
    "infinity" 
    ] 
     
    resources 
    : 
     
    requests 
    : 
     
    cpu 
    : 
     
    10m 
     
    memory 
    : 
     
    10Mi 
     
    ephemeral-storage 
    : 
     
    10Mi 
     
    limits 
    : 
     
    cpu 
    : 
     
    10m 
     
    memory 
    : 
     
    10Mi 
     
    ephemeral-storage 
    : 
     
    10Mi 
    
   

   This manifest describes a DaemonSet that configures the kernel setting on the host directly. A DaemonSet is a Kubernetes controller that ensures that a copy of a Pod runs on each node in a cluster.

   The preceding manifest is on an allowlist to run on Autopilot. Don't modify this manifest, including the container images.

2. Apply this manifest to your cluster:

    kubectl  
   apply  
   -f  
   max-map-count-setter-ds.yaml 
   

3. Review the following manifest:

     apiVersion 
    : 
     
    elasticsearch.k8s.elastic.co/v1 
    kind 
    : 
     
    Elasticsearch 
    metadata 
    : 
     
    name 
    : 
     
    elasticsearch 
     
    namespace 
    : 
     
    elastic-system 
    spec 
    : 
     
    version 
    : 
     
    "8.9.0" 
     
    volumeClaimDeletePolicy 
    : 
     
    DeleteOnScaledownOnly 
     
    podDisruptionBudget 
    : 
     
    spec 
    : 
     
    minAvailable 
    : 
     
    2 
     
    selector 
    : 
     
    matchLabels 
    : 
     
    elasticsearch.k8s.elastic.co/cluster-name 
    : 
     
    elasticsearch 
     
    nodeSets 
    : 
     
    - 
     
    name 
    : 
     
    default 
     
    config 
    : 
     
    node.roles 
    : 
     
    [ 
    "master" 
    , 
     
    "data" 
    , 
     
    "ingest" 
    , 
     
    "ml" 
    , 
     
    "remote_cluster_client" 
    ] 
     
    podTemplate 
    : 
     
    metadata 
    : 
     
    labels 
    : 
     
    app.kubernetes.io/name 
    : 
     
    elasticsearch 
     
    app.kubernetes.io/version 
    : 
     
    "8.9.0" 
     
    app.kubernetes.io/component 
    : 
     
    "elasticsearch" 
     
    app.kubernetes.io/part-of 
    : 
     
    "elk" 
     
    spec 
    : 
     
    nodeSelector 
    : 
     
    cloud.google.com/compute-class 
    : 
     
    "Balanced" 
     
    initContainers 
    : 
     
    - 
     
    name 
    : 
     
    max-map-count-check 
     
    command 
    : 
     
    - 
     
    sh 
     
    - 
     
    -c 
     
    - 
     
    while true; do mmc=$(cat /proc/sys/vm/max_map_count); if test ${mmc} -eq 262144; then exit 0; fi; sleep 1; done 
     
    resources 
    : 
     
    requests 
    : 
     
    cpu 
    : 
     
    10m 
     
    memory 
    : 
     
    16Mi 
     
    ephemeral-storage 
    : 
     
    16Mi 
     
    limits 
    : 
     
    cpu 
    : 
     
    10m 
     
    memory 
    : 
     
    16Mi 
     
    ephemeral-storage 
    : 
     
    16Mi 
     
    containers 
    : 
     
    - 
     
    name 
    : 
     
    elasticsearch 
     
    resources 
    : 
     
    requests 
    : 
     
    cpu 
    : 
     
    990m 
     
    memory 
    : 
     
    4080Mi 
     
    ephemeral-storage 
    : 
     
    1008Mi 
     
    limits 
    : 
     
    cpu 
    : 
     
    1000m 
     
    memory 
    : 
     
    4080Mi 
     
    ephemeral-storage 
    : 
     
    1008Mi 
     
    env 
    : 
     
    - 
     
    name 
    : 
     
    ES_JAVA_OPTS 
     
    value 
    : 
     
    "-Xms2g 
     
    -Xmx2g" 
     
    count 
    : 
     
    3 
     
    volumeClaimTemplates 
    : 
     
    - 
     
    metadata 
    : 
     
    name 
    : 
     
    elasticsearch-data 
     
    # Do not change this name unless you set up a volume mount for the data path. 
     
    spec 
    : 
     
    accessModes 
    : 
     
    - 
     
    ReadWriteOnce 
     
    resources 
    : 
     
    requests 
    : 
     
    storage 
    : 
     
    2Gi 
     
    storageClassName 
    : 
     
    standard-rwo 
    
   

   This manifest defines an Elasticsearch cluster with the following fields:

   • initContainers : waits for the virtual memory host's kernel settings to change.
   • podDisruptionBudget : specifies that the cluster won't be destroyed during the Pods' defragmentation process.
   • config.node.roles : Elasticsearch node roles configuration. For more information about node roles, see Node in the Elasticsearch documentation.

4. Apply this manifest to your cluster:

    kubectl  
   apply  
   -f  
   elasticsearch.yaml 
   

5. Wait for the Elasticsearch cluster to be ready:

    watch  
   kubectl  
   --namespace  
   elastic-system  
   get  
   elasticsearches.elasticsearch.k8s.elastic.co 
   

   The output is similar to the following:

    NAME            HEALTH   NODES   VERSION   PHASE   AGE
   elasticsearch   green    3       8.8.0     Ready   5m3s 
   

   When the Elasticsearch cluster HEALTH is green and PHASE is Ready , return to the command line by pressing Ctrl+C .

Configure Kibana

1. Review the following manifest:

     apiVersion 
    : 
     
    kibana.k8s.elastic.co/v1 
    kind 
    : 
     
    Kibana 
    metadata 
    : 
     
    name 
    : 
     
    kibana 
     
    namespace 
    : 
     
    elastic-system 
    spec 
    : 
     
    version 
    : 
     
    "8.9.0" 
     
    count 
    : 
     
    1 
     
    elasticsearchRef 
    : 
     
    name 
    : 
     
    elasticsearch 
     
    namespace 
    : 
     
    elastic-system 
     
    http 
    : 
     
    tls 
    : 
     
    selfSignedCertificate 
    : 
     
    disabled 
    : 
     
    true 
     
    config 
    : 
     
    server.publicBaseUrl 
    : 
     
    https://elk.BASE_DOMAIN 
     
    xpack.reporting.kibanaServer.port 
    : 
     
    5601 
     
    xpack.reporting.kibanaServer.protocol 
    : 
     
    http 
     
    xpack.reporting.kibanaServer.hostname 
    : 
     
    kibana-kb-http.elastic-system.svc 
     
    xpack.fleet.agents.elasticsearch.hosts 
    : 
     
    [ 
    "https://elasticsearch-es-http.elastic-system.svc:9200" 
    ] 
     
    xpack.fleet.agents.fleet_server.hosts 
    : 
     
    [ 
    "https://fleet-server-agent-http.elastic-system.svc:8220" 
    ] 
     
    xpack.fleet.packages 
    : 
     
    - 
     
    name 
    : 
     
    system 
     
    version 
    : 
     
    latest 
     
    - 
     
    name 
    : 
     
    elastic_agent 
     
    version 
    : 
     
    latest 
     
    - 
     
    name 
    : 
     
    fleet_server 
     
    version 
    : 
     
    latest 
     
    - 
     
    name 
    : 
     
    kubernetes 
     
    version 
    : 
     
    latest 
     
    xpack.fleet.agentPolicies 
    : 
     
    - 
     
    name 
    : 
     
    Fleet Server on ECK policy 
     
    id 
    : 
     
    eck-fleet-server 
     
    namespace 
    : 
     
    default 
     
    monitoring_enabled 
    : 
     
    - 
     
    logs 
     
    - 
     
    metrics 
     
    unenroll_timeout 
    : 
     
    900 
     
    package_policies 
    : 
     
    - 
     
    name 
    : 
     
    fleet_server-1 
     
    id 
    : 
     
    fleet_server-1 
     
    package 
    : 
     
    name 
    : 
     
    fleet_server 
     
    - 
     
    name 
    : 
     
    Elastic Agent on ECK policy 
     
    id 
    : 
     
    eck-agent 
     
    namespace 
    : 
     
    default 
     
    monitoring_enabled 
    : 
     
    - 
     
    logs 
     
    - 
     
    metrics 
     
    unenroll_timeout 
    : 
     
    900 
     
    package_policies 
    : 
     
    - 
     
    package 
    : 
     
    name 
    : 
     
    system 
     
    name 
    : 
     
    system-1 
     
    - 
     
    package 
    : 
     
    name 
    : 
     
    kubernetes 
     
    name 
    : 
     
    kubernetes-1 
     
    podTemplate 
    : 
     
    metadata 
    : 
     
    labels 
    : 
     
    app.kubernetes.io/name 
    : 
     
    kibana 
     
    app.kubernetes.io/version 
    : 
     
    "8.9.0" 
     
    app.kubernetes.io/component 
    : 
     
    "ui" 
     
    app.kubernetes.io/part-of 
    : 
     
    "elk" 
     
    spec 
    : 
     
    containers 
    : 
     
    - 
     
    name 
    : 
     
    kibana 
     
    resources 
    : 
     
    requests 
    : 
     
    memory 
    : 
     
    1Gi 
     
    cpu 
    : 
     
    500m 
     
    ephemeral-storage 
    : 
     
    1Gi 
     
    limits 
    : 
     
    memory 
    : 
     
    1Gi 
     
    cpu 
    : 
     
    500m 
     
    ephemeral-storage 
    : 
     
    1Gi 
    
   

   This manifest describes a Kibana custom resource that configures agent policies for the fleet server and agents.

2. Apply this manifest to your cluster:

    kubectl  
   apply  
   -f  
   kibana.yaml 
   

3. Wait for the Pods to be ready:

    watch  
   kubectl  
   --namespace  
   elastic-system  
   get  
   kibanas.kibana.k8s.elastic.co 
   

   The output is similar to the following:

    NAME     HEALTH   NODES   VERSION   AGE
   kibana   green    1       8.8.0     6m47s 
   

   When the Pods HEALTH is green , return to the command line by pressing Ctrl+C .

Configure a load balancer to access Kibana

To access Kibana, create a Kubernetes Ingress object, a Google-managed certificate, a global IP address, and a DNS Zone.

1. Create global external IP address:

    gcloud  
   compute  
   addresses  
   create  
    "elastic-stack" 
     
   --global 
   

2. Create a managed zone and record set in Cloud DNS:

    gcloud  
   dns  
   managed-zones  
   create  
    "elk" 
     
    \ 
     
   --description = 
    "DNS Zone for Airflow" 
     
    \ 
     
   --dns-name = 
    "elk. BASE_DOMAIN 
   " 
     
    \ 
     
   --visibility = 
    "public" 
   gcloud  
   dns  
   record-sets  
   create  
    "elk. BASE_DOMAIN 
   " 
     
    \ 
     
   --rrdatas = 
    " 
    $( 
   gcloud  
   compute  
   addresses  
   describe  
    "elastic-stack" 
     
   --global  
   --format = 
    "value(address)" 
    ) 
    " 
     
    \ 
     
   --ttl = 
    "300" 
     
    \ 
     
   --type = 
    "A" 
     
    \ 
     
   --zone = 
    "elk" 
    
   

3. Delegate the DNS zone as a subdomain of the base domain by creating an NS record set with a name servers list. You can get a list of name servers using the following command:

    gcloud  
   dns  
   record-sets  
   describe  
   elk. BASE_DOMAIN 
     
    \ 
     
   --type = 
    "NS" 
     
    \ 
     
   --zone = 
    "elk" 
     
    \ 
     
   --format = 
    "value(DATA)" 
    
   

4. Review the following manifest:

     apiVersion 
    : 
     
    networking.gke.io/v1 
    kind 
    : 
     
    ManagedCertificate 
    metadata 
    : 
     
    name 
    : 
     
    elastic-stack 
     
    namespace 
    : 
     
    elastic-system 
    spec 
    : 
     
    domains 
    : 
     
    - 
     
    elk.BASE_DOMAIN 
    
   

   This manifest describes a ManagedCertificate that provisions an SSL certificate to establish the TLS connection.

5. Apply the manifest to your cluster:

    kubectl  
   apply  
   -f  
   ingress.yaml 
   

Configure Elastic Agents

1. Review the following manifest:

     apiVersion 
    : 
     
    agent.k8s.elastic.co/v1alpha1 
    kind 
    : 
     
    Agent 
    metadata 
    : 
     
    name 
    : 
     
    fleet-server 
     
    namespace 
    : 
     
    elastic-system 
    spec 
    : 
     
    version 
    : 
     
    8.9.0 
     
    kibanaRef 
    : 
     
    name 
    : 
     
    kibana 
     
    namespace 
    : 
     
    elastic-system 
     
    elasticsearchRefs 
    : 
     
    - 
     
    name 
    : 
     
    elasticsearch 
     
    namespace 
    : 
     
    elastic-system 
     
    mode 
    : 
     
    fleet 
     
    fleetServerEnabled 
    : 
     
    true 
     
    policyID 
    : 
     
    eck-fleet-server 
     
    deployment 
    : 
     
    replicas 
    : 
     
    1 
     
    podTemplate 
    : 
     
    metadata 
    : 
     
    labels 
    : 
     
    app.kubernetes.io/name 
    : 
     
    fleet-server 
     
    app.kubernetes.io/version 
    : 
     
    "8.9.0" 
     
    app.kubernetes.io/component 
    : 
     
    "agent" 
     
    app.kubernetes.io/part-of 
    : 
     
    "elk" 
     
    spec 
    : 
     
    containers 
    : 
     
    - 
     
    name 
    : 
     
    agent 
     
    resources 
    : 
     
    requests 
    : 
     
    memory 
    : 
     
    512Mi 
     
    cpu 
    : 
     
    250m 
     
    ephemeral-storage 
    : 
     
    10Gi 
     
    limits 
    : 
     
    memory 
    : 
     
    512Mi 
     
    cpu 
    : 
     
    250m 
     
    ephemeral-storage 
    : 
     
    10Gi 
     
    volumes 
    : 
     
    - 
     
    name 
    : 
     
    "agent-data" 
     
    ephemeral 
    : 
     
    volumeClaimTemplate 
    : 
     
    spec 
    : 
     
    accessModes 
    : 
     
    [ 
    "ReadWriteOnce" 
    ] 
     
    storageClassName 
    : 
     
    "standard-rwo" 
     
    resources 
    : 
     
    requests 
    : 
     
    storage 
    : 
     
    10Gi 
     
    serviceAccountName 
    : 
     
    fleet-server 
     
    automountServiceAccountToken 
    : 
     
    true 
     
    securityContext 
    : 
     
    runAsUser 
    : 
     
    0 
    
   

   This manifest describes an Elastic Agent that configures a fleet server with ECK.

2. Apply this manifest to your cluster:

    kubectl  
   apply  
   -f  
   fleet-server-and-agents.yaml 
   

3. Wait for the Pods to be ready:

    watch  
   kubectl  
   --namespace  
   elastic-system  
   get  
   agents.agent.k8s.elastic.co 
   

   The output is similar to the following:

    NAME            HEALTH   AVAILABLE   EXPECTED   VERSION   AGE
   elastic-agent   green    5           5          8.8.0     14m
   fleet-server    green    1           1          8.8.0     16m 
   

   When the Pods HEALTH is green , return to the command line by pressing Ctrl+C .

Configure logging and monitoring

Elastic Stack can use the kube-state-metrics exporter to collect cluster-level metrics.

1. Install kube-state-metrics:

    helm  
   repo  
   add  
   prometheus-community  
   https://prometheus-community.github.io/helm-charts
   helm  
   repo  
   update
   helm  
   install  
   kube-state-metrics  
   prometheus-community/kube-state-metrics  
   --namespace  
   elastic-system 
   

2. Get the default Kibana elastic user credentials:

    kubectl  
   get  
   secret  
   elasticsearch-es-elastic-user  
   -o  
   yaml  
   -n  
   elastic-system  
   -o  
    jsonpath 
    = 
    '{.data.elastic}' 
     
    | 
     
   base64  
   -d 
   

3. Open https://elk. BASE_DOMAIN in your browser and login to Kibana with the credentials.

4. From the menu, select Analytics, then Dashboards.

5. In the search text field, enter Kubernetes overviewand select Overview dashboardto see base metrics.

   Some of the dashboard panels might show no data or error messages because GKE limits access to some of the control plane endpoints that Kibana uses to get cluster metrics.