ELK on Kubernetes with Helm Charts
25 May 2024 by dzlab
This post explains how to setup an ELK (Elasticsearch, Logstash, and Kibana) stack on Kubernetes using Helm Charts. We will first, setup Kubernetes cluster on Google Cloud using GKE, then install on it all the components of ELK using Helm Charts.
Setup
Setting up a k8s cluster on Google Cloud is fairly easy, we just need to follow the official documentation.
First, let’s set some global environment variables:
$ LOCATION=us-central1-a
$ CLUSTER_NAME=kubetest
By default GKE will create a k8s cluster with nodes having machine type e2-medium which is a medium-sized instance with 2 vCPUs and 4 GB of memory. We need instead to use a machine with more Memory. We can list the machine types available as follows:
$ gcloud compute machine-types list --filter="$LOCATION"
Pick a machine type with enough memory then start GKE cluster as follows:
$ gcloud container clusters create $CLUSTER_NAME \
--zone $LOCATION \
--node-locations $LOCATION \
--machine-type e2-standard-4
This will create GKE cluster
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
kubetest us-central1-a 1.28.8-gke.1095000 34.72.13.154 e2-standard-4 1.28.8-gke.1095000 3 RUNNING
Later to delete the cluster, we can simply do:
$ gcloud container clusters delete $CLUSTER_NAME --location $LOCATION
Deploy with Helm Charts
In this section, we will dive in details how to deploy the different components of our ELK stack on Kubernetes using Helm charts. Before going any further, it’s important to ensure that you have Helm installed on your local machine. For details on how to install Helm on your operating system refer to the documentation.
Add Elastic Helm Charts Repository: Elastic offers official Helm charts for deploying the ELK stack components. By adding the Elastic Helm repository to your Helm configuration, you attain access to these charts.
$ helm repo add elastic https://helm.elastic.co
"elastic" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "elastic" chart repository
Update Complete. ⎈Happy Helming!⎈
Create a k8s namespace where the ELK stack will be installed
$ kubectl create namespace monit
namespace/monit created
Elasticsearch
Elasticsearch is the core of the ELK stack, responsible for storing and indexing log data. When installing Elasticsearch with Helm, you can customize parameters such as the number of replicas (for high availability) and JVM heap size using Helm chart values.
$ helm install elasticsearch elastic/elasticsearch \
--namespace monit \
--set replicas=3 \
--set esJavaOpts="-Xmx512m -Xms512m"
Watch all cluster members come up.
$ kubectl get pods --namespace=monit -l app=elasticsearch-master -w
After few minutes the different replicates will be ready and the Elasticsearch cluster properly running
NAME READY STATUS RESTARTS AGE
elasticsearch-master-0 1/1 Running 0 9m20s
elasticsearch-master-1 1/1 Running 0 9m20s
elasticsearch-master-2 1/1 Running 0 9m20s
$ kubectl describe pod elasticsearch-master-0 -n monit
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 17m default-scheduler Successfully assigned monit/elasticsearch-master-0 to gke-kubetest-default-pool-ddfd6147-mqrr
Normal SuccessfulAttachVolume 17m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-b3137a25-241c-4337-89c1-d6e3953440fe"
Normal Pulling 17m kubelet Pulling image "docker.elastic.co/elasticsearch/elasticsearch:8.5.1"
Normal Pulled 17m kubelet Successfully pulled image "docker.elastic.co/elasticsearch/elasticsearch:8.5.1" in 19.341s (19.341s including waiting)
Normal Created 17m kubelet Created container configure-sysctl
Normal Started 17m kubelet Started container configure-sysctl
Normal Pulled 16m kubelet Container image "docker.elastic.co/elasticsearch/elasticsearch:8.5.1" already present on machine
Normal Created 16m kubelet Created container elasticsearch
Normal Started 16m kubelet Started container elasticsearch
Warning Unhealthy 15m (x6 over 16m) kubelet Readiness probe failed: Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )
Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )
Check cluster health using helm test:
$ helm --namespace=monit test elasticsearch
NAME: elasticsearch
LAST DEPLOYED: Tue May 28 11:53:00 2024
NAMESPACE: monit
STATUS: deployed
REVISION: 1
TEST SUITE: elasticsearch-qsmth-test
Last Started: Tue May 28 12:05:09 2024
Last Completed: Tue May 28 12:05:12 2024
Phase: Succeeded
To access our Elasticsearch instance, we need to retrieve the credentials:
Retrieve elastic user’s username.
$ kubectl get secrets --namespace=monit elasticsearch-master-credentials -ojsonpath='{.data.username}' | base64 -d
Retrieve elastic user’s password.
$ kubectl get secrets --namespace=monit elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -d
Kibana
Kibana is the visualization and dashboarding component of the ELK stack. It offers a user-friendly interface for exploring and analyzing log data stored in Elasticsearch. By installing Kibana with Helm, you can quickly deploy and configure it to connect to your Elasticsearch cluster.
$ helm install kibana elastic/kibana --namespace monit
Watch for all containers to come up:
$ kubectl get pods --namespace=monit -l release=kibana -w
Ater few seconds, Kibana will be ready
NAME READY STATUS RESTARTS AGE
kibana-kibana-8446b87c9f-2vh8g 1/1 Running 0 76s
Check the k8s service for Kibana
$ kubectl get svc --namespace=monit -l release=kibana
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kibana-kibana ClusterIP 10.30.38.169 <none> 5601/TCP 14m
Retrieve the kibana service account token.
$ kubectl get secrets --namespace=monit kibana-kibana-es-token -ojsonpath='{.data.token}' | base64 -d
Access Kibana
We may need to configure an Ingress to access Kibana. Otherwise, we can quickly access Kibana by using port-forwarding:
$ kubectl port-forward service/kibana-kibana :5601 --namespace monit
Forwarding from 127.0.0.1:51850 -> 5601
Forwarding from [::1]:51850 -> 5601
Then, open http://localhost:51850 in your web browser.
Logstash
Logstash is an optional component used for log ingestion, processing, and enrichment. If you have specific log processing requirements, such as parsing structured logs or applying filters, you can install Logstash using the Elastic Helm chart.
$ helm install logstash elastic/logstash --namespace monit
Check that the cluster members are up:
$ kubectl get pods --namespace=monit -l app=logstash-logstash -w
After few seconds, the logstash container will become ready:
NAME READY STATUS RESTARTS AGE
logstash-logstash-0 1/1 Running 0 2m17s
Next steps
Once the different components of the ELK stack are properly installed, you can explore the following tasks:
-
Configure Index Patterns in Kibana: Before visualizing log data in Kibana, you need to define index patterns that specify which Elasticsearch indices to query. Index patterns define the structure of your log data and enable Kibana’s powerful search and visualization capabilities.
-
Configure Logstash (Optional): If you installed Logstash, configure Logstash pipelines to ingest and process your logs. You can define Logstash configuration files and mount them as ConfigMaps or use other methods for configuration management - example.
-
Monitor Elasticsearch and Kibana: Monitor Elasticsearch and Kibana using built-in metrics or integrate with external monitoring solutions like Prometheus and Grafana for comprehensive observability.
That’s all folks
I hope you enjoyed this article, feel free to leave a comment or reach out on twitter @bachiirc.