Fluent Bit is a super fast, lightweight, and highly scalable logging and metrics processor and forwarder that allows you to collect data and logs from different sources, unify them, and send them to multiple destinations. Tanzu Kubernetes Grid includes signed binaries for Fluent Bit, that you can deploy on management clusters and on Tanzu Kubernetes clusters to provide a log-forwarding service. This post explains how to install the fluent-bit package onto a Tanzu Kubernetes (workload) cluster using TMC catalog and forward the logs to Azure blob storage.
Pre reqs
Bootstrap machine with the following installed: Tanzu CLI, kubectl installed as mentioned here
Tanzu Kubernetes Grid management cluster and workload cluster running on vSphere, Amazon EC2, or Azure, with the package repository installed. For this demo, I have deployed TKG on Azure.
Storage Account and a blob container: You can use an existing one if already exists. if not, below are the steps to create a storage account:
Navigate to Azure portal > Storage Accounts > Create
Project details:
Subscription:
Resource group: Create new or use existing one. In this case, I have used an existing resource group (images-rg)
Storage account name: provide a name, here I have given capvfluentd
Review + Create > Create
In Azure portal > Storage Accounts > click on newly created storage account (capvfluentd) > Create Container by clicking on + Container
Name it and Create. In this case, I named it as capv-logs
Once successfully created, you can find it under the containers list
Install the package
In TMC Console: Catalog > select the workload cluster (capv-workload in my case) from drop down and click on fluent-bit
Click on Install Package which can be found on top right side of the page
Name the Installed package as : capv-workload-fluentd and click NEXT
Package install resources: Leave to Default and click NEXT
Configure values: Provide below values
config file
namespace: "tanzu-system-logging" fluent_bit: config: service: | [Service] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 outputs: | [OUTPUT] Name azure_blob Match * account_name <redacted> # To be collected from Azure portal shared_key <redacted> # To be collected from Azure portal container_name <redacted> # To be collected from Azure portal path kubernetes blob_type blockblob auto_create_container on tls on inputs: | [INPUT] Name tail Tag kube.<namespace_name>.<pod_name>.<container_name> Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)- Path /var/log/containers/*.log Exclude_Path /var/log/containers/*_fluentbit2S3_*.log filters: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token Kube_Tag_Prefix kube. Merge_Log On Merge_Log_Key log_processed K8S-Logging.Parser On K8S-Logging.Exclude On Regex_Parser kubernetes-tag Labels Off Annotations Off parsers: | [PARSER] Name azure_blob Format json Time_Key requestReceivedTimestamp Time_Format %Y-%m-%dT%H:%M:%S.%LZ Time_Keep On [PARSER] Name kubernetes-tag Format regex Regex (?<namespace_name>.+)\.(?<pod_name>.+)\.(?<container_name>.+) #! optional configuration for the daemonset daemonset: resources: #! limits: #! cpu: 100m #! memory: 128Mi #! requests: #! cpu: 100m #! memory: 128Mi podAnnotations: {} podLabels: {}
Install Package
Validate:
In TMC Console: Clusters > capv-workload > Add-ons > Installed > check if the package ( fluent-bit.tanzu.vmware.com ) is succeeded and healthy
# Get the admin credentials of the workload cluster into which you want to deploy Multus cni. In this case, capv-workload is workload cluster:
$ tanzu cluster kubeconfig get capv-workload --admin
Note: If you see below errors in fluent bit logs, then follow the below steps. If not, then Navigate to Azure portal > Storage Accounts > click on container: capv-logs > kubernetes
