Logging feature (fluentbit) in Cloud Scale
Logging feature is introduced in 10.5 and later version of NetBackup for Cloud Scale Technology which allows you to consolidate log files that have been distributed during the process of running NetBackup in a scale-out environment. It helps you to gather all the log files in one place, making them easier to access and use.
To deploy fluentbit for logging feature, following are the required components:
Collector pod: The collector pod receives logs from the DaemonSet and Application sidecar containers. The collector pod itself consists of two containers:
Fluentbit-collector: This container is responsible for receiving the logs and then writing them to a central location based on the details such as date, namespace, pod, container and file path. The primary purpose of the collector is to consolidate and write the files to a centralized the destination.
Log-Cleanup Sidecar This container on the collector pod is responsible for cleaning up logs from the PVC (PersistentVolumeClaim) attached. There are variables that can be configured to determine retention and other parameters.
Sidecar sender: This container is the Kubernetes sidecar container that runs with NetBackup application pods. The pods produce NetBackup application specific logs that are to be collected, and access to the location of the logs is shared with the fluentbit sidecar. It scrapes those logs and sends them to the Collector Pod. The logs are stored in a shared volume mounted at
/mnt/nblogs.DaemonSet sender: DaemonSet senders in Kubernetes are the pods allocated to the nodes based on specific taints and tolerations. Nodes with certain taints reject DaemonSets without matching tolerations, while nodes with matching toleration are assigned DaemonSet sender pods. These sender pods have access to the container logs of all pods within the node. This allows the DaemonSet sender to gather
stdout/stderrlogs for NetBackup applications as well as the Kubernetes infrastructure.Log-Viewer: The log-viewer pod (introduced in NetBackup version 11.0) serves as the primary pod for users to exec into the pod and view the logs that have been collected by the fluentbit logging system. It also hosts APIs that provides access to extract the collected logs.
Note:
By default, only stdout logs are downloaded. To include sidecar logs, the flag must be manually enabled.
For more information on log extraction, see the following section:
Known issue
An abrupt restart of the listener service (including the fluentbit sidecar) while jobs are running may occasionally result in duplicate log entries being sent to the fluentbit-collector.
After deploying the NetBackup Kubernetes cluster, you might encounter pods stuck in an 'Init' state due to bootstrapper pod failure. This pod is short-lived and doesn't remain active after failing. To identify the cause of failure, check the bootstrapper logs within the NetBackup Fluent-bit collector.
Refer to the section See Troubleshooting issue for bootstrapper pod. for more details.