Elastic media server related issues
This section provides the issues and their respective workaround for the issues observed in elastic media server feature.
Jobs are going in queued state
This behavior can be observed in 10.5 or later versions. If the jobs are unexpectedly going in queued state -
Check the Activity monitor logs for the specific job. Verify if it is due to maximum count being reached for the media server.
Alternatively, operator logs can be checked and it may show following messages -
2024-06-18T18:16:33Z INFO Jobs are queued on all media servers due to throttling. {"controller": "mediaserver", "controllerGroup": "netbackup.veritas.com", "controllerKind": "MediaServer", "MediaServer": {"name":"nbu","namespace":"netbackup"}, "namespace": "netbackup", "name": "nbu", "reconcileID": "78a2599a-79fc-4ffa-b9e4-24a7ca29f80b", "MediaServer": "nbu"} 2024-06-18T18:16:33Z INFO Found 19 jobs in queue due to job based throttling. Maximum Job limit is 4. Requesting to scale out by 2 replicas. {"controller": "mediaserver", "controllerGroup": "netbackup.veritas.com", "controllerKind": "MediaServer", "MediaServer": {"name":"nbu","namespace":"netbackup"}, "namespace": "netbackup", "name": "nbu", "reconcileID": "78a2599a-79fc-4ffa-b9e4-24a7ca29f80b", "Media Server": "nbu"}
If the jobs are in queue due to this reason, the media server pods will scale out if the media server autoscaler is ON and the jobs will get assigned to scaled out pods. For more details on this configuration parameters, refer See Elastic media server..
Issue with autoscaler for scaling in the media server pods
This issue is observed when there is no load or only few jobs are running even when there are maximum number of media server pods/nodes that are in running state.
Verify if media server autoscaler is trying to scale-in but unable to shutdown the media server pods which are marked to be scaled-in.
Verify if there are any jobs or bpps processes running on the media pods with the higher indexed running pod by referring to the NetBackup operator logs as mentioned below:
2023-03-01T08:14:56.470Z INFO controller-runtime.manager.controller.mediaserver Running jobs 0: on Media Server nbux-xx-xxx-xx-xx.vxindia.cohesity.com. {"reconciler group": "netbackup.veritas.com", "reconciler kind": "MediaServer", "name": "media1", "namespace": "netbackup-environment", "Media Server": "nbux-xx-xxx-xx-xx.vxindia.cohesity.com"} 2023-03-01T08:14:56.646Z INFO controller-runtime.manager.controller.mediaserver bpps processes running status. false: on Media Server nbux-xx-xxx-xx-xx.vxindia.cohesity.com. {"reconciler group": "netbackup.veritas.com", "reconciler kind": "MediaServer", "name": "media1", "namespace": "netbackup-environment", "Media Server": "nbux-xx-xxx-xx-xx.vxindia.cohesity.com"}Perform the following to know which bpps processes are running and are not allowing to scale-in the media server pod:
Login to NetBackup Web UI portal.
Check the notifications tab for any notifications of Media server elasticity event category. The notification has the list of additional process running on specific media server. User must wait until the process listed in the additional process running exits.
Alternatively, user can also see the list of processes in the NetBackup operator logs as follows:
2023-07-11T13:33:44.142Z INFO controller-runtime.manager.controller.mediaserver Following processes are still running : bpbkar test1, bpbkar test2 {"reconciler group": "netbackup.veritas.com", "reconciler kind": "MediaServer", "name": "test-media-server", "namespace": "netbackup-environment"}
Verify if there are any jobs in queued state due to the maximum jobs per media server setting. If yes, the scale-in should happen when there are no more queued jobs.
Duplication job is not getting completed
Duplication job is not getting completed (in turn scale-down is not happening) because is set on the media server as part of scale down while duplication is still running.
While configuring destination storage unit, manually select media servers which are always up, running and would not be able to scale-in anytime. Number of media server which are always running would be same as that mentioned against field in CR. Above recommendation also applies when upgrading from older version to NetBackup version 10.3. Post-upgrade manually select media servers that are mentioned against field in CR during upgrade. Default is 1.
Error while connecting to media server
For scaled in media servers, certain resources and configurations are retained to avoid reconfiguration during subsequent scale out. Post entries for scaled in media servers are not removed from NetBackup primary server and hence if those media servers are used for any operation, connectivity issue is observed.
Workaround:
It is recommenced to use media servers that are always up, running and would never scale in (by the media server autoscaler). Number of media servers that are always up and running would be same as that of the value mentioned in field in CR.