Resolving issues when media server PVs are deleted
Media server describe events display the following error:
Warning FailedAttachVolume 55m (x13 over 66m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-e16fd50a-1b6e-4b5b-ab53-76553ea87507" : rpc error: code = NotFound desc = Volume not found, failed with error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-e16fd50a-1b6e-4b5b-ab53-76553ea87507' under resource group 'mc_nbu-k8s-network_nbux-deepak_eastus' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}"NetBackup media server and NetBackup primary server were in running state. Media server persistent volume claim or media server pod is deleted. In this case, reinstallation of respective media server can cause the issue.
To resolve the issues
- Open the NetBackup Web UI using primary server hostname given in the primary server CR status.
- Pause the media server reconciler by setting the value as paused: true in mediaServer section in environment CR using the following command:
helm upgrade cloudscale VRTSk8s-netbackup-11.2/helm/cloudscale-<version>.tgz -n <namespace> --reuse-values \
--set environment.mediaServers[0].name=media1
--set environment.mediaServers[0].nodeSelector.labelKey=<original value> \
--set environment.mediaServers[0].nodeSelector.labelValue=<original value>
--set environment.mediaServers[0].replicas=<original value> \
--set environment.mediaServers[0].minimumReplicas=<original value> --set environment.mediaServers[0].storage.data.capacity=<original value> \
--set environment.mediaServers[0].storage.data.storageClassName=<sc-name> \
--set environment.mediaServers[0].tag=11.2
--set environment.mediaServers[0].storage.log.capacity=<original value> \
--set environment.mediaServers[0].storage.log.storageClassName=<sc-name> \
--set environment.mediaServers[0].paused=true
The values for <original value> in the above command can be viewed using the helm get values cloudscale -n netbackup command prior to invoking the commands in the step 2 above.
Save the changes.
- Delete respective media server pod using kubectl delete <pod-name> -n <namespace> command.
- Delete data and logs PVC for respective media server only using the kubectl delete pvc <pvc-name> -n <namespace> command.
- Un pause the media server reconciler by changing the value as paused: false in mediaServer section in environment CR using the following command:
helm upgrade cloudscale VRTSk8s-netbackup-11.2/helm/cloudscale-<version>.tgz -n <namespace> --reuse-values \
--set environment.mediaServers[0].name=media1
--set environment.mediaServers[0].nodeSelector.labelKey=<original value> \
--set environment.mediaServers[0].nodeSelector.labelValue=<original value>
--set environment.mediaServers[0].replicas=<original value> \
--set environment.mediaServers[0].minimumReplicas=<original value>
--set environment.mediaServers[0].storage.data.capacity=<original value> \
--set environment.mediaServers[0].storage.data.storageClassName=<sc-name> \
--set environment.mediaServers[0].tag=11.2
--set environment.mediaServers[0].storage.log.capacity=<original value> \
--set environment.mediaServers[0].storage.log.storageClassName=<sc-name> \
--set environment.mediaServers[0].paused=false
Save the changes.