Upgrading Cloud Scale deployment using kubectl plugin

1. Before you proceed with upgrade, ensure that the following prerequisites are met:

   • Infrastructure readiness: The Kubernetes cluster and Cloud Scale environment is up and running.

   • Required container images: All Cloud Scale related images of the version you would like to upgrade to are pushed to your container registry.

   • Helm setup:

     • Helm is installed and configured.

     • The jetstack repository is added:helm repo add jetstack https://charts.jetstack.io

     • The cert-manager and trust-manager charts are installed via helm.

   Also review the prerequisites in the following section:

2. Execute the binary at bin/ with the upgrade option using the following command:

   ./kubectl-cloudscale upgrade

3. Follow the upgrade steps as follows:

   /root/VRTSk8s-netbackup-11.2.0.2-0019/bin/kubectl-cloudscale upgrade Cloud Scale Upgrade
           Before you proceed with installation, please ensure the following prerequisites are in place:
           1. Infrastructure readiness:
              - The Kubernetes cluster is up and running
              - Cloud Scale environment is up and running
           2. Required container images:
              - All Cloud Scale related images of the version you would like to upgrade to are pushed to your container registry
           3. Helm setup:
              - Helm is installed and configured
              - The "jetstack" repository is added:
                     helm repo add jetstack https://charts.jetstack.io
   	   - The cert-manager and trust-manager charts are installed via helm
   	4. Container registry is logged in to this machine
   	Also review the 'Prerequisites for Cloud Scale Technology upgrade' section in the 'Cohesity Cloud Scale Technology Manual Deployment Guide for Kubernetes Clusters' document for additional required steps.
           Once everything is ready, you can safely continue with the upgrade.
   Would you like to continue? (y/n): y
   Checking if the input file already exists.
   Input file is not present.
   Provide the file path of the extracted Cloud Scale folder: /root/VRTSk8s-netbackup-11.2.0.2-0019/
   
   Provide a namespace for the Environment: netbackup Environment is ready for upgrade 
   
   *******************************Checking for cert-manager******************************** 
   Checking if Cert Manager is installed or not Cert-manager is installed with version 'v1.13.3'. It is recommended to upgrade cert-manager to version v1.18.2 post Cloud Scale deployment
   
   *******************************Checking for trust-manager********************************
   
   Checking if Trust Manager is installed or not Trust-manager is installed with version ''v0.7.0''. It is recommended to upgrade trust-manager to version v0.19.0 post Cloud Scale deployment
   
   Provide the image tag for the NetBackup images that need to be upgraded: 11.2.0.2-0019 
   Validating images...
   
   Provide the image tag for the MSDP images that need to be upgraded: 21.1.0.2-0013 
   Validating images... 
   
   Provide the image tag for the NetBackup Snapshot Manager images that need to be upgraded: 11.2.0.2-3006 
   Validating images...
   
   Operator Namespace: netbackup-operator-system
   Upgrade of operators started...
   The operators chart has been installed successfully.
   
   Provide the image tag for the PostgreSQL images that need to be upgraded: 16.11.2.0-0001 
   Validating images...
   
   Current version of cloudscale: 11.1
   
   Starting upgrade of Helm-based CloudScale environment in namespace netbackup
   Successfully upgraded CloudScale Helm release to version 4
   Successfully initiated the upgrade for Helm-based CloudScale environment in netbackup namespace.
   Ensure that all pods are up and running before using the NetBackup.
   For kubectl plugin specific logs, check the log at /root/.cloudscale/setup-cloudscale.log
   

To patch StorageClass and PV with nconnect=4 for better NFS performance

1. Patch the StorageClass (affects newly created PVs):

   • Identify the StorageClass used by the NFS-mounted volume.

     Use the following command to list the PVCs in the netbackup namespace:

     $ kubectl get pvc -n netbackup

     Example output:

     NAME                                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      VOLUMEATTRIBUTESCLASS   AGEcatalog-nbu-primary-0                           Bound    pvc-37f2d9be-84e8-4e6c-8463-dceb16642370   110Gi      RWX            nb-file-premium   <unset>                 19dcertauth-pvc                                    Bound    pvc-defd5f28-0be5-4e70-8453-0e2472bfca06   1Gi        RWO            nb-disk-premium   <unset>                 19dcloudpoint-pvc                                  Bound    pvc-e910737e-4d69-4e0b-8fc5-d57e5787eace   5Gi        RWX            nb-file-premium   <unset>                 19ddata-flexsnap-rabbitmq-0                        Bound    pvc-794271f2-5440-47b8-b285-2929b0ab6669   5Gi        RWO            nb-disk-premium   <unset>                 19d

     From this output, the PVC catalog-nbu-primary-0 uses the StorageClass nb-file-premium, so this is the StorageClass we will patch.

   • Patch the StorageClass.

     Apply the nconnect=4 mount option:

     kubectl patch storageclass nb-file-premium --type merge -p '{"mountOptions":["nconnect=4"]}'

     The following content must be displayed:

     storageclass.storage.k8s.io/nb-file-premium patched

   • Verify the StorageClass patch.

     kubectl get sc nb-file-premium -o yaml | sed -n '/mountOptions/,$p'

     The following content must be displayed:

     mountOptions:
     - nconnect=4

2. Patch the existing PV (applies to currently mounted volumes):

   • Use the following command to export the PV name:

     export PV=pvc-37f2d9be-84e8-4e6c-8463-dceb16642370

   • Use the following command to check whether the PV already has the mount option:

     kubectl get pv "$PV" -o json | jq -e '.spec.mountOptions // [] | index("nconnect=4")'

     If output is null, the option is not present.

   • Use the following command to patch the PV:

     kubectl patch pv "$PV" --type merge -p '{"spec": {"mountOptions": ["nconnect=4"]}}'

     Example output:

     persistentvolume/<PV_NAME> patched

3. Restart pods to apply the new NFS mount option:

   Use the following command to delete each MediaServer replica pod:

   export POD=media1-media-0 kubectl delete pod $POD -n netbackup

   The pod will restart and re-mount with the updated NFS settings.

4. Verify nconnect=4 inside the pod:

   Once the pod has restarted, run:

   mount | grep nfs

   Example output:

   ... nconnect=4 ...

5. Apply changes to primary nodes:

   • Scale down the Primary nodepool to 0: This forces NFS unmounting. Then scale it back up so that all primary pods remount the storage with the updated nconnect=4 option.

   • Verify from primary pods:

     mount | grep nfs

     Example output:

     ... nconnect=4 ...

6. Verify from all other related pods:

   For example, kubectl exec -it nbu-nbatd-0 -n netbackup -- mount | grep nfs

   The nconnect=4 must be visible on on all relevant NFS mount paths.