Troubleshooting issues for kubectl plugin

Primary key not recreated after manual deletion from secrets/sp-keys

If the primary key is manually deleted by editing the secrets/sp-keys in the environment, the key is not automatically recreated by the system. This results in missing or invalid service principal keys, which can affect environment operations.

Workaround:

Perform the following steps to manually recreate the deleted key:

Pause the environment reconciler using helm command to set the following parameter:
paused: true
Delete the existing secret by running the following command:
- kubectl delete secrets/sp-keys -n netbackup
Delete all the API keys using the following API's call with a valid JWT token:
- Trigger GET netbackup/security/service-principal-configs API.
- Find all the service principal configurations where "servicePrincipalType"="netbackup-operator" and note the value of servicePrincipalId for all these service principal configurations.
- Trigger DELETE security/service-principal-configs/{id} API for each servicePrincipalId captured above.
Restart the operator pod by deleting the operator pod:
- kubectl delete pod <operator-pod-name> -n netbackup-operator-system
Un-pause the environment reconciler using helm command to set the following parameter:
paused: false

Cloud Scale installation interrupted midway

During the Cloud Scale installation, if the process breaks or stops midway, the deployment does not complete successfully and must be resumed from the point of failure.

Workaround:

To resume the installation process:

Run the following command:
kubectl-cloudscale install
When prompted to resume the installation, type yes to continue from where it stopped.

The plugin will automatically read the previously saved inputs and resume the installation from the point of interruption.

Unable to re-enter inputs during Cloud Scale upgrade

If a user enters an incorrect Cloud Scale folder path or provides any wrong input during the upgrade process, there is no option to correct it. Re-triggering the plugin simply skips to the next question without allowing the user to re-enter inputs.

Workaround:

To reset and re-enter all upgrade inputs:

Delete the following file:
rm /home/<user-name>/.cloudscale/upgrade.csv

Re-run the upgrade command:

kubectl-cloudscale upgrade

The plugin will prompt for all required user inputs again.

Following is an example of the logs:

*******************************Checking for cert-manager********************************
{Component: Installation of helm chart, ComponentName: jetstack/cert-manager}
INFO: 2025/09/26 12:38:32 logger.go:132: Checking if Cert Manager is installed or not {Component: Get pod status, ComponentName: Dependency of Cloudscale component}
INFO: 2025/09/26 12:38:32 logger.go:132: Waiting 10 seconds before retrying... {Component: Get pod status, ComponentName: Dependency of Cloudscale component}
INFO: 2025/09/26 12:38:50 logger.go:132: Cloudscale Upgrade

Before you proceed with upgrade, please ensure the following prerequisites are in place:
    
    1. Infrastructure readiness:
       - The Kubernetes cluster is up and running
       - Cloudscale environment is up and running
    
    2. Required container images:
       - All Cloudscale-related images of the version you would like to upgrade to
         are pushed to your container registry
    
    3. Helm setup:
       - Helm is installed and configured
       - The "jetstack" repository is added:
         helm repo add jetstack https://charts.jetstack.io
       - The cert-manager and trust-manager charts are installed via helm

Also review the 'Prerequisites for Cloud Scale Technology upgrade' section in the
'NetBackup™ Deployment Guide for Kubernetes Clusters' document for additional required steps.

Once everything is ready, you can safely continue with the upgrade.
{Component: CloudScale, ComponentName: Upgrade of CloudScale}
INFO: 2025/09/26 12:38:50 logger.go:132: Would you like to continue? (y/n):  {Component: CloudScale, ComponentName: Upgrade of CloudScale}
INFO: 2025/09/26 12:39:00 logger.go:132: Helm Version: version.BuildInfo{Version:"v3.18.6", GitCommit:"b76a950f6835474e0906b96c9ec68a2eff3a6430", GitTreeState:"clean", GoVersion:"go1.24.6"}
{Component: Precheck Config, ComponentName: Installation of CloudScale}
INFO: 2025/09/26 12:39:00 logger.go:132: kubectl Version: Client Version: v1.34.1
Kustomize Version: v5.7.1
Server Version: v1.32.6
{Component: Precheck Config, ComponentName: Installation of CloudScale}
INFO: 2025/09/26 12:39:00 logger.go:132: **************************************************
{Component: CloudScale Upgrade, ComponentName: Input Configuration}
INFO: 2025/09/26 12:39:00 logger.go:129:
Checking if the input file already exists.

INFO: 2025/09/26 12:39:00 logger.go:132: Data is being read from an input file that is present.
{Component: CloudScale, ComponentName: Input Configuration}
INFO: 2025/09/26 12:39:00 logger.go:132: The following values were loaded from the file:
{Component: CloudScale Upgrade, ComponentName: Input Configuration}

Plugin upgrade interrupted or skipped validation checks

The plugin was cancelled while checking for Cert Manager installation. When the upgrade was triggered again, it skipped the Cert Manager validation step and proceeded to the next question instead of restarting the validation process.
Workaround:
Delete the following file and re-run the upgrade:
rm /home/<user-name>/.cloudscale/upgrade.csv
The plugin was cancelled before completing the operator upgrade. When the upgrade was triggered again, it failed due to the Helm release being stuck in a pending-upgrade state with the following error:
```
Operator Namespace : netbackup-operator-system
Upgrade of operators started...
Helm upgrade of operators failed with error: another operation (install/upgrade/rollback) is in progress
Operators chart failed to upgrade.
Error while upgrading operators chart : another operation (install/upgrade/rollback) is in progress
Upgrade failed:
```
Workaround:
To resolve this issue:
- Check for pending Helm releases:
  helm ls -A --pending
- If the operator's release is in a pending-upgrade state, list previous revisions:
  helm history operators --namespace <operator_namespace>
- Identify a revision with the status deployed or superseded and roll back to that version:
  helm rollback operators <REVISION> --namespace <operator_namespace>
  This clears the pending-upgrade lock caused by the interrupted upgrade.
- Once the rollback completes, rerun the plugin upgrade command:
  kubectl-cloudscale upgrade
- When prompted to resume the installation, type yes to continue from where it stopped.

Plugin crash after operator upgrade

The plugin crashed while annotating the environment resources immediately after the operator upgrade.

Workaround:

Trigger the upgrade process again using the kubectl-cloudscale plugin:
kubectl-cloudscale upgrade
When prompted to resume the installation, type yes to continue the upgrade from where it stopped.

Plugin crash before Helm triggers Cloud Scale upgrade

If the plugin crashes at any point before Helm triggers the Cloud Scale upgrade command, the upgrade process is interrupted and cannot complete successfully.

Workaround:

Trigger the upgrade process using the kubectl-cloudscale plugin:
kubectl-cloudscale upgrade
When prompted to resume the installation, type yes to continue the upgrade from where it stopped.

Media server pod restarting due to read-only mount

The media server pod is continuously restarting. Upon inspection, it is observed that the mount point /mnt/nblogs inside the media pod is in a read-only state, which is likely causing the restarts.

Perform the following steps to verify if the mount point is read-only

Exec the following command into the media pod:
kubectl exec -it <media-pod-name> -- bash
Run the following command to check the mount point permissions:
cd /mnt/nblogs/fluentbit
touch test
Expected output (in case of an issue):
```
touch: cannot touch 'test': Read-only file system
```
The above error message confirms that the /mnt/nblogs mount is mounted as read-only.

Workaround:

Run the following command to manually restart the media pod by deleting the affected media pod:

kubectl delete pod <media-pod-name>

Image validation fails after providing image tag input

Image validation can fail due to one or more of the following reasons:

Invalid container registry name.

An incorrect or non-existent registry name was provided.

Example:

Image validation failed, failed to configure transport: error pinging v2 registry: 
Get "https://wrong.azurecr.io/v2/": dial tcp: lookup wrong.azurecr.io on 168.63.129.16:53: no such host

Container registry not logged in on the host VM.
The container registry is not logged in using the same user account that runs the plugin on the host VM.
Example:
```
Image validation failed, 
Get "https://CloudscaleACRshantaram.azurecr.io/v2/netbackup/dbm/manifests/11.1.0.2-0013": 
unauthorized: authentication required
```
Incorrect or non-existent image tags.
The specified image tags do not exist or have not been pushed to the container registry.
Example:
```
 Image validation failed, no such manifest: 
nbuk8sreg.azurecr.io/netbackup/dbm:wrong
```

Workaround: Ensure that all inputs are correct and that the container registry is logged in properly.

To skip image validation for tags, run the following command:

kubectl create configmap cs-image-validation-config -n <netbackup-namespace> --from-literal=SKIP_IMAGE_VALIDATION=true

Troubleshooting issues for kubectl plugin

Feedback

Feedback