Post Kubernetes cluster restart, issues observed in case of containerized Postgres deployment

After Kubernetes cluster restart in case of containerized Postgres deployment, couple of flexsnap pods went into CrashLoopBackoff state and primary pod was not started successfully.

This issue arises due to certificate expiration when the cluster was down. Perform the following to ensure that the issue occurred was due to certificate expiration:

Run the following command to exec into the nb-postgresql pod:

kubectl -n <namespace> exec pod/nb-postgresql-0 -it bash

Run the following command from within the Postgres pod

psql "host=nb-postgresql port=13787 dbname=postgres user=postgres sslmode=verify-full sslcert='/netbackup/postgresql/keys/server/tls.crt' sslkey='/netbackup/postgresql/keys/server/tls.key' sslrootcert=/netbackup/postgresql/keys/server/ca.crt"

Following error message is displayed:

psql: error: connection to server at "nb-postgresql" (10.240.3.27), port 13787 failed: SSL error: certificate verify failed.

Run the following command to check the ca.crt server certificate:
- Get the ca.crt of postgresql-server-crt:
  kubectl -n <namespace> get secret postgresql-server-crt -o "jsonpath={.data['ca\.crt']}" | base64 -d > server_ca.crt
- Get the ca.crt of postgresql-netbackup-ca:
  kubectl -n <namespace> get secret postgresql-netbackup-ca -o "jsonpath={.data['ca\.crt']}" | base64 -d > ca.crt
- Get current date on the system:
  date
  Following message is displayed:
```
Tue Feb 27 17:07:14 UTC 2024
```
- Check expiry of ca.crt:
  openssl x509 -enddate -noout -in ca.crt
  Following message is displayed:
```
notAfter=Feb 27 17:52:58 2024 GMT
```
- Check expiry of server_ca.crt:
  openssl x509 -enddate -noout -in server_ca.crt
  Following message is displayed:
```
notAfter=Feb 27 16:57:58 2024 GMT
```
The above displayed messages indicate that ca.crt of postgresql-server-crt is referring to an expired certificate.

Resolution:

Perform the following to resolve the certificate issue:

Delete the certificate secrets:
kubectl -n <namespace> delete secret postgresql-netbackup-ca postgresql-server-crt postgresql-client-crt
Execute the following commands to ensure that the correct certificate is referred:
- Get the ca.crt of postgresql-server-crt:
  kubectl -n <namespace> get secret postgresql-server-crt -o "jsonpath={.data['ca\.crt']}" | base64 -d > server_ca.crt
- Get the ca.crt of postgresql-netbackup-ca:
  kubectl -n <namespace> get secret postgresql-netbackup-ca -o "jsonpath={.data['ca\.crt']}" | base64 -d > ca.crt
- Get current date on the system:
  date
  Following message is displayed:
```
Tue Feb 27 17:11:59 UTC 2024
```
- Check expiry of ca.crt:
  openssl x509 -enddate -noout -in ca.crt
  Following message is displayed:
```
notAfter=Feb 27 18:11:38 2024 GMT
```
- Check expiry of server_ca.crt:
  openssl x509 -enddate -noout -in server_ca.crt
  Following message is displayed:
```
notAfter=Feb 27 18:11:38 2024 GMT
```
The above displayed messages indicate that ca.crt of postgresql-server-crt is referring to the correct certificate.