Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service
Do the following:
Ensure that the service is configured and started on the primary server. To configure the service, run the configureMQ command.
Refer to the NetBackup Commands Reference Guide.
Reasons:
Ports that are configured for the service is in use by some other process.
The configuration file is corrupted.
Do the following:
Check the configureMQ command logs for failure.
Check the nbmqbroker service logs for failure.
Run the configureMQ command.
Refer to the NetBackup Commands Reference Guide.
Reasons:
The port configured for the service is not available for use.
Connection fails with some SSL exception.
The nbmqbroker service is not restarted after the configureWebServerCerts command is run on the primary server.
Do the following:
Ensure that the port configured for the nbmqbroker service is available for use and accessible by NetBackup hosts.
Check the connectivity between the primary server and the NAT client using the nbcertcmd -ping command.
If the command is not successfully executed, refer to the troubleshooting section for the NetBackup web service.
If the command is successfully executed, run configureMQ command to configure the nbmqbroker service.
Restart the nbmqbroker service.
Reasons:
The JSON web token (JWT) for the NAT client cannot be refreshed.
The security certificate of the NAT client is revoked.
The NetBackup Web Management Console (or nbwmc) service is not running.
Do the following:
Refer to the subscriber troubleshooting steps.
If the client's security certificate is revoked, reissue the certificate.
Start the nbwmc service.
Reasons:
The disaster recovery package is lost.
The configureMQ command is not run after the disaster recovery (DR) installation.
Do the following:
Run the configureMQ or configureMQ -defaultPorts command.
Refer to the NetBackup Commands Reference Guide.
To check if the installation root folder has the 8dot3 file name setting enabled, run the following command from your folder:
>dir /x
Example: The 'Program Files' directory has the 8dot3 file name setting enabled, therefore the short name 'PROGRA~1' is generated.
But it differs for the 'not8 Dot3' directory.
C:\>dir /x
The volume in drive C has no label.
The Volume Serial Number is FE21-2F8E
Directory of C:\
-5.6.3
12/06/2019 02:24 PM <DIR> not8 Dot3
12/02/2019 06:35 AM <DIR> PROGRA~1 Program Files
12/02/2019 10:44 AM <DIR> PROGRA~2 Program Files (x86)
Do the following to resolve the issue:
- Enable 8dot3 name file setting for the NetBackup installation root folder using the fsutil command.
Refer to the following article: Fsutil 8dot3name
- If the problem persists, contact Technical Support.
Consider the following scenario:
NetBackup is configured to use only external CA-singed certificates at the time of catalog backup. Therefore, the disaster recovery package that was created during catalog backup contains the required external certificates. If the host identity is recovered using such disaster recovery package after NetBackup installation, the nbmqbroker service may behave incorrectly because of the NetBackup CA-signed certificates that were issued during installation.
To resolve the issue
- Verify if your the NetBackup environment uses only external CA-signed certificates. Run the following command:
nbcertcmd -getSecConfig -caUsage
- Check the certificates that the nbmqbroker service uses. Run the following command:
On Unix: cat /usr/openv/var/global/mqbroker/mqbroker.config | grep ssl_options
On windows: type "NetBackup_Install_path\var\global\mqbroker\mqbroker.config" | findstr "ssl_options"
If only external CA-signed certificates are used in your environment, the command shows the path with externalcacreds entry.
If the command shows the path with nbcacreds entry, NetBackup CA-signed certificates are used.
For example:
{ssl_options, [{cacertfile, "/usr/openv/var/global/mqbroker/certstore/nbcacreds/ca.pem"}, {ssl_options, [{cacertfile, "/usr/openv/var/global/mqbroker/certstore/nbcacreds/ca.pem"},
You need to remove the NetBackup certificates so that the nbmqbroker service works appropriately.
- Run the following command to remove the NetBackup certificates:
configureWebServerCerts -removeNBCert
- Restart the NetBackup Web Management Console (nbwmc) service and the nbqmbroker service to reflect the changes.
- Check the certificates that the nbmqbroker service uses. Run the following command:
On Unix: cat /usr/openv/var/global/mqbroker/mqbroker.config | grep ssl_options
On windows: type "NetBackup_Install_path\var\global\mqbroker\mqbroker.config" | findstr "ssl_options"
Expected output for external certificate only mode:
{ssl_options, [{cacertfile, "/usr/openv/var/global/mqbroker/certstore/externalcacreds/ca.pem"}, {ssl_options, [{cacertfile, "/usr/openv/var/global/mqbroker/certstore/externalcacreds/ca.pem"},
The nbmqbroker service logs show the following errors:
escript: exception error: undefined function rabbitmqctl_escript:main/1
in function escript:run/2 (escript.erl, line 758)
in call from escript:start/1 (escript.erl, line 277)
in call from init:start_em/1
in call from init:do_boot/3
Root cause:
Certain configuration changes on the primary server may result into inconsistency in nbmqbroker service configuration. To resolve the issue, you need to reconfigure the nbmqbroker service.
To reconfigure the nbmqbroker service
- Stop the nbmqbroker service by running the following command:
/usr/openv/mqbroker/bin/nbmqbroker stop
- Run the following command to configure the nbmqbroker environment:
/usr/openv/mqbroker/bin/install/configureMQEnv
- Run the following command to configure the nbmqbroker service:
/usr/openv/mqbroker/bin/install/configureMQ
- Start the nbmqbroker service by running any of the following commands:
/usr/openv/mqbroker/bin/nbmqbroker start
bp.start_all command
For more information on the commands, refer to the NetBackup Commands Reference Guide.
Reasons:
The primary sever name is possibly being resolved to both IPv4 and IPv6 addresses although only IPv6 address is used.
Run the following command to check if the output contains an IPv4 address: nslookup primary_server_name
Sample output:
# nslookup primary-server.com
Server: 2600:100:f0a1:9000::a
Address: 2600:100:f0a1:9000::a#53
Non-authoritative answer:
Name: primary-server.com
Address: 10.200.100.60
Name: primary-server.com
Address: 2600:100:f0a1:9014::335
Expected output:
# nslookup primary-server.com
Server: 2600:100:f0a1:9000::a
Address: 2600:100:f0a1:9000::a#53
Non-authoritative answer:
Name: primary-server.com
Address: 2600:100:f0a1:9014::335
Do the following:
Fix all the configurations to create an appropriate IPv6-only setup.
If the issue still persists, do the following configuration changes to start the nbmqbroker service.
With this configuration, the nbmqbroker service always attempts to first use the IPv6 address for name resolution.
To change the configurations
- Do the following to create the required file.
Use an appropriate text editor (vi on Linux or Notepad on Windows) and create a file called
erl_inetrcin the given directory:On Linux, create the
erl_inetrcfile in the following directory:/usr/openv/var/global/mqbroker/erl_inetrc
Run the following command:
cat > /usr/openv/var/global/mqbroker/erl_inetrc
On Windows, create the
erl_inetrcfile in the following directory: NetBackup_Install_path\var\global\mqbroker\ - Add the following line in the
erl_inetrcfile:{inet6,true}.
Note that the trailing dot (.) is mandatory.
- On UNIX, run the following command to check the permissions of the /usr/openv/mqbroker/bin/setmqenv file:
ls -l /usr/openv/mqbroker/bin/setmqenv
The output is as follows:
-rwxr-x---. 1 nbwebsvc nbwebgrp 3869 date /usr/openv/mqbroker/bin/setmqenv
- Do the following:
On Linux:
Add the following lines in the /usr/openv/var/global/mqbroker/advanced_setmqenv file:
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-kernel inetrc '/usr/openv/var/global/mqbroker/erl_inetrc' -proto_dist inet6_tcp"
RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp"
On Windows:
Add the following lines in the NetBackup_Install_path\var\global\mqbroker\advanced_setmqenv file:
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS=-kernel inetrc 'E:/NetBackup/var/global/mqbroker/erl_inetrc' -proto_dist inet6_tcp
RABBITMQ_CTL_ERL_ARGS=-proto_dist inet6_tcp
- Ensure that the file permissions are not changed after the update.
- Start the nbmqbroker service.