About log collection by job ID
NetBackup includes a command line interface and API option of gathering relevant logs by specifying a job ID, and then uploading the gathered logs. With the specified job ID, logs within the job run time frame are gathered from the primary server, media server, and clients if reachable.
Legacy logs and try file logs may include logs outside of job run time frame as those logs do not honor the time duration filter. Logs from all the hosts that are involved in a job hierarchy are gathered by specifying a job ID of the hierarchy. Cohesity recommends that you use time synchronization for log collection on all hosts that are included in the job time duration. A valid job ID must be present in the Activity monitor. By default, a job ID is removed one week after the job is completed. The nblogadm utility cannot gather the logs of a job ID if bpdbjobs or the Activity monitor cannot retrieve the job details of the specified job ID. In addition, the logs gathering command line interface and API option do not support Backup Now jobs. The VxUL logs are not gathered from a back-level media server or a client.
The gathered logs include NetBackup product and NetBackup support utility (nbsu) logs. The log gathering supports one record ID at a time, no concurrent log gathering from multiple record IDs.
To avoid filling up the file system on primary server, media server, and client during log gathering, Cohesity recommends that you use the KEEP_LOGS_SIZE_GB option. Cohesity recommends that you specify the size of NetBackup logs that are retained before you gather the logs. See the NetBackup Administrator's Guide, Volume I for more information.
A time-based log cleanup process is introduced in NetBackup 10.2. When logs are not removed 7 days after they are gathered, this process removes those gathered logs and the log record. To have a shorter log retention period of 5 days on a primary server or a media server, set the LOG_RECORD_EXPIRY_DAYS to 5 with bpsetconfig. To have a shorter log retention period of 5 days on a client, set the LOG_RECORD_EXPIRY_DAYS to 5 with nbsetconfig. The smaller number takes precedence. NetBackup may not remove logs from a back-level media server or a client if it encounters errors during the log cleanup process. Cohesity recommends that you remove the left behind logs manually when you encounter this situation.
To avoid the gathered logs filling up the file system on a primary server, a predefined 10GB free space watermark is used. NetBackup uses this watermark to check and prevent the start of log gathering when the available disk space is less than the sum of watermark and the estimated size of the gathered logs. Additionally, the log gathering process stops when the available space on a primary server falls under the sum of watermark and the estimated size of the gathered logs. In this release, the check of available space is extended to media servers and clients. To reduce the free space watermark to 5GB, set the HIGH_WATERMARK_TRB_LOG_RECORDS = 5 with bpsetconfig command.
You have two options to gather higher verbosity logs. You can manually enable logging and configure the desired logging level as documented in the NetBackup Logging Reference Guide. Or you can use the command line interface and API option to gather and to configure the logging level values on a primary server, a media server, or a client. Then restart the job and start a log gathering task. The feature includes an API option to retrieve the job ID of a new job after the originally specified job is restarted.
Two log record IDs are required to gather higher verbosity logs. The first log record ID (Record ID 1) is used for enabling logging and configuring the desired logging levels to the hosts of a job ID (Job ID 1). After logging levels are configured and the original job (Job ID 1) is restarted, a new job ID (Job ID 2) is generated. The second log record ID (Record ID 2) is used for gathering logs within the new restarted job (Job ID 2) run time frame from the primary server, media server, and clients if reachable. On a backup domain that consists of multiple media servers and clients, the media servers or clients of Record ID 1 and Record ID 2 may not be the same due to the job scheduling algorithms.
In NetBackup 10.2 and later, a SHA256 checksum of each collected log is included in the Progress.txt file of the directory shown. The checksum fails to compute on a media server or a client with back-level of NetBackup installed.
Location of the Progress.txt file:
Linux and UNIX
/usr/openv/netbackup/logs/nblastaging/record ID-timestamp: YYYYMMDD-HHMMSS
Windows
install_path\Veritas\NetBackup\logs\nblastaging\record ID-timestamp: YYYYMMDD-HHMMSS
NetBackup 10.2 and later includes a space usage enhancement to the required log storage space on a primary server. The log files that are gathered from a primary server, a media server, and a client are no longer stored on the primary server. The files reside with each host in the directory shown.
Linux and UNIX
/usr/openv/netbackup/logs/nblaevidence/nbla-hash
Windows
install_path\Veritas\NetBackup\logs\nblaevidence\nbla-hash
Supported job types:
Backup
Backup from Snapshot
Snapshot
Supported workload types:
File System
Hadoop (logs are only collected from primary and media servers)
Microsoft Exchange (logs are only collected from primary and media servers)
Windows Server Failover Cluster (WSFC)
Microsoft SQL Server Availability Group
NDMP (logs are only collected from primary and media servers)
Oracle
Snapshot Manager (logs are only collected from primary and media servers)
VMware
When you set the disableIPResolution option on a primary server, logs on the protected virtual machines are not gathered when you specify the job ID of the VMware workload type. See https://www.veritas.com/content/support/en_US/doc/21902280-158271263-0/v38310204-158271263 for more details of the setting.
Gathering logs from distributed workloads with multiple clients is supported with this release. Examples of distributed workloads include Oracle RAC and MSSQL availability groups.
You can upload the gathered logs to the Cohesity Technical Support organization with the command line interface and the API options as well as a valid support case ID. See https://support.cohesity.com/s/article/article-100038665 for more details.
The password that is provided to the API to upload the logs is stored as a credential object in the NetBackup credential management pane. It is removed after logs are uploaded.
A single tar file consisting of gathered logs is uploaded to the Cohesity Technical Support organization's SFTP server or the specified SFTP server. If the Cohesity Technical Support organization does not manage the SFTP server, the upload operation fails if a tar file with the same name exists on the SFTP server.
Use the nblogadm log to debug or troubleshoot log collection by job ID. Use the nblogadm log for both the command line interface and API option. To collect logs from the nblogadm process, confirm that the directory that is shown is present:
Linux and UNIX
/usr/openv/netbackup/logs/nblogadm
Windows
install_path\Veritas\NetBackup\logs\nblogadm
Table: New command line interface flags introduced to nblogadm utility
Command line interface | Description |
|---|---|
nblogadm --action getactivecollections --json | Get the number of records that are in-progress. (Does not collect logs for more than one record ID at a time) |
nblogadm --action createrecord --jobid job ID --json | Take a job ID, create an empty log record, and return the created record ID. |
nblogadm --action collectlogsforjob --recid record ID --runnbsu --json | Create a task to gather the logs for the specified record ID. |
nblogadm --action startupload --recid record ID --sftp_host sftp host --sftp_port sftp port --supportcase support case ID --target_folder sftp host folder --fingerprint sftp host fingerprint, use comma as delimiter without spaces --passcredentials --json | Create a task to upload the logs for the specified record ID and SFTP server access information. |
nblogadm --action deleterecord --recid record ID --json | Delete the collected logs and record for the specified record ID. This action also terminates any in-progress task. |
nblogadm --action casedetail --recid record ID --json | Get the log gather and the log upload task details for the specified record ID. |
nblogadm --action getlogging --recid record ID --json | Get the list of hosts, their components, and the corresponding logging level values for the specified record ID. |
nblogadm --action getlogging --recid record ID [--hostandlog MASTER|MEDIA|CLIENT:hostname] --json | When you specify the --hostandlog parameter, this command returns the components' logging level values of the specified host for the specified record ID. Without the --hostandlog parameter, the command returns the components' logging level values for the list of hosts for the specified record ID. |
nblogadm --action setlogging --recid record ID --hostandlog MASTER|MEDIA|CLIENT:hostname@legacy component1=legacy component1 level,vxul component1=debug level%diagnostic level,misc type=misc type value --json | Update the component's logging level settings of the specified host for the specified record ID. Separate calls are required to update each host. The specified legacy and vxul component names must be in lower case. |