The client direct backup fails with status code 50 due to client memory exhaustion

When running a backup job, you may observe a failure with status code 50: client process aborted in the Activity Monitor. This issue is often caused by insufficient memory on the client system, leading to the termination of the nbostpxy process by the operating system.

Check for the following on the Activity Monitor:

Info nbjm: started backup job for client <client_name>, policy <policy_name>, schedule Full on storage unit <storage_unit>
Info bpbrm: started process
Info bpbrm: connecting
Info bpbrm: connected
Info bpbrm: begin writing
Critical bpbrm: unexpected termination of client <backup_id>
Error bptm: media manager terminated by parent process
Info dbclient: done. status: 50: client process aborted

Check /var/log/messages for out of memory events. You may see entries like:

Out of memory: Killed process <PID> (nbostpxy) total-vm:<value>kB, anon-rss:<value>kB, ...

Run free -g on the client. If the output shows zero free memory, it confirms that the system is under memory pressure:

              total        used        free      shared  buff/cache   available
Mem:             15          12           0           0           2           1
Swap:             7           6           1

Cause:

The nbostpxy process allocates a large write buffer (default: 512 MB per instance). On systems with limited memory, this allocation can lead to excessive memory usage and OOM conditions, causing intermittent backup failures.

Resolution:

To resolve this issue, reduce the size of the write buffer that is allocated by nbostpxy. By default, each nbostpxy instance allocates 512 MB for writing. You can lower this value to prevent excessive thrashing by setting the maxWriteBufferSizeMB and maxPendingWriteDataSizeMB parameters in the spanfs.conf file.

For example:

{
    "logLevel": 5,
    "logSize": 30,
    "logPath": "/var/log/spanfs",
    "writeParams": {
        "maxWriteBufferSizeMB": 16,
        "maxPendingWriteDataSizeMB": 8
    }