The Problem
We recently came across a situation where SnapVault backups were forced by DFM to flow over the 100Mbps management NIC on the production filer.
Also worth pointing out - even though I would always advise using hostnames (so much easier to modify the underlying infrastructure later), DFM automatically assigns source/destination so it is not possible to specify this. Strangely, it uses a mixture of IP address and hostname - I haven't yet figured out the pattern and what triggers one approach or another.
In DFM we were seeing errors such as
cus_vfl01: Host FILER01 has IP address (192.168.0.200) which is different from the primaryHostAddress (172.22.60.201) stored in the database. Change the IP address in the database using hostPrimaryAddress option or use global option processHostPrimaryAddress to disable this message or to select a different behavior.
Steps attempted
I tried resolving this by disabling snapvault over e0M and specifying the subnets for snapvault.
FILER01> options snapvault.access
snapvault.access host=*
FILER01> options snapvault.access "host=192.168.0.0/24,10.20.30.0/27 AND if!=e0M"
FILER01> options snapvault.access
snapvault.access host=192.168.0.0/24,10.20.30.0/27 AND if!=e0M
However the problem still persisted.
I tried manually modifying the snapvault entries but again with no success
FILER02> snapvault status
Snapvault is ON.
Source Destination State Lag Status
172.22.60.201:/vol/vol01/nfs01 FILER2:/vol/vol01/nfs01 Snapvaulted 80:56:29 Idle
FILER2> snapvault modify -S 192.168.0.200:/vol/vol01/nfs01 FILER02:/vol/vol01/nfs01
Configuration for qtree /vol/vol01/nfs01 is:
/vol/vol01/nfs01 source=192.168.0.200:/vol/vol01/nfs01 kbs=unlimited tries=2 connection_mode=inet wsize=0 back_up_open_files=on,ignore_atime=off,utf8_primary_path=off
FILER02> snapvault status
Snapvault is ON.
Source Destination State Lag Status
172.22.60.201:/vol/vol01/nfs01 FILER2:/vol/vol01/nfs01 Snapvaulted 80:56:29 Idle
Note - NetApp helpdesk advised that it is not possible to modify the source IP address for a snapvault relationship. As shown later, this advice was not correct and we can modify the source IP.
I even tried rebooting the filer (after which snapvault status showed the correct source IP address, but as soon as DFM kicked off a new data protection job, it instantly reverted to e0M.
The Solution
DFM uses a field called HostPrimaryAddress to determine which interface to use when performing data protection tasks. Unfortunately this is a global (per filer) option, so you cannot force Customer A over one link, Customer B over a different link etc, nor can you split snapvault and snapmirror over different links. Not sure if C-DOT DFM has brought any improvements to this (we are yet to adopt CommVault 10 so can't test this yet). Equally disappointing is that DFM is unable to send management traffic over e0M (i.e. the "management" interface), it must use in-band communications for control traffic.
C:\Users\Administrator>dfm host get FILER01
Host: FILER01
Login: dfm_svc
Password: ********
Login Protocol: ssh
Preferred IP address 1:
Preferred IP address 2:
Primary IP Address: 172.22.60.201
Remote Platform Management IP Address:
Console Terminal Server Address:
...
C:\Users\Administrator>dfm host set FILER01 HostPrimaryAddress=192.168.0.200
C:\Users\Administrator>dfm host get FILER01
Host: FILER01
Login: dfm_svc
Password: ********
Login Protocol: ssh
Preferred IP address 1:
Preferred IP address 2:
Primary IP Address: 192.168.0.200
Remote Platform Management IP Address:
Console Terminal Server Address:
...
- Important - SNMP must be permitted between the DFM server and the backup-transport interface on the filer. If you have a firewall, port 161 must be opened.
- Additionally (thanks to my colleague Anuraag) - HTTPS must also be permitted over the same interrface between DFM and the filer, to facilitate ZAPI requests
And that's that - backups now working great!!