Robbie Cannings

SFO Aggregate giveback failed

Robbie Cannings -

When performing a cf takeover as part of a Data ONTAP update, I encountered this error

                              Takeover
Node           Partner        Possible State Description
-------------- -------------- -------- -------------------------------------
CLUSTER01-01
           CLUSTER01-02  true     Connected to CLUSTER01-02
CLUSTER01-02
           CLUSTER01-01  true     Connected to CLUSTER01-01,
                                  Giveback of one or more SFO
                                  aggregates failed

When I looked at the storage aggregate relocation show, I found the giveback was vetoed by a lock.

CLUSTER01::*> storage aggregate relocation show

Source         Aggregate  Destination   Relocation Status
-------------- ---------- -----------   -----------------
CLUSTER01-01
               -          -             Not attempted yet
CLUSTER01-02
               node_01_t3_aggr01
                          CLUSTER01-01
                                        Failed: Operation was vetoed by lock_manager. Check the event log.

According to the logs, it turns out the veto was caused by CIFS locks

CLUSTER01::> event log show
Time                Node             Severity      Event
------------------- ---------------- ------------- ---------------------------
11/26/2014 15:21:33 CLUSTER01-02  ERROR         aggr.relocation.failed: Relocation of aggregate node_01_t3_aggr01 failed by node CLUSTER01-02 because operation was vetoed.
11/26/2014 15:21:33 CLUSTER01-02  ERROR         arl.subsystemAbort: The relocation operation of 'node_01_t3_aggr01' was aborted by 'lock_manager'.
11/26/2014 15:21:33 CLUSTER01-02  ERROR         lmgr.arl.nonCA.locks.exist: Could not complete aggregate relocation because of non-CA locks on volume     vol_21@vserver:30382217-525c-22b5-a431-123473543532.

According to this NetApp relocation veto documentation, this is likely to be because of SMB locking.

I simply stopped CIFS, and the aggregate relocate completed successfully.

This raises a few concerns around the objectives of "Non disruptive Operations" for CDOT, but at least I know for next time, there may be a requirement to completely stop CIFS.

The 8.2 upgrade/revert guide does indicate that you should expect outages for CIFS (yes, buried in fine print somewhere in the 140 page document), so I guess I know for next time, simply schedule outages for any operations which involve aggregate relocations (e.g. NDO upgrades).