When we setup our environment over two years ago we set it up with the MTU of 9000 in the LAN Cloud QOS System class. We are a FlexPod reference design, running iSCSI for our VMware storage. This worked fine up until we got new controllers for our cluster mode; FAS8060 with NetApps UTA2 ports. We were not able to run Jumbo frame storage connections to these. After over a month of troubleshooting, it came down to the UCS Fabric Interconnections were dropping the Jumbo packets.
Fix: Change the MTU from 9000 to 9216 in the LAN Cloud QOS System class. Once this was done everything connected and tested fine.
Reason: As I understand it, our other controllers used on straight 10GB Ethernet adapters. However, the new UTA2 adapters on the NetApp are CNA cards and act as FCoE even when running Ethernet only traffic. Because of that there is extra packet header information. This caused the packet to be larger than 9000 and the FI dropped it.
As stated above there is a lot of information about Jumbo Frames with the NetApp and UCS with conflicting configurations. We found our original docs used and it had conflicting settings in it as well. However, if you do as mentioned above use the FlexPod reference design it does state to use 9216 for your settings. I would say if you are connecting to a NetApp, even if you are not doing a reference design, use the design as a guide for settings.
