Good post on reddit from /r/kcbnac:
As with many things in IT – It Depends.
SD card is identified by ESXi/vSphere as being flash-based media, so it won’t log to it – you’d need another volume somewhere for logs and dumps to go (not a bad idea anyway; but then if THAT goes down you have no logs – which isn’t helpful when the Vendor Blame Game begins)
SD is cheaper in a large cluster, provided you can meet the above logging requirement. (Saved ~$10k on 8 blade hosts buying SD cards instead of a RAID-1 mirror’d pair of drives for each; and nothing big/fancy either – would’ve been 300GB 10K SAS in Gen8 HP blades)
HDD/SSD gives you room on not-shared-storage to put some critical VMs in a pinch, or just ones that can’t live on the SAN. (I’m picturing the VMs that assist the SAN that shouldn’t live on the arrays they support; or extra space in a shuffle)
HDD uses more power, generating more heat. Another moving part.
Its a choice of tradeoffs, as usual.