Ferromagnetic resonance foldover and spin-wave instability in single-crystal YIG films