Symmetry structure of special geometries