Coherence effects in hybridly mode-locked dye lasers: theory and experiment