Existence of Markov Controls and Characterization of Optimal Markov Controls