Scanning tunneling and atomic force microscopy study of layered transition metal halides Nb3X8 (X = Cl, Br, I)