The potential for higher spectral efficiency and recent activities in the area of coherent optical systems raise the interest in new multi-level modulation formats for optical transmission. In this paper square-16-QAM is introduced as a novel candidate for future high-capacity and high spectrally efficient optical systems. Different possible transmitters as well as the homodyne IQ-receiver are illustrated. At the transmitter side the implementation of an electrical level-generator is shown experimentally. At the receiver side options for the realization of the optical 2/spl times/4 90 /spl deg/-hybrid are discussed. Furthermore, an algorithm for square-16-QAM digital phase estimation is developed, whose performance and limitations are investigated in simulations.