Kosterlitz-Thouless transition in the smectic vortex state of a layered superconductor