The recovery phase of galactic cosmic ray modulation in the outer heliosphere