Directional memory effects in inelastic photoelectron scattering