Numerical study of a diffraction problem by a modified method of non-orthogonal series