Non-linear mode-mode coupling at a plasma-vacuum transition layer