Hadronic matrix elements of gluon operators in the instanton vacuum