Theoretical analysis of instabilities in optically pumped molecular lasers