Abstract
This paper presents a rigorous theoretical investigation of the filamentation and modulational instabilities of an upper hybrid laser radiation in a hot, collisionless and homogeneous plasma in the presence of the self-generated dc magnetic field of the order of a few megagauss. Fluid equations have been employed to find the nonlinear response of electrons. The low-frequency nonlinearity arises through the ponderomotive force on electrons, whereas the high-frequency nonlinearity arises through the current densities associated with the scattered sidebands. It is observed that the growth rate of the filamentation instability decreases with increasing magnetic field, while the growth rate of the modulational instability increases with increasing magnetic field. Furthermore, the growth rate of the filamentation instability is higher by about one order of magnitude than that of the modulational instability for the same set of plasma parameters.