Gluon-mediated rare decays of the top quark: Anomalous threshold and its phenomenological consequences

Flavor-changing neutral-current decays of the top quark by one-gluon emission (e.g., $t\to \mathrm{cg}$) and exchange (e.g., $t\to c{g}^{*}\to c\overline{q}q$) are analyzed in the three-generation standard and two-Higgs-doublet models. These rare decays provide a sensitive probe of the analytic properties of (penguin-dominated) loop-induced neutral currents. The occurrence of an anomalous threshold in the amplitude for the one-gluon exchange mode requires a careful description of unstable (or off-mass-shell) external particles, in order to obtain sensible results for physical quantities. The branching fraction $\Sigma q^{}B(t\to c{g}^{*}\to c\overline{q}q)$ in the standard model turns out to be larger than $B(t\to cg)$ for top quark masses ≤ 180 GeV, due in part to the effects of the anomalous threshold. This is despite a suppression of the exchange mode by a factor of order $\frac{{\alpha }_s\left(m_t^2\right)}{12\pi }\approx 3\times {10}^{-3\mathrm{}}$, coming from coupling constants and phase space. The largest branching fractions occur in the two-Higgs-doublet models, where $B(t\to cg)\sim {10}^{-5\mathrm{}}$ and $\Sigma q^{}B(t\to c{g}^{*}\to c\overline{q}q)\sim {10}^{-6\mathrm{}}$ in some regions of parameter space.