Removing instrumental polarization from infrared solar polarimetric observations