Experimental study of the dissipative trapped ion instability