A nuclear scaffold (NS) protease has previously been implicated in production of the M(r) 46,000 ATP-binding protein in NS (which may acquire nucleoside triphosphatase activity and participate in nucleocytoplasmic transport) by cleavage of a subset of lamins A/C. In a preceding paper (G. Clawson, L. Norbeck, C. Hatem, C. Rhodes, P. Amiri, J. McKerrow, S. Patierno, and G. Fiskum, Cell Growth & Differ., 3: 827-838), this NS protease was identified as a novel, Ca(2+)-regulated serine protease, which was found only in the NS and which appears to represent a unique multicatalytic protease complex. Based upon its predominantly chymotrypsin-like substrate preference, a peptide-chloromethylketone inhibitor (succinyl-AAPF-chloromethylketone, AAPFcmk) was identified. AAPFcmk showed a KI = 56 nM for the NS protease versus 1.4 microM for the endoplasmic reticulum activity. Treatment of C3H/10T1/2 mouse embryo fibroblast cells with 1 microM AAPFcmk produced effects which were confined to the nuclear (and to a lesser extent the endoplasmic reticulum) compartment. In this report, we examine the effects of the AAPFcmk inhibitor on cellular transformation and growth. Growth of C3H/10T1/2 cells was decreased by 34% and 56% at 25 microM and 50 microM AAPFcmk, respectively. Growth inhibition occurred without any major change in DNA content distribution, suggesting effects throughout the cell cycle. Growth inhibition was not observed at lower (< or = 10 microM) concentrations, which decreased transformation of C3H/10T1/2 fibroblasts in a dose-dependent manner by up to 90%, even at femtomolar concentrations of AAPFcmk (in the absence of growth inhibition). Inclusion of irrelevant inhibitors was without affect.(ABSTRACT TRUNCATED AT 250 WORDS)