Optimizing DistilBERT for Low-Resource Sentiment Analysis: A Study on Global Unstructured L1-Norm Pruning and Accuracy-Efficiency Trade-Offs

Transformer models are not always deployed to low-resource devices due to the computational intensity of the model. Although some distilled models, such as DistilBERT, provide a minimum level of efficiency, additional compression is required to apply to any edge application. This paper offers a systematic evaluation of L1-norm unstructured pruning globally on a DistilBERT model that has been fine-tuned on binary sentiment analysis on the Stanford Sentiment Treebank (SST-2) dataset. The approach consists of pruning the model sequentially by 10 % to 60 % to check its performance using the accuracy of a stratified validation set and inference latency on a CPU. These findings indicate that model accuracy is very robust, and it still has more than 99% of the baseline performance until 40% sparsity. Only at this stage major degradation has taken place, and the accuracy retention was found to be 97.25 % and 94.22% at <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{5 0 \%}$</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{6 0 \%}$</tex> sparsity, respectively. An important conclusion here is that this unstructured sparsity fails to bring inference speedups in conventional CPU hardware, highlighting a lack of a connection between theory and practical efficiency improvements. As a result, the paper has found the best operational point to be <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{5 0 \%}$</tex> sparsity, which is an evidence-based guide to developers. The research also concludes that L1-norm unstructured pruning is a very effective method of maximizing model compression without significant accuracy impairment and thus can be used to achieve efficient deployment of NLP models in resource-constrained settings.