Adaptive Fusion of Low-Rank and Soft-Gated Sparse Updates for Parameter-Efficient Instruction Tuning

Abstract

This paper addresses the high cost and management challenges of fine-tuning large-scale pre-trained models in instruction-based scenarios, proposing a parameter-efficient fine-tuning algorithm framework that integrates low-rank adaptation and sparse updates. Under the premise of freezing basic parameters, linear layer weight updates are represented as structured incremental stacking. The low-rank component characterizes the main update direction in a two-factor manner, providing compressible global expressiveness; the sparse component generates soft masks through continuous gating and selects free increments element-wise, achieving selective reshaping of key connections. Furthermore, an intra-layer fusion coefficient is introduced to coordinate the contribution ratio of the two types of increments, and the update complexity is explicitly incorporated into the optimization objective through low-rank norm constraints and gating density regularization, thus balancing update stability and expressive flexibility under a unified training objective. For data processing, an open-source instruction-based fine-tuning dataset is used, and a standard input-output pairing and label masking mechanism is constructed to ensure that the training signal focuses on the response generation segment. Comparative experiments evaluate the proposed method and representative parameter-efficient fine-tuning strategies from dimensions such as language modeling reliability, semantic consistency, and answer matching, verifying that this integrated structural update has superior comprehensive performance across multiple metrics.