Layer-Wise Structural Mapping for Efficient Domain Transfer in Language Model Distillation

Abstract

This paper addresses the challenges of high computational cost and low semantic transfer efficiency in adapting large language models to specific domains. It proposes a domain-oriented knowledge distillation framework for large language models. The framework employs a teacher-student architecture to enable model compression and knowledge transfer. On this basis, it incorporates a structural alignment mechanism and a domain-aware module to enhance the student model's ability to represent domain-specific semantic structures. Specifically, the teacher model first constructs a domain representation based on the raw input. This representation is then projected into a unified semantic space through structural mapping. At the same time, the student model is guided to learn semantic representations and domain features layer by layer. To improve semantic compression efficiency, the student model integrates a multi-granularity aggregation mechanism. This component structurally fuses semantic information, enhancing the compactness and consistency of representations. In the experimental section, multiple sensitivity experiments are designed to evaluate the impact of distillation depth, projection dimension, and sampling strategy. The evaluation focuses on the student model's ability to align semantics and model domain features. Comparative analysis based on real-world domain datasets shows that the proposed method outperforms several mainstream distillation baselines. It achieves better performance in semantic retention, structural consistency, and model efficiency. These results confirm the effectiveness and robustness of the proposed approach in domain adaptation tasks.