LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional
Adaptation
LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional
Adaptation
Low-rank adaptation (LoRA) has become the default approach to fine-tune large language models (LLMs) due to its significant reduction in trainable parameters. However, trainable parameter demand for LoRA increases with increasing model embedding dimensions, leading to high compute costs. Additionally, its backward updates require storing high-dimensional intermediate activations and optimizer …