Matches in SemOpenAlex for { <https://semopenalex.org/work/W4385681736> ?p ?o ?g. }
Showing items 1 to 63 of
63
with 100 items per page.
- W4385681736 abstract "The low-rank adaptation (LoRA) method can largely reduce the amount of trainable parameters for fine-tuning large language models (LLMs), however, it still requires expensive activation memory to update low-rank weights. Reducing the number of LoRA layers or using activation recomputation could harm the fine-tuning performance or increase the computational overhead. In this work, we present LoRA-FA, a memory-efficient fine-tuning method that reduces the activation memory without performance degradation and expensive recomputation. LoRA-FA chooses to freeze the projection-down weight of $A$ and update the projection-up weight of $B$ in each LoRA layer. It ensures the change of model weight reside in a low-rank space during LLMs fine-tuning, while eliminating the requirement to store full-rank input activations. We conduct extensive experiments across multiple model types (RoBERTa, T5, LLaMA) and model scales. Our results show that LoRA-FA can always achieve close fine-tuning accuracy across different tasks compared to full parameter fine-tuning and LoRA. Furthermore, LoRA-FA can reduce the overall memory cost by up to 1.4$times$ compared to LoRA." @default.
- W4385681736 created "2023-08-09" @default.
- W4385681736 creator A5016836702 @default.
- W4385681736 creator A5021541475 @default.
- W4385681736 creator A5064192009 @default.
- W4385681736 creator A5080982309 @default.
- W4385681736 creator A5090592237 @default.
- W4385681736 date "2023-08-07" @default.
- W4385681736 modified "2023-09-29" @default.
- W4385681736 title "LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning" @default.
- W4385681736 doi "https://doi.org/10.48550/arxiv.2308.03303" @default.
- W4385681736 hasPublicationYear "2023" @default.
- W4385681736 type Work @default.
- W4385681736 citedByCount "0" @default.
- W4385681736 crossrefType "posted-content" @default.
- W4385681736 hasAuthorship W4385681736A5016836702 @default.
- W4385681736 hasAuthorship W4385681736A5021541475 @default.
- W4385681736 hasAuthorship W4385681736A5064192009 @default.
- W4385681736 hasAuthorship W4385681736A5080982309 @default.
- W4385681736 hasAuthorship W4385681736A5090592237 @default.
- W4385681736 hasBestOaLocation W43856817361 @default.
- W4385681736 hasConcept C11413529 @default.
- W4385681736 hasConcept C114614502 @default.
- W4385681736 hasConcept C120665830 @default.
- W4385681736 hasConcept C121332964 @default.
- W4385681736 hasConcept C139807058 @default.
- W4385681736 hasConcept C157524613 @default.
- W4385681736 hasConcept C164226766 @default.
- W4385681736 hasConcept C199360897 @default.
- W4385681736 hasConcept C2779960059 @default.
- W4385681736 hasConcept C33923547 @default.
- W4385681736 hasConcept C41008148 @default.
- W4385681736 hasConcept C57493831 @default.
- W4385681736 hasConcept C62520636 @default.
- W4385681736 hasConceptScore W4385681736C11413529 @default.
- W4385681736 hasConceptScore W4385681736C114614502 @default.
- W4385681736 hasConceptScore W4385681736C120665830 @default.
- W4385681736 hasConceptScore W4385681736C121332964 @default.
- W4385681736 hasConceptScore W4385681736C139807058 @default.
- W4385681736 hasConceptScore W4385681736C157524613 @default.
- W4385681736 hasConceptScore W4385681736C164226766 @default.
- W4385681736 hasConceptScore W4385681736C199360897 @default.
- W4385681736 hasConceptScore W4385681736C2779960059 @default.
- W4385681736 hasConceptScore W4385681736C33923547 @default.
- W4385681736 hasConceptScore W4385681736C41008148 @default.
- W4385681736 hasConceptScore W4385681736C57493831 @default.
- W4385681736 hasConceptScore W4385681736C62520636 @default.
- W4385681736 hasLocation W43856817361 @default.
- W4385681736 hasOpenAccess W4385681736 @default.
- W4385681736 hasPrimaryLocation W43856817361 @default.
- W4385681736 hasRelatedWork W1571518467 @default.
- W4385681736 hasRelatedWork W1576801573 @default.
- W4385681736 hasRelatedWork W2001850503 @default.
- W4385681736 hasRelatedWork W2020291234 @default.
- W4385681736 hasRelatedWork W2028024605 @default.
- W4385681736 hasRelatedWork W2094520212 @default.
- W4385681736 hasRelatedWork W2496161296 @default.
- W4385681736 hasRelatedWork W2625550807 @default.
- W4385681736 hasRelatedWork W2767525681 @default.
- W4385681736 hasRelatedWork W87991986 @default.
- W4385681736 isParatext "false" @default.
- W4385681736 isRetracted "false" @default.
- W4385681736 workType "article" @default.