Matches in SemOpenAlex for { <https://semopenalex.org/work/W4386148429> ?p ?o ?g. }
Showing items 1 to 79 of
79
with 100 items per page.
- W4386148429 abstract "With the increasing size of pre-trained language models (PLMs), fine-tuning all the parameters in the model is not efficient, especially when there are a large number of downstream tasks, which incur significant training and storage costs. Many parameter-efficient fine-tuning (PEFT) approaches have been proposed, among which, Low-Rank Adaptation (LoRA) is a representative approach that injects trainable rank decomposition matrices into every target module. Yet LoRA ignores the importance of parameters in different modules. To address this problem, many works have been proposed to prune the parameters of LoRA. However, under limited training conditions, the upper bound of the rank of the pruned parameter matrix is still affected by the preset values. We, therefore, propose IncreLoRA, an incremental parameter allocation method that adaptively adds trainable parameters during training based on the importance scores of each module. This approach is different from the pruning method as it is not limited by the initial number of training parameters, and each parameter matrix has a higher rank upper bound for the same training overhead. We conduct extensive experiments on GLUE to demonstrate the effectiveness of IncreLoRA. The results show that our method owns higher parameter efficiency, especially when under the low-resource settings where our method significantly outperforms the baselines. Our code is publicly available." @default.
- W4386148429 created "2023-08-25" @default.
- W4386148429 creator A5006131292 @default.
- W4386148429 creator A5046672528 @default.
- W4386148429 creator A5058659834 @default.
- W4386148429 creator A5064412438 @default.
- W4386148429 creator A5069295779 @default.
- W4386148429 creator A5071266785 @default.
- W4386148429 date "2023-08-23" @default.
- W4386148429 modified "2023-10-16" @default.
- W4386148429 title "IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuning" @default.
- W4386148429 doi "https://doi.org/10.48550/arxiv.2308.12043" @default.
- W4386148429 hasPublicationYear "2023" @default.
- W4386148429 type Work @default.
- W4386148429 citedByCount "0" @default.
- W4386148429 crossrefType "posted-content" @default.
- W4386148429 hasAuthorship W4386148429A5006131292 @default.
- W4386148429 hasAuthorship W4386148429A5046672528 @default.
- W4386148429 hasAuthorship W4386148429A5058659834 @default.
- W4386148429 hasAuthorship W4386148429A5064412438 @default.
- W4386148429 hasAuthorship W4386148429A5069295779 @default.
- W4386148429 hasAuthorship W4386148429A5071266785 @default.
- W4386148429 hasBestOaLocation W43861484291 @default.
- W4386148429 hasConcept C106487976 @default.
- W4386148429 hasConcept C108010975 @default.
- W4386148429 hasConcept C111919701 @default.
- W4386148429 hasConcept C11413529 @default.
- W4386148429 hasConcept C114614502 @default.
- W4386148429 hasConcept C119857082 @default.
- W4386148429 hasConcept C126255220 @default.
- W4386148429 hasConcept C134306372 @default.
- W4386148429 hasConcept C159985019 @default.
- W4386148429 hasConcept C164226766 @default.
- W4386148429 hasConcept C177264268 @default.
- W4386148429 hasConcept C192562407 @default.
- W4386148429 hasConcept C199360897 @default.
- W4386148429 hasConcept C2776760102 @default.
- W4386148429 hasConcept C2779960059 @default.
- W4386148429 hasConcept C33923547 @default.
- W4386148429 hasConcept C41008148 @default.
- W4386148429 hasConcept C6557445 @default.
- W4386148429 hasConcept C77553402 @default.
- W4386148429 hasConcept C86803240 @default.
- W4386148429 hasConceptScore W4386148429C106487976 @default.
- W4386148429 hasConceptScore W4386148429C108010975 @default.
- W4386148429 hasConceptScore W4386148429C111919701 @default.
- W4386148429 hasConceptScore W4386148429C11413529 @default.
- W4386148429 hasConceptScore W4386148429C114614502 @default.
- W4386148429 hasConceptScore W4386148429C119857082 @default.
- W4386148429 hasConceptScore W4386148429C126255220 @default.
- W4386148429 hasConceptScore W4386148429C134306372 @default.
- W4386148429 hasConceptScore W4386148429C159985019 @default.
- W4386148429 hasConceptScore W4386148429C164226766 @default.
- W4386148429 hasConceptScore W4386148429C177264268 @default.
- W4386148429 hasConceptScore W4386148429C192562407 @default.
- W4386148429 hasConceptScore W4386148429C199360897 @default.
- W4386148429 hasConceptScore W4386148429C2776760102 @default.
- W4386148429 hasConceptScore W4386148429C2779960059 @default.
- W4386148429 hasConceptScore W4386148429C33923547 @default.
- W4386148429 hasConceptScore W4386148429C41008148 @default.
- W4386148429 hasConceptScore W4386148429C6557445 @default.
- W4386148429 hasConceptScore W4386148429C77553402 @default.
- W4386148429 hasConceptScore W4386148429C86803240 @default.
- W4386148429 hasLocation W43861484291 @default.
- W4386148429 hasOpenAccess W4386148429 @default.
- W4386148429 hasPrimaryLocation W43861484291 @default.
- W4386148429 hasRelatedWork W1482618134 @default.
- W4386148429 hasRelatedWork W1998812252 @default.
- W4386148429 hasRelatedWork W2017983317 @default.
- W4386148429 hasRelatedWork W2028024605 @default.
- W4386148429 hasRelatedWork W2326122716 @default.
- W4386148429 hasRelatedWork W2373724792 @default.
- W4386148429 hasRelatedWork W2496161296 @default.
- W4386148429 hasRelatedWork W2625550807 @default.
- W4386148429 hasRelatedWork W2767525681 @default.
- W4386148429 hasRelatedWork W3047144510 @default.
- W4386148429 isParatext "false" @default.
- W4386148429 isRetracted "false" @default.
- W4386148429 workType "article" @default.