Matches in SemOpenAlex for { <https://semopenalex.org/work/W4375959188> ?p ?o ?g. }
Showing items 1 to 65 of
65
with 100 items per page.
- W4375959188 abstract "To fully leverage the advantages of large-scale pre-trained language models (PLMs) on downstream tasks, it has become a ubiquitous adaptation paradigm to fine-tune the entire parameters of PLMs. However, this paradigm poses issues of inefficient updating and resource over-consuming for fine-tuning in data-scarce and resource-limited scenarios, because of the large scale of parameters in PLMs. To alleviate these concerns, in this paper, we propose a parameter-efficient fine-tuning method HiFi, that is, only the highly informative and strongly correlated attention heads for the specific task are fine-tuned. To search for those significant attention heads, we develop a novel framework to analyze the effectiveness of heads. Specifically, we first model the relationship between heads into a graph from two perspectives of information richness and correlation, and then apply PageRank algorithm to determine the relative importance of each head. Extensive experiments on the GLUE benchmark demonstrate the effectiveness of our method, and show that HiFi obtains state-of-the-art performance over the prior baselines." @default.
- W4375959188 created "2023-05-10" @default.
- W4375959188 creator A5063826779 @default.
- W4375959188 creator A5069678275 @default.
- W4375959188 date "2023-05-08" @default.
- W4375959188 modified "2023-09-29" @default.
- W4375959188 title "HiFi: High-Information Attention Heads Hold for Parameter-Efficient Model Adaptation" @default.
- W4375959188 doi "https://doi.org/10.48550/arxiv.2305.04573" @default.
- W4375959188 hasPublicationYear "2023" @default.
- W4375959188 type Work @default.
- W4375959188 citedByCount "0" @default.
- W4375959188 crossrefType "posted-content" @default.
- W4375959188 hasAuthorship W4375959188A5063826779 @default.
- W4375959188 hasAuthorship W4375959188A5069678275 @default.
- W4375959188 hasBestOaLocation W43759591881 @default.
- W4375959188 hasConcept C119857082 @default.
- W4375959188 hasConcept C120665830 @default.
- W4375959188 hasConcept C121332964 @default.
- W4375959188 hasConcept C132525143 @default.
- W4375959188 hasConcept C13280743 @default.
- W4375959188 hasConcept C139807058 @default.
- W4375959188 hasConcept C153083717 @default.
- W4375959188 hasConcept C154945302 @default.
- W4375959188 hasConcept C162324750 @default.
- W4375959188 hasConcept C185798385 @default.
- W4375959188 hasConcept C187736073 @default.
- W4375959188 hasConcept C205649164 @default.
- W4375959188 hasConcept C206345919 @default.
- W4375959188 hasConcept C2780451532 @default.
- W4375959188 hasConcept C31258907 @default.
- W4375959188 hasConcept C41008148 @default.
- W4375959188 hasConcept C80444323 @default.
- W4375959188 hasConceptScore W4375959188C119857082 @default.
- W4375959188 hasConceptScore W4375959188C120665830 @default.
- W4375959188 hasConceptScore W4375959188C121332964 @default.
- W4375959188 hasConceptScore W4375959188C132525143 @default.
- W4375959188 hasConceptScore W4375959188C13280743 @default.
- W4375959188 hasConceptScore W4375959188C139807058 @default.
- W4375959188 hasConceptScore W4375959188C153083717 @default.
- W4375959188 hasConceptScore W4375959188C154945302 @default.
- W4375959188 hasConceptScore W4375959188C162324750 @default.
- W4375959188 hasConceptScore W4375959188C185798385 @default.
- W4375959188 hasConceptScore W4375959188C187736073 @default.
- W4375959188 hasConceptScore W4375959188C205649164 @default.
- W4375959188 hasConceptScore W4375959188C206345919 @default.
- W4375959188 hasConceptScore W4375959188C2780451532 @default.
- W4375959188 hasConceptScore W4375959188C31258907 @default.
- W4375959188 hasConceptScore W4375959188C41008148 @default.
- W4375959188 hasConceptScore W4375959188C80444323 @default.
- W4375959188 hasLocation W43759591881 @default.
- W4375959188 hasOpenAccess W4375959188 @default.
- W4375959188 hasPrimaryLocation W43759591881 @default.
- W4375959188 hasRelatedWork W112744582 @default.
- W4375959188 hasRelatedWork W1485630101 @default.
- W4375959188 hasRelatedWork W189527659 @default.
- W4375959188 hasRelatedWork W2076610045 @default.
- W4375959188 hasRelatedWork W2498017833 @default.
- W4375959188 hasRelatedWork W2961085424 @default.
- W4375959188 hasRelatedWork W2990267819 @default.
- W4375959188 hasRelatedWork W3193574253 @default.
- W4375959188 hasRelatedWork W4306674287 @default.
- W4375959188 hasRelatedWork W4367191088 @default.
- W4375959188 isParatext "false" @default.
- W4375959188 isRetracted "false" @default.
- W4375959188 workType "article" @default.