Matches in SemOpenAlex for { <https://semopenalex.org/work/W4382322415> ?p ?o ?g. }
Showing items 1 to 77 of
77
with 100 items per page.
- W4382322415 abstract "The components underpinning PLMs -- large weight matrices -- were shown to bear considerable redundancy. Matrix factorization, a well-established technique from matrix theory, has been utilized to reduce the number of parameters in PLM. However, it fails to retain satisfactory performance under moderate to high compression rate. In this paper, we identify the textit{full-rankness} of fine-tuned PLM as the fundamental bottleneck for the failure of matrix factorization and explore the use of network pruning to extract low-rank sparsity pattern desirable to matrix factorization. We find such low-rank sparsity pattern exclusively exists in models generated by first-order pruning, which motivates us to unite the two approaches and achieve more effective model compression. We further propose two techniques: sparsity-aware SVD and mixed-rank fine-tuning, which improve the initialization and training of the compression procedure, respectively. Experiments on GLUE and question-answering tasks show that the proposed method has superior compression-performance trade-off compared to existing approaches." @default.
- W4382322415 created "2023-06-28" @default.
- W4382322415 creator A5026808863 @default.
- W4382322415 creator A5074279178 @default.
- W4382322415 date "2023-06-25" @default.
- W4382322415 modified "2023-09-25" @default.
- W4382322415 title "Low-Rank Prune-And-Factorize for Language Model Compression" @default.
- W4382322415 doi "https://doi.org/10.48550/arxiv.2306.14152" @default.
- W4382322415 hasPublicationYear "2023" @default.
- W4382322415 type Work @default.
- W4382322415 citedByCount "0" @default.
- W4382322415 crossrefType "posted-content" @default.
- W4382322415 hasAuthorship W4382322415A5026808863 @default.
- W4382322415 hasAuthorship W4382322415A5074279178 @default.
- W4382322415 hasBestOaLocation W43823224151 @default.
- W4382322415 hasConcept C106487976 @default.
- W4382322415 hasConcept C108010975 @default.
- W4382322415 hasConcept C111919701 @default.
- W4382322415 hasConcept C11413529 @default.
- W4382322415 hasConcept C114466953 @default.
- W4382322415 hasConcept C114614502 @default.
- W4382322415 hasConcept C121332964 @default.
- W4382322415 hasConcept C149635348 @default.
- W4382322415 hasConcept C152124472 @default.
- W4382322415 hasConcept C158693339 @default.
- W4382322415 hasConcept C159985019 @default.
- W4382322415 hasConcept C164226766 @default.
- W4382322415 hasConcept C180016635 @default.
- W4382322415 hasConcept C187834632 @default.
- W4382322415 hasConcept C192562407 @default.
- W4382322415 hasConcept C199360897 @default.
- W4382322415 hasConcept C2780513914 @default.
- W4382322415 hasConcept C33923547 @default.
- W4382322415 hasConcept C41008148 @default.
- W4382322415 hasConcept C42355184 @default.
- W4382322415 hasConcept C62520636 @default.
- W4382322415 hasConcept C6557445 @default.
- W4382322415 hasConcept C86803240 @default.
- W4382322415 hasConceptScore W4382322415C106487976 @default.
- W4382322415 hasConceptScore W4382322415C108010975 @default.
- W4382322415 hasConceptScore W4382322415C111919701 @default.
- W4382322415 hasConceptScore W4382322415C11413529 @default.
- W4382322415 hasConceptScore W4382322415C114466953 @default.
- W4382322415 hasConceptScore W4382322415C114614502 @default.
- W4382322415 hasConceptScore W4382322415C121332964 @default.
- W4382322415 hasConceptScore W4382322415C149635348 @default.
- W4382322415 hasConceptScore W4382322415C152124472 @default.
- W4382322415 hasConceptScore W4382322415C158693339 @default.
- W4382322415 hasConceptScore W4382322415C159985019 @default.
- W4382322415 hasConceptScore W4382322415C164226766 @default.
- W4382322415 hasConceptScore W4382322415C180016635 @default.
- W4382322415 hasConceptScore W4382322415C187834632 @default.
- W4382322415 hasConceptScore W4382322415C192562407 @default.
- W4382322415 hasConceptScore W4382322415C199360897 @default.
- W4382322415 hasConceptScore W4382322415C2780513914 @default.
- W4382322415 hasConceptScore W4382322415C33923547 @default.
- W4382322415 hasConceptScore W4382322415C41008148 @default.
- W4382322415 hasConceptScore W4382322415C42355184 @default.
- W4382322415 hasConceptScore W4382322415C62520636 @default.
- W4382322415 hasConceptScore W4382322415C6557445 @default.
- W4382322415 hasConceptScore W4382322415C86803240 @default.
- W4382322415 hasLocation W43823224151 @default.
- W4382322415 hasOpenAccess W4382322415 @default.
- W4382322415 hasPrimaryLocation W43823224151 @default.
- W4382322415 hasRelatedWork W1195508317 @default.
- W4382322415 hasRelatedWork W2022065959 @default.
- W4382322415 hasRelatedWork W2050515752 @default.
- W4382322415 hasRelatedWork W2074550915 @default.
- W4382322415 hasRelatedWork W2823336313 @default.
- W4382322415 hasRelatedWork W2903666957 @default.
- W4382322415 hasRelatedWork W2951579230 @default.
- W4382322415 hasRelatedWork W2991853703 @default.
- W4382322415 hasRelatedWork W3021308261 @default.
- W4382322415 hasRelatedWork W4289117577 @default.
- W4382322415 isParatext "false" @default.
- W4382322415 isRetracted "false" @default.
- W4382322415 workType "article" @default.