Matches in SemOpenAlex for { <https://semopenalex.org/work/W4322759986> ?p ?o ?g. }
Showing items 1 to 93 of
93
with 100 items per page.
- W4322759986 abstract "The training for deep neural networks (DNNs) demands immense energy consumption, which restricts the development of deep learning as well as increases carbon emissions. Thus, the study of energy-efficient training for DNNs is essential. In training, the linear layers consume the most energy because of the intense use of energy-consuming full-precision (FP32) multiplication in multiply-accumulate (MAC). The energy-efficient works try to decrease the precision of multiplication or replace the multiplication with energy-efficient operations such as addition or bitwise shift, to reduce the energy consumption of FP32 multiplications. However, the existing energy-efficient works cannot replace all of the FP32 multiplications during both forward and backward propagation with low-precision energy-efficient operations. In this work, we propose an Adaptive Layer-wise Scaling PoT Quantization (ALS-POTQ) method and a Multiplication-Free MAC (MF-MAC) to replace all of the FP32 multiplications with the INT4 additions and 1-bit XOR operations. In addition, we propose Weight Bias Correction and Parameterized Ratio Clipping techniques for stable training and improving accuracy. In our training scheme, all of the above methods do not introduce extra multiplications, so we reduce up to 95.8% of the energy consumption in linear layers during training. Experimentally, we achieve an accuracy degradation of less than 1% for CNN models on ImageNet and Transformer model on the WMT En-De task. In summary, we significantly outperform the existing methods for both energy efficiency and accuracy." @default.
- W4322759986 created "2023-03-03" @default.
- W4322759986 creator A5008497740 @default.
- W4322759986 creator A5009782388 @default.
- W4322759986 creator A5022256556 @default.
- W4322759986 creator A5036205023 @default.
- W4322759986 creator A5052252896 @default.
- W4322759986 creator A5056949827 @default.
- W4322759986 creator A5065037360 @default.
- W4322759986 creator A5072861069 @default.
- W4322759986 date "2023-02-28" @default.
- W4322759986 modified "2023-10-16" @default.
- W4322759986 title "Ultra-low Precision Multiplication-free Training for Deep Neural Networks" @default.
- W4322759986 doi "https://doi.org/10.48550/arxiv.2302.14458" @default.
- W4322759986 hasPublicationYear "2023" @default.
- W4322759986 type Work @default.
- W4322759986 citedByCount "0" @default.
- W4322759986 crossrefType "posted-content" @default.
- W4322759986 hasAuthorship W4322759986A5008497740 @default.
- W4322759986 hasAuthorship W4322759986A5009782388 @default.
- W4322759986 hasAuthorship W4322759986A5022256556 @default.
- W4322759986 hasAuthorship W4322759986A5036205023 @default.
- W4322759986 hasAuthorship W4322759986A5052252896 @default.
- W4322759986 hasAuthorship W4322759986A5056949827 @default.
- W4322759986 hasAuthorship W4322759986A5065037360 @default.
- W4322759986 hasAuthorship W4322759986A5072861069 @default.
- W4322759986 hasBestOaLocation W43227599861 @default.
- W4322759986 hasConcept C105795698 @default.
- W4322759986 hasConcept C108583219 @default.
- W4322759986 hasConcept C113775141 @default.
- W4322759986 hasConcept C11413529 @default.
- W4322759986 hasConcept C114614502 @default.
- W4322759986 hasConcept C119599485 @default.
- W4322759986 hasConcept C127413603 @default.
- W4322759986 hasConcept C134765980 @default.
- W4322759986 hasConcept C138885662 @default.
- W4322759986 hasConcept C154945302 @default.
- W4322759986 hasConcept C164620267 @default.
- W4322759986 hasConcept C186370098 @default.
- W4322759986 hasConcept C199360897 @default.
- W4322759986 hasConcept C2742236 @default.
- W4322759986 hasConcept C2776848632 @default.
- W4322759986 hasConcept C2780165032 @default.
- W4322759986 hasConcept C2780595030 @default.
- W4322759986 hasConcept C28855332 @default.
- W4322759986 hasConcept C2984842247 @default.
- W4322759986 hasConcept C33923547 @default.
- W4322759986 hasConcept C41008148 @default.
- W4322759986 hasConcept C41895202 @default.
- W4322759986 hasConcept C50644808 @default.
- W4322759986 hasConcept C76155785 @default.
- W4322759986 hasConcept C82876162 @default.
- W4322759986 hasConceptScore W4322759986C105795698 @default.
- W4322759986 hasConceptScore W4322759986C108583219 @default.
- W4322759986 hasConceptScore W4322759986C113775141 @default.
- W4322759986 hasConceptScore W4322759986C11413529 @default.
- W4322759986 hasConceptScore W4322759986C114614502 @default.
- W4322759986 hasConceptScore W4322759986C119599485 @default.
- W4322759986 hasConceptScore W4322759986C127413603 @default.
- W4322759986 hasConceptScore W4322759986C134765980 @default.
- W4322759986 hasConceptScore W4322759986C138885662 @default.
- W4322759986 hasConceptScore W4322759986C154945302 @default.
- W4322759986 hasConceptScore W4322759986C164620267 @default.
- W4322759986 hasConceptScore W4322759986C186370098 @default.
- W4322759986 hasConceptScore W4322759986C199360897 @default.
- W4322759986 hasConceptScore W4322759986C2742236 @default.
- W4322759986 hasConceptScore W4322759986C2776848632 @default.
- W4322759986 hasConceptScore W4322759986C2780165032 @default.
- W4322759986 hasConceptScore W4322759986C2780595030 @default.
- W4322759986 hasConceptScore W4322759986C28855332 @default.
- W4322759986 hasConceptScore W4322759986C2984842247 @default.
- W4322759986 hasConceptScore W4322759986C33923547 @default.
- W4322759986 hasConceptScore W4322759986C41008148 @default.
- W4322759986 hasConceptScore W4322759986C41895202 @default.
- W4322759986 hasConceptScore W4322759986C50644808 @default.
- W4322759986 hasConceptScore W4322759986C76155785 @default.
- W4322759986 hasConceptScore W4322759986C82876162 @default.
- W4322759986 hasLocation W43227599861 @default.
- W4322759986 hasOpenAccess W4322759986 @default.
- W4322759986 hasPrimaryLocation W43227599861 @default.
- W4322759986 hasRelatedWork W2776343959 @default.
- W4322759986 hasRelatedWork W2786771851 @default.
- W4322759986 hasRelatedWork W2798544842 @default.
- W4322759986 hasRelatedWork W2971450774 @default.
- W4322759986 hasRelatedWork W3011969003 @default.
- W4322759986 hasRelatedWork W3102452718 @default.
- W4322759986 hasRelatedWork W3158216587 @default.
- W4322759986 hasRelatedWork W4200429237 @default.
- W4322759986 hasRelatedWork W4210961870 @default.
- W4322759986 hasRelatedWork W4308650821 @default.
- W4322759986 isParatext "false" @default.
- W4322759986 isRetracted "false" @default.
- W4322759986 workType "article" @default.