FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W4321487996> ?p ?o ?g. }

• W4321487996 endingPage "963" @default.
• W4321487996 startingPage "941" @default.
• W4321487996 abstract "Distributed data-parallel training (DDP) is prevalent in large-scale deep learning. To increase the training throughput and scalability, high-performance collective communication methods such as AllReduce have recently proliferated for DDP use. However, these approaches require long communication periods with increasing model sizes. Collective communication transmits many sparse gradient values that can be efficiently compressed to reduce the required training time. State-of-the-art compression approaches do not provide mergeable compression for AllReduce and lack convergence bounds. We present a sparse sketch reducer (S2Reducer), a sparsity-preserving sketch-based collective communication method. S2Reducer preserves gradient sparsity and reduces communication costs via a bitmap informed count sketch structure and adapts to efficient AllReduce operators. We tune the count sketch organization to minimize the hash conflicts in a fixed-size budget. We prove that our method has the same convergence rate as vanilla data-parallel training and a much smaller communication overhead than those of state-of-the-art methods. We implement a GPU-accelerated S2Reducer for the Ring AllReduce-based DDP system. We perform extensive evaluations against four state-of-the-art methods across seven deep learning models. Our results show that S2Reducer converges to the same accuracy as that of state-of-the-art approaches while reducing the sparse communication overhead by up to 86% and achieving a speedup of up to <inline-formula xmlns:mml=http://www.w3.org/1998/Math/MathML xmlns:xlink=http://www.w3.org/1999/xlink> <tex-math notation=LaTeX>$3.5times $ </tex-math></inline-formula> in distributed training." @default.
• W4321487996 created "2023-02-23" @default.
• W4321487996 creator A5003730869 @default.
• W4321487996 creator A5012979904 @default.
• W4321487996 creator A5019549276 @default.
• W4321487996 creator A5055426388 @default.
• W4321487996 creator A5074708780 @default.
• W4321487996 creator A5081308434 @default.
• W4321487996 date "2023-04-01" @default.
• W4321487996 modified "2023-10-14" @default.
• W4321487996 title "Compressed Collective Sparse-Sketch for Distributed Data-Parallel Training of Deep Learning Models" @default.
• W4321487996 cites W1972525637 @default.
• W4321487996 cites W2039548408 @default.
• W4321487996 cites W2057332538 @default.
• W4321487996 cites W2060393849 @default.
• W4321487996 cites W2194775991 @default.
• W4321487996 cites W2294895103 @default.
• W4321487996 cites W2405578611 @default.
• W4321487996 cites W2512924740 @default.
• W4321487996 cites W2606722458 @default.
• W4321487996 cites W2745269232 @default.
• W4321487996 cites W2906007643 @default.
• W4321487996 cites W2935041335 @default.
• W4321487996 cites W2963446712 @default.
• W4321487996 cites W2964110616 @default.
• W4321487996 cites W2966527647 @default.
• W4321487996 cites W2975712713 @default.
• W4321487996 cites W2985108934 @default.
• W4321487996 cites W3081168214 @default.
• W4321487996 cites W3091097978 @default.
• W4321487996 cites W3097777922 @default.
• W4321487996 cites W3132107458 @default.
• W4321487996 cites W3160525311 @default.
• W4321487996 cites W3173523152 @default.
• W4321487996 cites W3204434815 @default.
• W4321487996 cites W3206636350 @default.
• W4321487996 cites W4294106961 @default.
• W4321487996 doi "https://doi.org/10.1109/jsac.2023.3242733" @default.
• W4321487996 hasPublicationYear "2023" @default.
• W4321487996 type Work @default.
• W4321487996 citedByCount "0" @default.
• W4321487996 crossrefType "journal-article" @default.
• W4321487996 hasAuthorship W4321487996A5003730869 @default.
• W4321487996 hasAuthorship W4321487996A5012979904 @default.
• W4321487996 hasAuthorship W4321487996A5019549276 @default.
• W4321487996 hasAuthorship W4321487996A5055426388 @default.
• W4321487996 hasAuthorship W4321487996A5074708780 @default.
• W4321487996 hasAuthorship W4321487996A5081308434 @default.
• W4321487996 hasConcept C108583219 @default.
• W4321487996 hasConcept C111919701 @default.
• W4321487996 hasConcept C11413529 @default.
• W4321487996 hasConcept C154945302 @default.
• W4321487996 hasConcept C157764524 @default.
• W4321487996 hasConcept C162324750 @default.
• W4321487996 hasConcept C173608175 @default.
• W4321487996 hasConcept C2777303404 @default.
• W4321487996 hasConcept C2779231336 @default.
• W4321487996 hasConcept C2779960059 @default.
• W4321487996 hasConcept C38652104 @default.
• W4321487996 hasConcept C41008148 @default.
• W4321487996 hasConcept C48044578 @default.
• W4321487996 hasConcept C50522688 @default.
• W4321487996 hasConcept C555944384 @default.
• W4321487996 hasConcept C68339613 @default.
• W4321487996 hasConcept C76155785 @default.
• W4321487996 hasConcept C77088390 @default.
• W4321487996 hasConcept C80444323 @default.
• W4321487996 hasConcept C99138194 @default.
• W4321487996 hasConceptScore W4321487996C108583219 @default.
• W4321487996 hasConceptScore W4321487996C111919701 @default.
• W4321487996 hasConceptScore W4321487996C11413529 @default.
• W4321487996 hasConceptScore W4321487996C154945302 @default.
• W4321487996 hasConceptScore W4321487996C157764524 @default.
• W4321487996 hasConceptScore W4321487996C162324750 @default.
• W4321487996 hasConceptScore W4321487996C173608175 @default.
• W4321487996 hasConceptScore W4321487996C2777303404 @default.
• W4321487996 hasConceptScore W4321487996C2779231336 @default.
• W4321487996 hasConceptScore W4321487996C2779960059 @default.
• W4321487996 hasConceptScore W4321487996C38652104 @default.
• W4321487996 hasConceptScore W4321487996C41008148 @default.
• W4321487996 hasConceptScore W4321487996C48044578 @default.
• W4321487996 hasConceptScore W4321487996C50522688 @default.
• W4321487996 hasConceptScore W4321487996C555944384 @default.
• W4321487996 hasConceptScore W4321487996C68339613 @default.
• W4321487996 hasConceptScore W4321487996C76155785 @default.
• W4321487996 hasConceptScore W4321487996C77088390 @default.
• W4321487996 hasConceptScore W4321487996C80444323 @default.
• W4321487996 hasConceptScore W4321487996C99138194 @default.
• W4321487996 hasFunder F4320321001 @default.
• W4321487996 hasFunder F4320335777 @default.
• W4321487996 hasIssue "4" @default.
• W4321487996 hasLocation W43214879961 @default.
• W4321487996 hasOpenAccess W4321487996 @default.
• W4321487996 hasPrimaryLocation W43214879961 @default.
• W4321487996 hasRelatedWork W1595151633 @default.
• W4321487996 hasRelatedWork W1800827217 @default.
• W4321487996 hasRelatedWork W1989119024 @default.
• W4321487996 hasRelatedWork W2059256796 @default.

• next