Matches in SemOpenAlex for { <https://semopenalex.org/work/W4285504003> ?p ?o ?g. }
Showing items 1 to 70 of
70
with 100 items per page.
- W4285504003 abstract "In the last few years, the memory requirements to train state-of-the-art neural networks have far exceeded the DRAM capacities of modern hardware accelerators. This has necessitated the development of efficient algorithms to train these neural networks in parallel on large-scale GPU-based clusters. Since computation is relatively inexpensive on modern GPUs, designing and implementing extremely efficient communication in these parallel training algorithms is critical for extracting the maximum performance. This paper presents AxoNN, a parallel deep learning framework that exploits asynchrony and message-driven execution to schedule neural network operations on each GPU, thereby reducing GPU idle time and maximizing hardware efficiency. By using the CPU memory as a scratch space for offloading data periodically during training, AxoNN is able to reduce GPU memory consumption by four times. This allows us to increase the number of parameters per GPU by four times, thus reducing the amount of communication and increasing performance by over 13%. When tested against large transformer models with 12–100 billion parameters on 48–384 NVIDIA Tesla V100 GPUs, AxoNN achieves a per-GPU throughput of 49.4–54.78% of theoretical peak and reduces the training time by 22-37 days (15–25% speedup) as compared to the state-of-the-art." @default.
- W4285504003 created "2022-07-15" @default.
- W4285504003 creator A5012548250 @default.
- W4285504003 creator A5066077446 @default.
- W4285504003 date "2022-05-01" @default.
- W4285504003 modified "2023-09-23" @default.
- W4285504003 title "AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learning" @default.
- W4285504003 cites W1498436455 @default.
- W4285504003 cites W2969388332 @default.
- W4285504003 cites W3086105743 @default.
- W4285504003 cites W3129831491 @default.
- W4285504003 cites W3164436820 @default.
- W4285504003 cites W3175937250 @default.
- W4285504003 doi "https://doi.org/10.1109/ipdps53621.2022.00065" @default.
- W4285504003 hasPublicationYear "2022" @default.
- W4285504003 type Work @default.
- W4285504003 citedByCount "3" @default.
- W4285504003 countsByYear W42855040032023 @default.
- W4285504003 crossrefType "proceedings-article" @default.
- W4285504003 hasAuthorship W4285504003A5012548250 @default.
- W4285504003 hasAuthorship W4285504003A5066077446 @default.
- W4285504003 hasBestOaLocation W42855040032 @default.
- W4285504003 hasConcept C108583219 @default.
- W4285504003 hasConcept C111919701 @default.
- W4285504003 hasConcept C11413529 @default.
- W4285504003 hasConcept C118524514 @default.
- W4285504003 hasConcept C151319957 @default.
- W4285504003 hasConcept C154945302 @default.
- W4285504003 hasConcept C173608175 @default.
- W4285504003 hasConcept C2781357197 @default.
- W4285504003 hasConcept C31258907 @default.
- W4285504003 hasConcept C41008148 @default.
- W4285504003 hasConcept C45374587 @default.
- W4285504003 hasConcept C50644808 @default.
- W4285504003 hasConcept C68339613 @default.
- W4285504003 hasConcept C68387754 @default.
- W4285504003 hasConcept C83283714 @default.
- W4285504003 hasConceptScore W4285504003C108583219 @default.
- W4285504003 hasConceptScore W4285504003C111919701 @default.
- W4285504003 hasConceptScore W4285504003C11413529 @default.
- W4285504003 hasConceptScore W4285504003C118524514 @default.
- W4285504003 hasConceptScore W4285504003C151319957 @default.
- W4285504003 hasConceptScore W4285504003C154945302 @default.
- W4285504003 hasConceptScore W4285504003C173608175 @default.
- W4285504003 hasConceptScore W4285504003C2781357197 @default.
- W4285504003 hasConceptScore W4285504003C31258907 @default.
- W4285504003 hasConceptScore W4285504003C41008148 @default.
- W4285504003 hasConceptScore W4285504003C45374587 @default.
- W4285504003 hasConceptScore W4285504003C50644808 @default.
- W4285504003 hasConceptScore W4285504003C68339613 @default.
- W4285504003 hasConceptScore W4285504003C68387754 @default.
- W4285504003 hasConceptScore W4285504003C83283714 @default.
- W4285504003 hasFunder F4320306084 @default.
- W4285504003 hasLocation W42855040031 @default.
- W4285504003 hasLocation W42855040032 @default.
- W4285504003 hasOpenAccess W4285504003 @default.
- W4285504003 hasPrimaryLocation W42855040031 @default.
- W4285504003 hasRelatedWork W1509211761 @default.
- W4285504003 hasRelatedWork W156843270 @default.
- W4285504003 hasRelatedWork W1588481459 @default.
- W4285504003 hasRelatedWork W1905659066 @default.
- W4285504003 hasRelatedWork W2007449167 @default.
- W4285504003 hasRelatedWork W2036853476 @default.
- W4285504003 hasRelatedWork W2119821807 @default.
- W4285504003 hasRelatedWork W2154055002 @default.
- W4285504003 hasRelatedWork W2391299576 @default.
- W4285504003 hasRelatedWork W2185731423 @default.
- W4285504003 isParatext "false" @default.
- W4285504003 isRetracted "false" @default.
- W4285504003 workType "article" @default.