Matches in SemOpenAlex for { <https://semopenalex.org/work/W3206827202> ?p ?o ?g. }
- W3206827202 abstract "In decentralized machine learning, workers compute model updates on their local data. Because the workers only communicate with few neighbors without central coordination, these updates propagate progressively over the network. This paradigm enables distributed training on networks without all-to-all connectivity, helping to protect data privacy as well as to reduce the communication cost of distributed training in data centers. A key challenge, primarily in decentralized deep learning, remains the handling of differences between the workers' local data distributions. To tackle this challenge, we introduce the RelaySum mechanism for information propagation in decentralized learning. RelaySum uses spanning trees to distribute information exactly uniformly across all workers with finite delays depending on the distance between nodes. In contrast, the typical gossip averaging mechanism only distributes data uniformly asymptotically while using the same communication volume per step as RelaySum. We prove that RelaySGD, based on this mechanism, is independent of data heterogeneity and scales to many workers, enabling highly accurate decentralized deep learning on heterogeneous data. Our code is available at this http URL" @default.
- W3206827202 created "2021-10-25" @default.
- W3206827202 creator A5020200055 @default.
- W3206827202 creator A5026569995 @default.
- W3206827202 creator A5045519627 @default.
- W3206827202 creator A5050008056 @default.
- W3206827202 creator A5065163339 @default.
- W3206827202 creator A5073756389 @default.
- W3206827202 creator A5089153368 @default.
- W3206827202 date "2021-10-08" @default.
- W3206827202 modified "2023-09-26" @default.
- W3206827202 title "RelaySum for Decentralized Deep Learning on Heterogeneous Data" @default.
- W3206827202 cites W1836465849 @default.
- W3206827202 cites W1909326920 @default.
- W3206827202 cites W1990917528 @default.
- W3206827202 cites W1993869096 @default.
- W3206827202 cites W2044212084 @default.
- W3206827202 cites W2049469158 @default.
- W3206827202 cites W2049659086 @default.
- W3206827202 cites W2063403497 @default.
- W3206827202 cites W2074796812 @default.
- W3206827202 cites W2093777540 @default.
- W3206827202 cites W2108598243 @default.
- W3206827202 cites W2110100895 @default.
- W3206827202 cites W2132022337 @default.
- W3206827202 cites W2135920560 @default.
- W3206827202 cites W2160054705 @default.
- W3206827202 cites W2169167445 @default.
- W3206827202 cites W2266148951 @default.
- W3206827202 cites W2541884796 @default.
- W3206827202 cites W2590635385 @default.
- W3206827202 cites W2789243870 @default.
- W3206827202 cites W2902410171 @default.
- W3206827202 cites W2946573451 @default.
- W3206827202 cites W2958992393 @default.
- W3206827202 cites W2963156126 @default.
- W3206827202 cites W2963228337 @default.
- W3206827202 cites W2963649943 @default.
- W3206827202 cites W2964103300 @default.
- W3206827202 cites W2964121744 @default.
- W3206827202 cites W2964165783 @default.
- W3206827202 cites W2973303672 @default.
- W3206827202 cites W2978017171 @default.
- W3206827202 cites W2981831462 @default.
- W3206827202 cites W2995022099 @default.
- W3206827202 cites W3006548771 @default.
- W3206827202 cites W3022064003 @default.
- W3206827202 cites W3034741284 @default.
- W3206827202 cites W3037932933 @default.
- W3206827202 cites W3099715789 @default.
- W3206827202 cites W3104016060 @default.
- W3206827202 cites W3105632507 @default.
- W3206827202 cites W3118608800 @default.
- W3206827202 cites W3127286811 @default.
- W3206827202 cites W3166004119 @default.
- W3206827202 cites W3203694565 @default.
- W3206827202 hasPublicationYear "2021" @default.
- W3206827202 type Work @default.
- W3206827202 sameAs 3206827202 @default.
- W3206827202 citedByCount "1" @default.
- W3206827202 countsByYear W32068272022021 @default.
- W3206827202 crossrefType "posted-content" @default.
- W3206827202 hasAuthorship W3206827202A5020200055 @default.
- W3206827202 hasAuthorship W3206827202A5026569995 @default.
- W3206827202 hasAuthorship W3206827202A5045519627 @default.
- W3206827202 hasAuthorship W3206827202A5050008056 @default.
- W3206827202 hasAuthorship W3206827202A5065163339 @default.
- W3206827202 hasAuthorship W3206827202A5073756389 @default.
- W3206827202 hasAuthorship W3206827202A5089153368 @default.
- W3206827202 hasConcept C111472728 @default.
- W3206827202 hasConcept C120314980 @default.
- W3206827202 hasConcept C138885662 @default.
- W3206827202 hasConcept C154945302 @default.
- W3206827202 hasConcept C15744967 @default.
- W3206827202 hasConcept C177264268 @default.
- W3206827202 hasConcept C19417346 @default.
- W3206827202 hasConcept C199360897 @default.
- W3206827202 hasConcept C26517878 @default.
- W3206827202 hasConcept C2776760102 @default.
- W3206827202 hasConcept C2779582901 @default.
- W3206827202 hasConcept C38652104 @default.
- W3206827202 hasConcept C41008148 @default.
- W3206827202 hasConcept C62989814 @default.
- W3206827202 hasConcept C77805123 @default.
- W3206827202 hasConcept C80444323 @default.
- W3206827202 hasConcept C89611455 @default.
- W3206827202 hasConceptScore W3206827202C111472728 @default.
- W3206827202 hasConceptScore W3206827202C120314980 @default.
- W3206827202 hasConceptScore W3206827202C138885662 @default.
- W3206827202 hasConceptScore W3206827202C154945302 @default.
- W3206827202 hasConceptScore W3206827202C15744967 @default.
- W3206827202 hasConceptScore W3206827202C177264268 @default.
- W3206827202 hasConceptScore W3206827202C19417346 @default.
- W3206827202 hasConceptScore W3206827202C199360897 @default.
- W3206827202 hasConceptScore W3206827202C26517878 @default.
- W3206827202 hasConceptScore W3206827202C2776760102 @default.
- W3206827202 hasConceptScore W3206827202C2779582901 @default.
- W3206827202 hasConceptScore W3206827202C38652104 @default.
- W3206827202 hasConceptScore W3206827202C41008148 @default.
- W3206827202 hasConceptScore W3206827202C62989814 @default.