Matches in SemOpenAlex for { <https://semopenalex.org/work/W4281482092> ?p ?o ?g. }
Showing items 1 to 83 of
83
with 100 items per page.
- W4281482092 abstract "High-concurrency asynchronous training upon parameter server (PS) architecture and high-performance synchronous training upon all-reduce (AR) architecture are the most commonly deployed distributed training modes for recommendation models. Although synchronous AR training is designed to have higher training efficiency, asynchronous PS training would be a better choice for training speed when there are stragglers (slow workers) in the shared cluster, especially under limited computing resources. An ideal way to take full advantage of these two training modes is to switch between them upon the cluster status. However, switching training modes often requires tuning hyper-parameters, which is extremely time- and resource-consuming. We find two obstacles to a tuning-free approach: the different distribution of the gradient values and the stale gradients from the stragglers. This paper proposes Global Batch gradients Aggregation (GBA) over PS, which aggregates and applies gradients with the same global batch size as the synchronous training. A token-control process is implemented to assemble the gradients and decay the gradients with severe staleness. We provide the convergence analysis to reveal that GBA has comparable convergence properties with the synchronous training, and demonstrate the robustness of GBA the recommendation models against the gradient staleness. Experiments on three industrial-scale recommendation tasks show that GBA is an effective tuning-free approach for switching. Compared to the state-of-the-art derived asynchronous training, GBA achieves up to 0.2% improvement on the AUC metric, which is significant for the recommendation models. Meanwhile, under the strained hardware resource, GBA speeds up at least 2.4x compared to synchronous training." @default.
- W4281482092 created "2022-05-26" @default.
- W4281482092 creator A5004319733 @default.
- W4281482092 creator A5012610862 @default.
- W4281482092 creator A5020022791 @default.
- W4281482092 creator A5038906848 @default.
- W4281482092 creator A5041272869 @default.
- W4281482092 creator A5048379858 @default.
- W4281482092 creator A5051698439 @default.
- W4281482092 creator A5054263974 @default.
- W4281482092 creator A5056693481 @default.
- W4281482092 creator A5059884027 @default.
- W4281482092 creator A5080668659 @default.
- W4281482092 creator A5088917149 @default.
- W4281482092 date "2022-05-23" @default.
- W4281482092 modified "2023-10-16" @default.
- W4281482092 title "GBA: A Tuning-free Approach to Switch between Synchronous and Asynchronous Training for Recommendation Model" @default.
- W4281482092 doi "https://doi.org/10.48550/arxiv.2205.11048" @default.
- W4281482092 hasPublicationYear "2022" @default.
- W4281482092 type Work @default.
- W4281482092 citedByCount "0" @default.
- W4281482092 crossrefType "posted-content" @default.
- W4281482092 hasAuthorship W4281482092A5004319733 @default.
- W4281482092 hasAuthorship W4281482092A5012610862 @default.
- W4281482092 hasAuthorship W4281482092A5020022791 @default.
- W4281482092 hasAuthorship W4281482092A5038906848 @default.
- W4281482092 hasAuthorship W4281482092A5041272869 @default.
- W4281482092 hasAuthorship W4281482092A5048379858 @default.
- W4281482092 hasAuthorship W4281482092A5051698439 @default.
- W4281482092 hasAuthorship W4281482092A5054263974 @default.
- W4281482092 hasAuthorship W4281482092A5056693481 @default.
- W4281482092 hasAuthorship W4281482092A5059884027 @default.
- W4281482092 hasAuthorship W4281482092A5080668659 @default.
- W4281482092 hasAuthorship W4281482092A5088917149 @default.
- W4281482092 hasBestOaLocation W42814820921 @default.
- W4281482092 hasConcept C104317684 @default.
- W4281482092 hasConcept C120314980 @default.
- W4281482092 hasConcept C121332964 @default.
- W4281482092 hasConcept C151319957 @default.
- W4281482092 hasConcept C153294291 @default.
- W4281482092 hasConcept C162324750 @default.
- W4281482092 hasConcept C185592680 @default.
- W4281482092 hasConcept C193702766 @default.
- W4281482092 hasConcept C206345919 @default.
- W4281482092 hasConcept C2777211547 @default.
- W4281482092 hasConcept C2777303404 @default.
- W4281482092 hasConcept C31258907 @default.
- W4281482092 hasConcept C41008148 @default.
- W4281482092 hasConcept C50522688 @default.
- W4281482092 hasConcept C55493867 @default.
- W4281482092 hasConcept C63479239 @default.
- W4281482092 hasConceptScore W4281482092C104317684 @default.
- W4281482092 hasConceptScore W4281482092C120314980 @default.
- W4281482092 hasConceptScore W4281482092C121332964 @default.
- W4281482092 hasConceptScore W4281482092C151319957 @default.
- W4281482092 hasConceptScore W4281482092C153294291 @default.
- W4281482092 hasConceptScore W4281482092C162324750 @default.
- W4281482092 hasConceptScore W4281482092C185592680 @default.
- W4281482092 hasConceptScore W4281482092C193702766 @default.
- W4281482092 hasConceptScore W4281482092C206345919 @default.
- W4281482092 hasConceptScore W4281482092C2777211547 @default.
- W4281482092 hasConceptScore W4281482092C2777303404 @default.
- W4281482092 hasConceptScore W4281482092C31258907 @default.
- W4281482092 hasConceptScore W4281482092C41008148 @default.
- W4281482092 hasConceptScore W4281482092C50522688 @default.
- W4281482092 hasConceptScore W4281482092C55493867 @default.
- W4281482092 hasConceptScore W4281482092C63479239 @default.
- W4281482092 hasLocation W42814820921 @default.
- W4281482092 hasOpenAccess W4281482092 @default.
- W4281482092 hasPrimaryLocation W42814820921 @default.
- W4281482092 hasRelatedWork W1561560534 @default.
- W4281482092 hasRelatedWork W1748496945 @default.
- W4281482092 hasRelatedWork W2071774063 @default.
- W4281482092 hasRelatedWork W2110053516 @default.
- W4281482092 hasRelatedWork W2250656952 @default.
- W4281482092 hasRelatedWork W2313989154 @default.
- W4281482092 hasRelatedWork W2364921833 @default.
- W4281482092 hasRelatedWork W2748403830 @default.
- W4281482092 hasRelatedWork W2899486387 @default.
- W4281482092 hasRelatedWork W90634539 @default.
- W4281482092 isParatext "false" @default.
- W4281482092 isRetracted "false" @default.
- W4281482092 workType "article" @default.