Matches in SemOpenAlex for { <https://semopenalex.org/work/W3048346032> ?p ?o ?g. }
Showing items 1 to 85 of
85
with 100 items per page.
- W3048346032 abstract "We observe three problems in existing storage and caching systems for deep-learning training (DLT) tasks: (1) accessing a dataset containing a large number of small files takes a long time, (2) global in-memory caching systems are vulnerable to node failures and slow to recover, and (3) repeatedly reading a dataset of files in shuffled orders is inefficient when the dataset is too large to be cached in memory. Therefore, we propose DIESEL, a dataset-based distributed storage and caching system for DLT tasks. Our approach is via a storage-caching system co-design. Firstly, since accessing small files is a metadata-intensive operation, DIESEL decouples the metadata processing from metadata storage, and introduces metadata snapshot mechanisms for each dataset. This approach speeds up metadata access significantly. Secondly, DIESEL deploys a task-grained distributed cache across the worker nodes of a DLT task. This way node failures are contained within each DLT task. Furthermore, the files are grouped into large chunks in storage, so the recovery time of the caching system is reduced greatly. Thirdly, DIESEL provides chunk-based shuffle so that the performance of random file access is improved without sacrificing training accuracy. Our experiments show that DIESEL achieves a linear speedup on metadata access, and outperforms an existing distributed caching system in both file caching and file reading. In real DLT tasks, DIESEL halves the data access time of an existing storage system, and reduces the training time by hours without changing any training code." @default.
- W3048346032 created "2020-08-13" @default.
- W3048346032 creator A5001421482 @default.
- W3048346032 creator A5033436625 @default.
- W3048346032 creator A5037488697 @default.
- W3048346032 creator A5044170622 @default.
- W3048346032 creator A5045798455 @default.
- W3048346032 creator A5049910854 @default.
- W3048346032 creator A5052095173 @default.
- W3048346032 date "2020-08-17" @default.
- W3048346032 modified "2023-10-06" @default.
- W3048346032 title "DIESEL: A Dataset-Based Distributed Storage and Caching System for Large-Scale Deep Learning Training" @default.
- W3048346032 cites W1981420413 @default.
- W3048346032 cites W2020765652 @default.
- W3048346032 cites W2027758281 @default.
- W3048346032 cites W2068739275 @default.
- W3048346032 cites W2117539524 @default.
- W3048346032 cites W2119738171 @default.
- W3048346032 cites W2194775991 @default.
- W3048346032 cites W2293862011 @default.
- W3048346032 cites W2768070465 @default.
- W3048346032 cites W2900042580 @default.
- W3048346032 cites W2902685930 @default.
- W3048346032 cites W4238584892 @default.
- W3048346032 doi "https://doi.org/10.1145/3404397.3404472" @default.
- W3048346032 hasPublicationYear "2020" @default.
- W3048346032 type Work @default.
- W3048346032 sameAs 3048346032 @default.
- W3048346032 citedByCount "15" @default.
- W3048346032 countsByYear W30483460322021 @default.
- W3048346032 countsByYear W30483460322022 @default.
- W3048346032 countsByYear W30483460322023 @default.
- W3048346032 crossrefType "proceedings-article" @default.
- W3048346032 hasAuthorship W3048346032A5001421482 @default.
- W3048346032 hasAuthorship W3048346032A5033436625 @default.
- W3048346032 hasAuthorship W3048346032A5037488697 @default.
- W3048346032 hasAuthorship W3048346032A5044170622 @default.
- W3048346032 hasAuthorship W3048346032A5045798455 @default.
- W3048346032 hasAuthorship W3048346032A5049910854 @default.
- W3048346032 hasAuthorship W3048346032A5052095173 @default.
- W3048346032 hasConcept C111919701 @default.
- W3048346032 hasConcept C115537543 @default.
- W3048346032 hasConcept C120314980 @default.
- W3048346032 hasConcept C162324750 @default.
- W3048346032 hasConcept C187736073 @default.
- W3048346032 hasConcept C194739806 @default.
- W3048346032 hasConcept C24885549 @default.
- W3048346032 hasConcept C2779489174 @default.
- W3048346032 hasConcept C2780451532 @default.
- W3048346032 hasConcept C2780940931 @default.
- W3048346032 hasConcept C41008148 @default.
- W3048346032 hasConcept C68339613 @default.
- W3048346032 hasConcept C77088390 @default.
- W3048346032 hasConcept C93518851 @default.
- W3048346032 hasConceptScore W3048346032C111919701 @default.
- W3048346032 hasConceptScore W3048346032C115537543 @default.
- W3048346032 hasConceptScore W3048346032C120314980 @default.
- W3048346032 hasConceptScore W3048346032C162324750 @default.
- W3048346032 hasConceptScore W3048346032C187736073 @default.
- W3048346032 hasConceptScore W3048346032C194739806 @default.
- W3048346032 hasConceptScore W3048346032C24885549 @default.
- W3048346032 hasConceptScore W3048346032C2779489174 @default.
- W3048346032 hasConceptScore W3048346032C2780451532 @default.
- W3048346032 hasConceptScore W3048346032C2780940931 @default.
- W3048346032 hasConceptScore W3048346032C41008148 @default.
- W3048346032 hasConceptScore W3048346032C68339613 @default.
- W3048346032 hasConceptScore W3048346032C77088390 @default.
- W3048346032 hasConceptScore W3048346032C93518851 @default.
- W3048346032 hasLocation W30483460321 @default.
- W3048346032 hasOpenAccess W3048346032 @default.
- W3048346032 hasPrimaryLocation W30483460321 @default.
- W3048346032 hasRelatedWork W1976306141 @default.
- W3048346032 hasRelatedWork W1978949740 @default.
- W3048346032 hasRelatedWork W1997671956 @default.
- W3048346032 hasRelatedWork W2348772280 @default.
- W3048346032 hasRelatedWork W2354642172 @default.
- W3048346032 hasRelatedWork W2365800529 @default.
- W3048346032 hasRelatedWork W2370313323 @default.
- W3048346032 hasRelatedWork W2991475544 @default.
- W3048346032 hasRelatedWork W4285137472 @default.
- W3048346032 hasRelatedWork W2551034554 @default.
- W3048346032 isParatext "false" @default.
- W3048346032 isRetracted "false" @default.
- W3048346032 magId "3048346032" @default.
- W3048346032 workType "article" @default.