Matches in SemOpenAlex for { <https://semopenalex.org/work/W3048345275> ?p ?o ?g. }
Showing items 1 to 92 of
92
with 100 items per page.
- W3048345275 abstract "In recent years, distributed computing frameworks, such as Hadoop MapReduce and Spark, are widely used for big data processing. With the explosive growth of the amount of data, companies tend to store intermediate data of the shuffle phase on disk instead of memory. Therefore, intensive network and disk I/O are both involved in the shuffle phase. To optimize the overhead of the shuffle phase, we propose OPS, an open-source distributed computing shuffle management system based on Spark, which provides an independent shuffle service for Spark. By using early-merge and early-shuffle strategy, OPS alleviates the I/O overhead in the shuffle phase and efficiently schedules the I/O and computing resources. OPS also proposes a slot-based scheduling algorithm to predict and calculate the optimal scheduling result of the reduce task. Besides, OPS provides a taint-redo strategy to ensure the fault tolerance of computing jobs. We evaluated the performance of OPS on a 100-node Amazon AWS EC2 cluster. Overall, OPS optimizes the overhead of shuffle by nearly 50%. In the test cases of HiBench, OPS improves end-to-end completion time by nearly 30% on average." @default.
- W3048345275 created "2020-08-13" @default.
- W3048345275 creator A5003318473 @default.
- W3048345275 creator A5011323970 @default.
- W3048345275 creator A5022420830 @default.
- W3048345275 creator A5032453255 @default.
- W3048345275 creator A5052741300 @default.
- W3048345275 creator A5053323823 @default.
- W3048345275 creator A5072309548 @default.
- W3048345275 date "2020-08-17" @default.
- W3048345275 modified "2023-09-25" @default.
- W3048345275 title "OPS: Optimized Shuffle Management System for Apache Spark" @default.
- W3048345275 cites W1963832798 @default.
- W3048345275 cites W2045271686 @default.
- W3048345275 cites W2067051372 @default.
- W3048345275 cites W2076164405 @default.
- W3048345275 cites W2080131844 @default.
- W3048345275 cites W2088687427 @default.
- W3048345275 cites W2104680817 @default.
- W3048345275 cites W2117702591 @default.
- W3048345275 cites W2118566694 @default.
- W3048345275 cites W2118676041 @default.
- W3048345275 cites W2132353061 @default.
- W3048345275 cites W2140509629 @default.
- W3048345275 cites W2163404313 @default.
- W3048345275 cites W2173213060 @default.
- W3048345275 cites W2313564095 @default.
- W3048345275 cites W2465838301 @default.
- W3048345275 cites W2497069692 @default.
- W3048345275 cites W2787779818 @default.
- W3048345275 cites W2798457757 @default.
- W3048345275 cites W4239666938 @default.
- W3048345275 doi "https://doi.org/10.1145/3404397.3404430" @default.
- W3048345275 hasPublicationYear "2020" @default.
- W3048345275 type Work @default.
- W3048345275 sameAs 3048345275 @default.
- W3048345275 citedByCount "4" @default.
- W3048345275 countsByYear W30483452752022 @default.
- W3048345275 countsByYear W30483452752023 @default.
- W3048345275 crossrefType "proceedings-article" @default.
- W3048345275 hasAuthorship W3048345275A5003318473 @default.
- W3048345275 hasAuthorship W3048345275A5011323970 @default.
- W3048345275 hasAuthorship W3048345275A5022420830 @default.
- W3048345275 hasAuthorship W3048345275A5032453255 @default.
- W3048345275 hasAuthorship W3048345275A5052741300 @default.
- W3048345275 hasAuthorship W3048345275A5053323823 @default.
- W3048345275 hasAuthorship W3048345275A5072309548 @default.
- W3048345275 hasConcept C111919701 @default.
- W3048345275 hasConcept C120314980 @default.
- W3048345275 hasConcept C162324750 @default.
- W3048345275 hasConcept C173608175 @default.
- W3048345275 hasConcept C197129107 @default.
- W3048345275 hasConcept C199360897 @default.
- W3048345275 hasConcept C206729178 @default.
- W3048345275 hasConcept C21547014 @default.
- W3048345275 hasConcept C2779960059 @default.
- W3048345275 hasConcept C2781215313 @default.
- W3048345275 hasConcept C29140674 @default.
- W3048345275 hasConcept C41008148 @default.
- W3048345275 hasConcept C63540848 @default.
- W3048345275 hasConcept C75684735 @default.
- W3048345275 hasConceptScore W3048345275C111919701 @default.
- W3048345275 hasConceptScore W3048345275C120314980 @default.
- W3048345275 hasConceptScore W3048345275C162324750 @default.
- W3048345275 hasConceptScore W3048345275C173608175 @default.
- W3048345275 hasConceptScore W3048345275C197129107 @default.
- W3048345275 hasConceptScore W3048345275C199360897 @default.
- W3048345275 hasConceptScore W3048345275C206729178 @default.
- W3048345275 hasConceptScore W3048345275C21547014 @default.
- W3048345275 hasConceptScore W3048345275C2779960059 @default.
- W3048345275 hasConceptScore W3048345275C2781215313 @default.
- W3048345275 hasConceptScore W3048345275C29140674 @default.
- W3048345275 hasConceptScore W3048345275C41008148 @default.
- W3048345275 hasConceptScore W3048345275C63540848 @default.
- W3048345275 hasConceptScore W3048345275C75684735 @default.
- W3048345275 hasLocation W30483452751 @default.
- W3048345275 hasOpenAccess W3048345275 @default.
- W3048345275 hasPrimaryLocation W30483452751 @default.
- W3048345275 hasRelatedWork W1882733036 @default.
- W3048345275 hasRelatedWork W1992741870 @default.
- W3048345275 hasRelatedWork W2092071486 @default.
- W3048345275 hasRelatedWork W2160425906 @default.
- W3048345275 hasRelatedWork W2285867394 @default.
- W3048345275 hasRelatedWork W2354144074 @default.
- W3048345275 hasRelatedWork W2391167130 @default.
- W3048345275 hasRelatedWork W2546696010 @default.
- W3048345275 hasRelatedWork W3102446781 @default.
- W3048345275 hasRelatedWork W4283067488 @default.
- W3048345275 isParatext "false" @default.
- W3048345275 isRetracted "false" @default.
- W3048345275 magId "3048345275" @default.
- W3048345275 workType "article" @default.