Matches in SemOpenAlex for { <https://semopenalex.org/work/W3203325676> ?p ?o ?g. }
Showing items 1 to 83 of
83
with 100 items per page.
- W3203325676 abstract "An increasing number of big data applications in various domains generate datasets continuously, which must be processed for various purposes in a timely manner. As one of the most popular streaming data processing systems, Spark Streaming applies a batch-based mechanism, which receives real-time input data streams and divides the data into multiple batches before passing them to Spark processing engine. As such, inappropriate system configurations including batch interval and executor count may lead to unstable states, hence undermining the capability and efficiency of real-time computing. Hence, determining suitable configurations is crucial to the performance of such systems. Many machine learning- and search-based algorithms have been proposed to provide configuration recommendations for streaming applications where input data streams are fed at a constant speed, which, however, is extremely rare in practice. Most real-life streaming applications process data streams arriving at a time-varying rate and hence require real-time system monitoring and continuous configuration adjustment, which still remains largely unexplored. We propose a novel streaming optimization scheme based on Simultaneous Perturbation Stochastic Approximation (SPSA), referred to as NoStop, which dynamically tunes system configurations to optimize real-time system performance with negligible overhead and proved convergence. The performance superiority of NoStop is illustrated by real-life experiments in comparison with Bayesian Optimization and Spark Back Pressure solutions. Extensive experimental results show that NoStop is able to keep track of the changing pattern of input data in real time and provide optimal configuration settings to achieve the best system performance. This optimization scheme could also be applied to other streaming data processing engines with tunable parameters." @default.
- W3203325676 created "2021-10-11" @default.
- W3203325676 creator A5031393751 @default.
- W3203325676 creator A5034932607 @default.
- W3203325676 creator A5054307870 @default.
- W3203325676 date "2021-08-09" @default.
- W3203325676 modified "2023-09-24" @default.
- W3203325676 title "NoStop: A Novel Configuration Optimization Scheme for Spark Streaming" @default.
- W3203325676 cites W2009797711 @default.
- W3203325676 cites W2488300951 @default.
- W3203325676 cites W2542459869 @default.
- W3203325676 cites W2615179483 @default.
- W3203325676 cites W2807796354 @default.
- W3203325676 cites W3083262785 @default.
- W3203325676 doi "https://doi.org/10.1145/3472456.3472515" @default.
- W3203325676 hasPublicationYear "2021" @default.
- W3203325676 type Work @default.
- W3203325676 sameAs 3203325676 @default.
- W3203325676 citedByCount "1" @default.
- W3203325676 countsByYear W32033256762023 @default.
- W3203325676 crossrefType "proceedings-article" @default.
- W3203325676 hasAuthorship W3203325676A5031393751 @default.
- W3203325676 hasAuthorship W3203325676A5034932607 @default.
- W3203325676 hasAuthorship W3203325676A5054307870 @default.
- W3203325676 hasBestOaLocation W32033256761 @default.
- W3203325676 hasConcept C107027933 @default.
- W3203325676 hasConcept C111919701 @default.
- W3203325676 hasConcept C120314980 @default.
- W3203325676 hasConcept C124101348 @default.
- W3203325676 hasConcept C134306372 @default.
- W3203325676 hasConcept C154945302 @default.
- W3203325676 hasConcept C17744445 @default.
- W3203325676 hasConcept C180591056 @default.
- W3203325676 hasConcept C199360897 @default.
- W3203325676 hasConcept C199539241 @default.
- W3203325676 hasConcept C2777611316 @default.
- W3203325676 hasConcept C2778049539 @default.
- W3203325676 hasConcept C2779960059 @default.
- W3203325676 hasConcept C2781215313 @default.
- W3203325676 hasConcept C33923547 @default.
- W3203325676 hasConcept C41008148 @default.
- W3203325676 hasConcept C75684735 @default.
- W3203325676 hasConcept C77618280 @default.
- W3203325676 hasConcept C79403827 @default.
- W3203325676 hasConcept C89198739 @default.
- W3203325676 hasConceptScore W3203325676C107027933 @default.
- W3203325676 hasConceptScore W3203325676C111919701 @default.
- W3203325676 hasConceptScore W3203325676C120314980 @default.
- W3203325676 hasConceptScore W3203325676C124101348 @default.
- W3203325676 hasConceptScore W3203325676C134306372 @default.
- W3203325676 hasConceptScore W3203325676C154945302 @default.
- W3203325676 hasConceptScore W3203325676C17744445 @default.
- W3203325676 hasConceptScore W3203325676C180591056 @default.
- W3203325676 hasConceptScore W3203325676C199360897 @default.
- W3203325676 hasConceptScore W3203325676C199539241 @default.
- W3203325676 hasConceptScore W3203325676C2777611316 @default.
- W3203325676 hasConceptScore W3203325676C2778049539 @default.
- W3203325676 hasConceptScore W3203325676C2779960059 @default.
- W3203325676 hasConceptScore W3203325676C2781215313 @default.
- W3203325676 hasConceptScore W3203325676C33923547 @default.
- W3203325676 hasConceptScore W3203325676C41008148 @default.
- W3203325676 hasConceptScore W3203325676C75684735 @default.
- W3203325676 hasConceptScore W3203325676C77618280 @default.
- W3203325676 hasConceptScore W3203325676C79403827 @default.
- W3203325676 hasConceptScore W3203325676C89198739 @default.
- W3203325676 hasFunder F4320306076 @default.
- W3203325676 hasLocation W32033256761 @default.
- W3203325676 hasOpenAccess W3203325676 @default.
- W3203325676 hasPrimaryLocation W32033256761 @default.
- W3203325676 hasRelatedWork W1521353230 @default.
- W3203325676 hasRelatedWork W2038299727 @default.
- W3203325676 hasRelatedWork W2071213595 @default.
- W3203325676 hasRelatedWork W2351279671 @default.
- W3203325676 hasRelatedWork W2353072008 @default.
- W3203325676 hasRelatedWork W2373250155 @default.
- W3203325676 hasRelatedWork W2507920278 @default.
- W3203325676 hasRelatedWork W2756322059 @default.
- W3203325676 hasRelatedWork W3018461255 @default.
- W3203325676 hasRelatedWork W3100552002 @default.
- W3203325676 isParatext "false" @default.
- W3203325676 isRetracted "false" @default.
- W3203325676 magId "3203325676" @default.
- W3203325676 workType "article" @default.