Matches in SemOpenAlex for { <https://semopenalex.org/work/W3009370840> ?p ?o ?g. }
Showing items 1 to 83 of
83
with 100 items per page.
- W3009370840 abstract "The Apache Spark cluster computing platform is being increasingly used to develop big data analytics applications. There are many scenarios that require quick estimates of the execution time of any given Spark application. For example, users and operators of a Spark cluster often require quick insights on how the execution time of an application is likely to be impacted by the resources allocated to the application, e.g., the number of Spark executor cores assigned, and the size of the data to be processed. Job schedulers can benefit from fast estimates at runtime that would allow them to quickly conFigure a Spark application for a desired execution time using the least amount of resources. While others have developed models to predict the execution time of Spark applications, such models typically require extensive prior executions of applications under various resource allocation settings and data sizes. Consequently, these techniques are not suited for situations where quick predictions are required and very little cluster resources are available for the experimentation needed to build a model. This paper proposes an alternative approach called PERIDOT that addresses this limitation. The approach involves executing a given application under a fixed resource allocation setting with two different-sized, small subsets of its input data. It analyzes logs from these two executions to estimate the dependencies between internal stages in the application. Information on these dependencies combined with knowledge of Spark's data partitioning mechanisms is used to derive an analytic model that can predict execution times for other resource allocation settings and input data sizes. We show that deriving a model using just these two reference executions allows PERIDOT to accurately predict the performance of a variety of Spark applications spanning text analytics, linear algebra, machine learning and Spark SQL. In contrast, we show that a state-of-the-art machine learning based execution time prediction algorithm performs poorly when presented with such limited training data." @default.
- W3009370840 created "2020-03-13" @default.
- W3009370840 creator A5001117576 @default.
- W3009370840 creator A5020669873 @default.
- W3009370840 creator A5049988473 @default.
- W3009370840 creator A5086312071 @default.
- W3009370840 date "2019-10-01" @default.
- W3009370840 modified "2023-10-16" @default.
- W3009370840 title "Quick Execution Time Predictions for Spark Applications" @default.
- W3009370840 cites W2150871235 @default.
- W3009370840 cites W2175075972 @default.
- W3009370840 cites W2570373436 @default.
- W3009370840 cites W2573911191 @default.
- W3009370840 cites W2582789036 @default.
- W3009370840 cites W2770935003 @default.
- W3009370840 cites W2796039710 @default.
- W3009370840 cites W2809559041 @default.
- W3009370840 cites W2970105940 @default.
- W3009370840 doi "https://doi.org/10.23919/cnsm46954.2019.9012752" @default.
- W3009370840 hasPublicationYear "2019" @default.
- W3009370840 type Work @default.
- W3009370840 sameAs 3009370840 @default.
- W3009370840 citedByCount "16" @default.
- W3009370840 countsByYear W30093708402020 @default.
- W3009370840 countsByYear W30093708402021 @default.
- W3009370840 countsByYear W30093708402022 @default.
- W3009370840 countsByYear W30093708402023 @default.
- W3009370840 crossrefType "proceedings-article" @default.
- W3009370840 hasAuthorship W3009370840A5001117576 @default.
- W3009370840 hasAuthorship W3009370840A5020669873 @default.
- W3009370840 hasAuthorship W3009370840A5049988473 @default.
- W3009370840 hasAuthorship W3009370840A5086312071 @default.
- W3009370840 hasConcept C111919701 @default.
- W3009370840 hasConcept C120314980 @default.
- W3009370840 hasConcept C124101348 @default.
- W3009370840 hasConcept C164866538 @default.
- W3009370840 hasConcept C17744445 @default.
- W3009370840 hasConcept C180591056 @default.
- W3009370840 hasConcept C199360897 @default.
- W3009370840 hasConcept C199539241 @default.
- W3009370840 hasConcept C206345919 @default.
- W3009370840 hasConcept C2781215313 @default.
- W3009370840 hasConcept C29140674 @default.
- W3009370840 hasConcept C2989134064 @default.
- W3009370840 hasConcept C31258907 @default.
- W3009370840 hasConcept C41008148 @default.
- W3009370840 hasConcept C75684735 @default.
- W3009370840 hasConcept C77088390 @default.
- W3009370840 hasConcept C79158427 @default.
- W3009370840 hasConceptScore W3009370840C111919701 @default.
- W3009370840 hasConceptScore W3009370840C120314980 @default.
- W3009370840 hasConceptScore W3009370840C124101348 @default.
- W3009370840 hasConceptScore W3009370840C164866538 @default.
- W3009370840 hasConceptScore W3009370840C17744445 @default.
- W3009370840 hasConceptScore W3009370840C180591056 @default.
- W3009370840 hasConceptScore W3009370840C199360897 @default.
- W3009370840 hasConceptScore W3009370840C199539241 @default.
- W3009370840 hasConceptScore W3009370840C206345919 @default.
- W3009370840 hasConceptScore W3009370840C2781215313 @default.
- W3009370840 hasConceptScore W3009370840C29140674 @default.
- W3009370840 hasConceptScore W3009370840C2989134064 @default.
- W3009370840 hasConceptScore W3009370840C31258907 @default.
- W3009370840 hasConceptScore W3009370840C41008148 @default.
- W3009370840 hasConceptScore W3009370840C75684735 @default.
- W3009370840 hasConceptScore W3009370840C77088390 @default.
- W3009370840 hasConceptScore W3009370840C79158427 @default.
- W3009370840 hasLocation W30093708401 @default.
- W3009370840 hasOpenAccess W3009370840 @default.
- W3009370840 hasPrimaryLocation W30093708401 @default.
- W3009370840 hasRelatedWork W2603352950 @default.
- W3009370840 hasRelatedWork W2793083950 @default.
- W3009370840 hasRelatedWork W3112375919 @default.
- W3009370840 hasRelatedWork W3191926225 @default.
- W3009370840 hasRelatedWork W3195722949 @default.
- W3009370840 hasRelatedWork W4233648438 @default.
- W3009370840 hasRelatedWork W4241425178 @default.
- W3009370840 hasRelatedWork W4246264554 @default.
- W3009370840 hasRelatedWork W4246956338 @default.
- W3009370840 hasRelatedWork W4322731236 @default.
- W3009370840 isParatext "false" @default.
- W3009370840 isRetracted "false" @default.
- W3009370840 magId "3009370840" @default.
- W3009370840 workType "article" @default.