Matches in SemOpenAlex for { <https://semopenalex.org/work/W4312780108> ?p ?o ?g. }
Showing items 1 to 85 of
85
with 100 items per page.
- W4312780108 abstract "Distributed dataflow systems like Apache Spark and Apache Hadoop enable data-parallel processing of large datasets on clusters. Yet, selecting appropriate computational resources for dataflow jobs -- that neither lead to bottlenecks nor to low resource utilization -- is often challenging, even for expert users such as data engineers. Further, existing automated approaches to resource selection rely on the assumption that a job is recurring to learn from previous runs or to warrant the cost of full test runs to learn from. However, this assumption often does not hold since many jobs are too unique. Therefore, we present Crispy, a method for optimizing data processing cluster configurations based on job profiling runs with small samples of the dataset on just a single machine. Crispy attempts to extrapolate the memory usage for the full dataset to then choose a cluster configuration with enough total memory. In our evaluation on a dataset with 1031 Spark and Hadoop jobs, we see a reduction of job execution costs by 56% compared to the baseline, while on average spending less than ten minutes on profiling runs per job on a consumer-grade laptop." @default.
- W4312780108 created "2023-01-05" @default.
- W4312780108 creator A5002540373 @default.
- W4312780108 creator A5024675213 @default.
- W4312780108 creator A5042349846 @default.
- W4312780108 creator A5051029385 @default.
- W4312780108 creator A5084056435 @default.
- W4312780108 date "2022-09-01" @default.
- W4312780108 modified "2023-10-02" @default.
- W4312780108 title "Get Your Memory Right: The Crispy Resource Allocation Assistant for Large-Scale Data Processing" @default.
- W4312780108 cites W2019751555 @default.
- W4312780108 cites W2023214828 @default.
- W4312780108 cites W2070600700 @default.
- W4312780108 cites W2108907578 @default.
- W4312780108 cites W2114896543 @default.
- W4312780108 cites W2156697773 @default.
- W4312780108 cites W2160121678 @default.
- W4312780108 cites W2521550930 @default.
- W4312780108 cites W2546571074 @default.
- W4312780108 cites W2572526791 @default.
- W4312780108 cites W2910172404 @default.
- W4312780108 cites W2963642335 @default.
- W4312780108 cites W2963822306 @default.
- W4312780108 cites W2968631515 @default.
- W4312780108 cites W3029946860 @default.
- W4312780108 cites W3102428287 @default.
- W4312780108 cites W3126778967 @default.
- W4312780108 cites W3137810510 @default.
- W4312780108 cites W3197557987 @default.
- W4312780108 cites W3207844152 @default.
- W4312780108 cites W3211837897 @default.
- W4312780108 cites W3213171603 @default.
- W4312780108 cites W3213987511 @default.
- W4312780108 cites W3215201015 @default.
- W4312780108 cites W3216845212 @default.
- W4312780108 cites W4281490613 @default.
- W4312780108 doi "https://doi.org/10.1109/ic2e55432.2022.00014" @default.
- W4312780108 hasPublicationYear "2022" @default.
- W4312780108 type Work @default.
- W4312780108 citedByCount "4" @default.
- W4312780108 countsByYear W43127801082022 @default.
- W4312780108 crossrefType "proceedings-article" @default.
- W4312780108 hasAuthorship W4312780108A5002540373 @default.
- W4312780108 hasAuthorship W4312780108A5024675213 @default.
- W4312780108 hasAuthorship W4312780108A5042349846 @default.
- W4312780108 hasAuthorship W4312780108A5051029385 @default.
- W4312780108 hasAuthorship W4312780108A5084056435 @default.
- W4312780108 hasBestOaLocation W43127801082 @default.
- W4312780108 hasConcept C111919701 @default.
- W4312780108 hasConcept C138827492 @default.
- W4312780108 hasConcept C187191949 @default.
- W4312780108 hasConcept C199360897 @default.
- W4312780108 hasConcept C2780008327 @default.
- W4312780108 hasConcept C2781215313 @default.
- W4312780108 hasConcept C41008148 @default.
- W4312780108 hasConcept C75684735 @default.
- W4312780108 hasConcept C77088390 @default.
- W4312780108 hasConceptScore W4312780108C111919701 @default.
- W4312780108 hasConceptScore W4312780108C138827492 @default.
- W4312780108 hasConceptScore W4312780108C187191949 @default.
- W4312780108 hasConceptScore W4312780108C199360897 @default.
- W4312780108 hasConceptScore W4312780108C2780008327 @default.
- W4312780108 hasConceptScore W4312780108C2781215313 @default.
- W4312780108 hasConceptScore W4312780108C41008148 @default.
- W4312780108 hasConceptScore W4312780108C75684735 @default.
- W4312780108 hasConceptScore W4312780108C77088390 @default.
- W4312780108 hasFunder F4320320879 @default.
- W4312780108 hasFunder F4320321114 @default.
- W4312780108 hasLocation W43127801081 @default.
- W4312780108 hasLocation W43127801082 @default.
- W4312780108 hasOpenAccess W4312780108 @default.
- W4312780108 hasPrimaryLocation W43127801081 @default.
- W4312780108 hasRelatedWork W2569819632 @default.
- W4312780108 hasRelatedWork W2604594937 @default.
- W4312780108 hasRelatedWork W3006311829 @default.
- W4312780108 hasRelatedWork W3085221890 @default.
- W4312780108 hasRelatedWork W3097345360 @default.
- W4312780108 hasRelatedWork W3099307300 @default.
- W4312780108 hasRelatedWork W3202731209 @default.
- W4312780108 hasRelatedWork W3217778767 @default.
- W4312780108 hasRelatedWork W4282025595 @default.
- W4312780108 hasRelatedWork W4294158474 @default.
- W4312780108 isParatext "false" @default.
- W4312780108 isRetracted "false" @default.
- W4312780108 workType "article" @default.