Matches in SemOpenAlex for { <https://semopenalex.org/work/W2899041175> ?p ?o ?g. }
Showing items 1 to 69 of
69
with 100 items per page.
- W2899041175 abstract "In this paper, we describe a use case applying a scientific workflow system and a distributed file system to improve the performance of telescope data processing. The application is pipeline processing of data generated by Hyper Suprime-Cam (HSC) which is a focal plane camera mounted on the Subaru telescope. In this paper, we focus on the scalability of parallel I/O and core utilization. The IBM Spectrum Scale (GPFS) used for actual operation has a limit on scalability due to the configuration using storage servers. Therefore, we introduce the Gfarm file system which uses the storage of the worker node for parallel I/O performance. To improve core utilization, we introduce the Pwrake workflow system instead of the parallel processing framework developed for the HSC pipeline. Descriptions of task dependencies are necessary to further improve core utilization by overlapping different types of tasks. We discuss the usefulness of the workflow description language with the function of scripting language for defining complex task dependency. In the experiment, the performance of the pipeline is evaluated using a quarter of the observation data per night (input files: 80 GB, output files: 1.2 TB). Measurements on strong scaling from 48 to 576 cores show that the processing with Gfarm file system is more scalable than that with GPFS. Measurement using 576 cores shows that our method improves the processing speed of the pipeline by 2.2 times compared with the method used in actual operation." @default.
- W2899041175 created "2018-11-09" @default.
- W2899041175 creator A5009328856 @default.
- W2899041175 creator A5021977279 @default.
- W2899041175 creator A5072526434 @default.
- W2899041175 date "2018-09-01" @default.
- W2899041175 modified "2023-09-24" @default.
- W2899041175 title "Applying Pwrake Workflow System and Gfarm File System to Telescope Data Processing" @default.
- W2899041175 cites W1601588053 @default.
- W2899041175 cites W1621944069 @default.
- W2899041175 cites W1767392915 @default.
- W2899041175 cites W1966832294 @default.
- W2899041175 cites W2006519720 @default.
- W2899041175 cites W2050569055 @default.
- W2899041175 cites W2066047894 @default.
- W2899041175 cites W2077579791 @default.
- W2899041175 cites W2096812688 @default.
- W2899041175 cites W2129666400 @default.
- W2899041175 cites W2137515654 @default.
- W2899041175 cites W2146917903 @default.
- W2899041175 cites W2152956081 @default.
- W2899041175 doi "https://doi.org/10.1109/cluster.2018.00024" @default.
- W2899041175 hasPublicationYear "2018" @default.
- W2899041175 type Work @default.
- W2899041175 sameAs 2899041175 @default.
- W2899041175 citedByCount "3" @default.
- W2899041175 countsByYear W28990411752022 @default.
- W2899041175 countsByYear W28990411752023 @default.
- W2899041175 crossrefType "proceedings-article" @default.
- W2899041175 hasAuthorship W2899041175A5009328856 @default.
- W2899041175 hasAuthorship W2899041175A5021977279 @default.
- W2899041175 hasAuthorship W2899041175A5072526434 @default.
- W2899041175 hasConcept C111919701 @default.
- W2899041175 hasConcept C138827492 @default.
- W2899041175 hasConcept C177212765 @default.
- W2899041175 hasConcept C2780940931 @default.
- W2899041175 hasConcept C41008148 @default.
- W2899041175 hasConcept C43521106 @default.
- W2899041175 hasConcept C48044578 @default.
- W2899041175 hasConcept C61423126 @default.
- W2899041175 hasConcept C77088390 @default.
- W2899041175 hasConcept C93996380 @default.
- W2899041175 hasConceptScore W2899041175C111919701 @default.
- W2899041175 hasConceptScore W2899041175C138827492 @default.
- W2899041175 hasConceptScore W2899041175C177212765 @default.
- W2899041175 hasConceptScore W2899041175C2780940931 @default.
- W2899041175 hasConceptScore W2899041175C41008148 @default.
- W2899041175 hasConceptScore W2899041175C43521106 @default.
- W2899041175 hasConceptScore W2899041175C48044578 @default.
- W2899041175 hasConceptScore W2899041175C61423126 @default.
- W2899041175 hasConceptScore W2899041175C77088390 @default.
- W2899041175 hasConceptScore W2899041175C93996380 @default.
- W2899041175 hasLocation W28990411751 @default.
- W2899041175 hasOpenAccess W2899041175 @default.
- W2899041175 hasPrimaryLocation W28990411751 @default.
- W2899041175 hasRelatedWork W1496652440 @default.
- W2899041175 hasRelatedWork W1513485744 @default.
- W2899041175 hasRelatedWork W1601416600 @default.
- W2899041175 hasRelatedWork W2131206796 @default.
- W2899041175 hasRelatedWork W2293862011 @default.
- W2899041175 hasRelatedWork W2350294564 @default.
- W2899041175 hasRelatedWork W2886811623 @default.
- W2899041175 hasRelatedWork W4294281692 @default.
- W2899041175 hasRelatedWork W2518924816 @default.
- W2899041175 hasRelatedWork W2591580413 @default.
- W2899041175 isParatext "false" @default.
- W2899041175 isRetracted "false" @default.
- W2899041175 magId "2899041175" @default.
- W2899041175 workType "article" @default.