Matches in SemOpenAlex for { <https://semopenalex.org/work/W4313447051> ?p ?o ?g. }
Showing items 1 to 90 of
90
with 100 items per page.
- W4313447051 endingPage "54" @default.
- W4313447051 startingPage "37" @default.
- W4313447051 abstract "As a top cybersecurity vendor, Sangfor needs collects log streams from thousands of endpoint detection devices such as NTA, STA, EDR and identifies security threats in real-time way everyday. The discovery and disposal of network security incidents are highly real-time in nature with seconds or even milliseconds response time to prevent possible cyber attacks and data leaks. In order to extract more valuable information, the log streams are analyzed using stream processing with pattern matching like CEP (Complex Event Processing) in memory, and then stored in a persistent storage systems such as a data warehouse system or a search engine system for data scientists and network security engineers to do OLAP (Online Analytical Processing). Sangfor needs to build a low-latency big data platform to meet the challenges of massive logs. More and more open source systems are proposed to solve the problem of data processing in a certain aspect. Many decisions must be made to balance the benefits when designing a real-time big data infrastructure. What’s more, how to architecture these systems and construct a one-stack unified big data platform have been the key obstacles for big data analytics. In this paper, we present the overall architecture of our low-latency big data infrastructure and identify four important design decisions i.e. message queue, stream processing, OLAP, and data lake. We analyze the advantages and disadvantages of existing open source system and clarify the reason behind our choices. We also describe the improvements and optimizations to make the open-source stacks fit in Sangfor’s environments, including designing a real-time development platform based on Flink and re-architecting Apache Kylin, Clickhouse and Presto as a HOLAP system. Then we highlight two important use cases to verify the rationality of our infrastructure." @default.
- W4313447051 created "2023-01-06" @default.
- W4313447051 creator A5054709496 @default.
- W4313447051 creator A5063350388 @default.
- W4313447051 creator A5090468585 @default.
- W4313447051 date "2022-01-01" @default.
- W4313447051 modified "2023-09-26" @default.
- W4313447051 title "Towards Low-Latency Big Data Infrastructure at Sangfor" @default.
- W4313447051 cites W1976821017 @default.
- W4313447051 cites W2008503861 @default.
- W4313447051 cites W2057714068 @default.
- W4313447051 cites W2153972927 @default.
- W4313447051 cites W2247317079 @default.
- W4313447051 cites W2752238980 @default.
- W4313447051 cites W2753910905 @default.
- W4313447051 cites W2913854892 @default.
- W4313447051 cites W2971076479 @default.
- W4313447051 cites W3085477028 @default.
- W4313447051 cites W3176171201 @default.
- W4313447051 cites W3197174867 @default.
- W4313447051 doi "https://doi.org/10.1007/978-3-031-23098-1_3" @default.
- W4313447051 hasPublicationYear "2022" @default.
- W4313447051 type Work @default.
- W4313447051 citedByCount "0" @default.
- W4313447051 crossrefType "book-chapter" @default.
- W4313447051 hasAuthorship W4313447051A5054709496 @default.
- W4313447051 hasAuthorship W4313447051A5063350388 @default.
- W4313447051 hasAuthorship W4313447051A5090468585 @default.
- W4313447051 hasConcept C107027933 @default.
- W4313447051 hasConcept C111919701 @default.
- W4313447051 hasConcept C120314980 @default.
- W4313447051 hasConcept C123606473 @default.
- W4313447051 hasConcept C123657996 @default.
- W4313447051 hasConcept C124101348 @default.
- W4313447051 hasConcept C135572916 @default.
- W4313447051 hasConcept C142362112 @default.
- W4313447051 hasConcept C144133560 @default.
- W4313447051 hasConcept C153349607 @default.
- W4313447051 hasConcept C162853370 @default.
- W4313447051 hasConcept C201932085 @default.
- W4313447051 hasConcept C2522767166 @default.
- W4313447051 hasConcept C26324664 @default.
- W4313447051 hasConcept C2777338717 @default.
- W4313447051 hasConcept C31258907 @default.
- W4313447051 hasConcept C38652104 @default.
- W4313447051 hasConcept C41008148 @default.
- W4313447051 hasConcept C75684735 @default.
- W4313447051 hasConcept C77088390 @default.
- W4313447051 hasConcept C89198739 @default.
- W4313447051 hasConcept C98045186 @default.
- W4313447051 hasConceptScore W4313447051C107027933 @default.
- W4313447051 hasConceptScore W4313447051C111919701 @default.
- W4313447051 hasConceptScore W4313447051C120314980 @default.
- W4313447051 hasConceptScore W4313447051C123606473 @default.
- W4313447051 hasConceptScore W4313447051C123657996 @default.
- W4313447051 hasConceptScore W4313447051C124101348 @default.
- W4313447051 hasConceptScore W4313447051C135572916 @default.
- W4313447051 hasConceptScore W4313447051C142362112 @default.
- W4313447051 hasConceptScore W4313447051C144133560 @default.
- W4313447051 hasConceptScore W4313447051C153349607 @default.
- W4313447051 hasConceptScore W4313447051C162853370 @default.
- W4313447051 hasConceptScore W4313447051C201932085 @default.
- W4313447051 hasConceptScore W4313447051C2522767166 @default.
- W4313447051 hasConceptScore W4313447051C26324664 @default.
- W4313447051 hasConceptScore W4313447051C2777338717 @default.
- W4313447051 hasConceptScore W4313447051C31258907 @default.
- W4313447051 hasConceptScore W4313447051C38652104 @default.
- W4313447051 hasConceptScore W4313447051C41008148 @default.
- W4313447051 hasConceptScore W4313447051C75684735 @default.
- W4313447051 hasConceptScore W4313447051C77088390 @default.
- W4313447051 hasConceptScore W4313447051C89198739 @default.
- W4313447051 hasConceptScore W4313447051C98045186 @default.
- W4313447051 hasLocation W43134470511 @default.
- W4313447051 hasOpenAccess W4313447051 @default.
- W4313447051 hasPrimaryLocation W43134470511 @default.
- W4313447051 hasRelatedWork W1503826383 @default.
- W4313447051 hasRelatedWork W2293225714 @default.
- W4313447051 hasRelatedWork W2972171275 @default.
- W4313447051 hasRelatedWork W3113817755 @default.
- W4313447051 hasRelatedWork W3215629042 @default.
- W4313447051 hasRelatedWork W4206415557 @default.
- W4313447051 hasRelatedWork W4241100870 @default.
- W4313447051 hasRelatedWork W4313315914 @default.
- W4313447051 hasRelatedWork W4313447051 @default.
- W4313447051 hasRelatedWork W2549101268 @default.
- W4313447051 isParatext "false" @default.
- W4313447051 isRetracted "false" @default.
- W4313447051 workType "book-chapter" @default.