Matches in SemOpenAlex for { <https://semopenalex.org/work/W4308642998> ?p ?o ?g. }
Showing items 1 to 94 of
94
with 100 items per page.
- W4308642998 abstract "Machine Learning (ML) has become the cornerstone of information retrieval (IR) software, as it can drive better user experience by leveraging information-rich data and complex models. However, evaluating the emergent behavior of ML-based IR software can be challenging with traditional software testing approaches: when developers modify the software, they cannot often extract useful information from individual test instances; rather, they seek to holistically verify whether—and where—their modifications caused significant regressions or improvements at scale. In this paper, we introduce not only such a holistic approach to evaluate the system-level behavior of the software, but also the concept of a defect class, which represents a partition of the input space on which the ML-based software does measurably worse for an existing feature or on which the ML task is more challenging for a new feature. We leverage large volumes of functional test cases, automatically obtained, to derive these defect classes, and propose new ways to improve the IR software from an end-user’s perspective. Applying our approach on a real production Search-AutoComplete system that contains a query interpretation ML component, we demonstrate that (1) our holistic metrics successfully identified two regressions and one improvement, where all 3 were independently verified with retrospective A/B experiments, (2) the automatically obtained defect classes provided actionable insights during early-stage ML development, and (3) we also detected defect classes at the finer sub-component level for which there were significant regressions, which we blocked prior to different releases." @default.
- W4308642998 created "2022-11-13" @default.
- W4308642998 creator A5046597133 @default.
- W4308642998 creator A5048436440 @default.
- W4308642998 creator A5054182168 @default.
- W4308642998 creator A5079784371 @default.
- W4308642998 date "2022-11-07" @default.
- W4308642998 modified "2023-09-29" @default.
- W4308642998 title "Improving ML-based information retrieval software with user-driven functional testing and defect class analysis" @default.
- W4308642998 cites W1996574389 @default.
- W4308642998 cites W1996898994 @default.
- W4308642998 cites W2020278455 @default.
- W4308642998 cites W2026784708 @default.
- W4308642998 cites W2027691696 @default.
- W4308642998 cites W2065219958 @default.
- W4308642998 cites W2087198174 @default.
- W4308642998 cites W2110441383 @default.
- W4308642998 cites W2110933113 @default.
- W4308642998 cites W2126752493 @default.
- W4308642998 cites W2150608210 @default.
- W4308642998 cites W2173213060 @default.
- W4308642998 cites W2293505944 @default.
- W4308642998 cites W2464171116 @default.
- W4308642998 cites W2787894218 @default.
- W4308642998 cites W2883656129 @default.
- W4308642998 cites W2922234936 @default.
- W4308642998 cites W2945883466 @default.
- W4308642998 cites W2963856968 @default.
- W4308642998 cites W2973084513 @default.
- W4308642998 cites W2998064020 @default.
- W4308642998 cites W3003212147 @default.
- W4308642998 cites W3007157104 @default.
- W4308642998 cites W3034708216 @default.
- W4308642998 cites W3039522238 @default.
- W4308642998 cites W3043238989 @default.
- W4308642998 cites W3048821638 @default.
- W4308642998 cites W3100925971 @default.
- W4308642998 cites W3110368543 @default.
- W4308642998 cites W3158950919 @default.
- W4308642998 cites W3162654950 @default.
- W4308642998 cites W3163698227 @default.
- W4308642998 cites W3194588521 @default.
- W4308642998 cites W4205947740 @default.
- W4308642998 doi "https://doi.org/10.1145/3540250.3558941" @default.
- W4308642998 hasPublicationYear "2022" @default.
- W4308642998 type Work @default.
- W4308642998 citedByCount "0" @default.
- W4308642998 crossrefType "proceedings-article" @default.
- W4308642998 hasAuthorship W4308642998A5046597133 @default.
- W4308642998 hasAuthorship W4308642998A5048436440 @default.
- W4308642998 hasAuthorship W4308642998A5054182168 @default.
- W4308642998 hasAuthorship W4308642998A5079784371 @default.
- W4308642998 hasConcept C1009929 @default.
- W4308642998 hasConcept C119857082 @default.
- W4308642998 hasConcept C124101348 @default.
- W4308642998 hasConcept C138885662 @default.
- W4308642998 hasConcept C153083717 @default.
- W4308642998 hasConcept C154945302 @default.
- W4308642998 hasConcept C199360897 @default.
- W4308642998 hasConcept C23123220 @default.
- W4308642998 hasConcept C2776401178 @default.
- W4308642998 hasConcept C2777212361 @default.
- W4308642998 hasConcept C2777904410 @default.
- W4308642998 hasConcept C41008148 @default.
- W4308642998 hasConcept C41895202 @default.
- W4308642998 hasConceptScore W4308642998C1009929 @default.
- W4308642998 hasConceptScore W4308642998C119857082 @default.
- W4308642998 hasConceptScore W4308642998C124101348 @default.
- W4308642998 hasConceptScore W4308642998C138885662 @default.
- W4308642998 hasConceptScore W4308642998C153083717 @default.
- W4308642998 hasConceptScore W4308642998C154945302 @default.
- W4308642998 hasConceptScore W4308642998C199360897 @default.
- W4308642998 hasConceptScore W4308642998C23123220 @default.
- W4308642998 hasConceptScore W4308642998C2776401178 @default.
- W4308642998 hasConceptScore W4308642998C2777212361 @default.
- W4308642998 hasConceptScore W4308642998C2777904410 @default.
- W4308642998 hasConceptScore W4308642998C41008148 @default.
- W4308642998 hasConceptScore W4308642998C41895202 @default.
- W4308642998 hasLocation W43086429981 @default.
- W4308642998 hasOpenAccess W4308642998 @default.
- W4308642998 hasPrimaryLocation W43086429981 @default.
- W4308642998 hasRelatedWork W2088791420 @default.
- W4308642998 hasRelatedWork W2179540415 @default.
- W4308642998 hasRelatedWork W2384888906 @default.
- W4308642998 hasRelatedWork W2961085424 @default.
- W4308642998 hasRelatedWork W4206403383 @default.
- W4308642998 hasRelatedWork W4285233257 @default.
- W4308642998 hasRelatedWork W4306674287 @default.
- W4308642998 hasRelatedWork W4310720697 @default.
- W4308642998 hasRelatedWork W4362564695 @default.
- W4308642998 hasRelatedWork W4224009465 @default.
- W4308642998 isParatext "false" @default.
- W4308642998 isRetracted "false" @default.
- W4308642998 workType "article" @default.