FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W2200523128> ?p ?o ?g. }

• W2200523128 abstract "Many current test collections require the use of expert judgments during construction. The true label of each document is given by an expert assessor. However, the cost and effort associated with expert training and judging are typically quite high in the event where we have a high number of documents to judge. One way to address this issue is to have each document judged by multiple non-expert assessors at a lower expense. However, there are two key factors that can make this method difficult: the variability across assessors’ judging abilities, and the aggregation of the noisy labels into one single consensus label. Much previous work has shown how to utilize this method to replace expert labels in the relevance evaluation. However, the effects of relevance judgment errors on the ranking system evaluation have been less explored. This thesis mainly investigates how to best evaluate information retrieval systems with noisy labels, where no ground-truth labels are provided, and where each document may receive multiple noisy labels. Based on our simulation results on two datasets, we find that conservative assessors that tend to label incoming documents as non-relevant are preferable. And there are two important factors affect the overall conservativeness of the consensus labels: the assessor’s conservativeness and the relevance standard. This important observation essentially provides a guideline on what kind of consensus algorithms or assessors are needed in order to preserve the high correlation with expert labels in ranking system evaluation. Also, we systematically investigate how to find the consensus labels for those documents with equal confidence to be either relevant or non-relevant. We investigate a content-based consensus algorithm which links the noisy labels with document content. We compare it against the state-of-art consensus algorithms, and find that, depending on the document collection, this content-based approach may help or hurt the performance." @default.
• W2200523128 created "2016-06-24" @default.
• W2200523128 creator A5082893297 @default.
• W2200523128 date "2013-08-21" @default.
• W2200523128 modified "2023-09-23" @default.
• W2200523128 title "Evaluating Information Retrieval Systems With Multiple Non-Expert Assessors" @default.
• W2200523128 cites W135190683 @default.
• W2200523128 cites W1497983610 @default.
• W2200523128 cites W1546195864 @default.
• W2200523128 cites W1577841485 @default.
• W2200523128 cites W1685426458 @default.
• W2200523128 cites W1737790567 @default.
• W2200523128 cites W1938221098 @default.
• W2200523128 cites W1965690069 @default.
• W2200523128 cites W1966835268 @default.
• W2200523128 cites W1967879792 @default.
• W2200523128 cites W1972821056 @default.
• W2200523128 cites W1979483312 @default.
• W2200523128 cites W1983595289 @default.
• W2200523128 cites W1985514943 @default.
• W2200523128 cites W1987016195 @default.
• W2200523128 cites W1990170776 @default.
• W2200523128 cites W2003497265 @default.
• W2200523128 cites W2005917549 @default.
• W2200523128 cites W2008285050 @default.
• W2200523128 cites W2009954908 @default.
• W2200523128 cites W2010135967 @default.
• W2200523128 cites W2024166834 @default.
• W2200523128 cites W2026969101 @default.
• W2200523128 cites W2036431079 @default.
• W2200523128 cites W2048045485 @default.
• W2200523128 cites W2048679005 @default.
• W2200523128 cites W2053584122 @default.
• W2200523128 cites W2057495142 @default.
• W2200523128 cites W2062236692 @default.
• W2200523128 cites W2075893676 @default.
• W2200523128 cites W2098865355 @default.
• W2200523128 cites W2102698119 @default.
• W2200523128 cites W2114269021 @default.
• W2200523128 cites W2120308175 @default.
• W2200523128 cites W2130145422 @default.
• W2200523128 cites W2132232498 @default.
• W2200523128 cites W2134305421 @default.
• W2200523128 cites W2139516750 @default.
• W2200523128 cites W2143539737 @default.
• W2200523128 cites W2150234222 @default.
• W2200523128 cites W2153292911 @default.
• W2200523128 cites W2153763032 @default.
• W2200523128 cites W2160719354 @default.
• W2200523128 cites W2165062189 @default.
• W2200523128 cites W2165387257 @default.
• W2200523128 cites W2181558882 @default.
• W2200523128 cites W2187262079 @default.
• W2200523128 cites W2187291759 @default.
• W2200523128 cites W2187456046 @default.
• W2200523128 cites W2221430606 @default.
• W2200523128 cites W2288846786 @default.
• W2200523128 cites W279631868 @default.
• W2200523128 cites W2915843878 @default.
• W2200523128 cites W2916045930 @default.
• W2200523128 cites W3124258878 @default.
• W2200523128 cites W320171562 @default.
• W2200523128 cites W9014458 @default.
• W2200523128 hasPublicationYear "2013" @default.
• W2200523128 type Work @default.
• W2200523128 sameAs 2200523128 @default.
• W2200523128 citedByCount "5" @default.
• W2200523128 countsByYear W22005231282014 @default.
• W2200523128 crossrefType "dissertation" @default.
• W2200523128 hasAuthorship W2200523128A5082893297 @default.
• W2200523128 hasConcept C119857082 @default.
• W2200523128 hasConcept C124101348 @default.
• W2200523128 hasConcept C151730666 @default.
• W2200523128 hasConcept C154945302 @default.
• W2200523128 hasConcept C158154518 @default.
• W2200523128 hasConcept C17744445 @default.
• W2200523128 hasConcept C189430467 @default.
• W2200523128 hasConcept C199539241 @default.
• W2200523128 hasConcept C23123220 @default.
• W2200523128 hasConcept C26517878 @default.
• W2200523128 hasConcept C2777267654 @default.
• W2200523128 hasConcept C38652104 @default.
• W2200523128 hasConcept C41008148 @default.
• W2200523128 hasConcept C58328972 @default.
• W2200523128 hasConcept C86803240 @default.
• W2200523128 hasConceptScore W2200523128C119857082 @default.
• W2200523128 hasConceptScore W2200523128C124101348 @default.
• W2200523128 hasConceptScore W2200523128C151730666 @default.
• W2200523128 hasConceptScore W2200523128C154945302 @default.
• W2200523128 hasConceptScore W2200523128C158154518 @default.
• W2200523128 hasConceptScore W2200523128C17744445 @default.
• W2200523128 hasConceptScore W2200523128C189430467 @default.
• W2200523128 hasConceptScore W2200523128C199539241 @default.
• W2200523128 hasConceptScore W2200523128C23123220 @default.
• W2200523128 hasConceptScore W2200523128C26517878 @default.
• W2200523128 hasConceptScore W2200523128C2777267654 @default.
• W2200523128 hasConceptScore W2200523128C38652104 @default.
• W2200523128 hasConceptScore W2200523128C41008148 @default.
• W2200523128 hasConceptScore W2200523128C58328972 @default.
• W2200523128 hasConceptScore W2200523128C86803240 @default.

• next