Matches in SemOpenAlex for { <https://semopenalex.org/work/W3122359963> ?p ?o ?g. }
- W3122359963 abstract "Manual evaluation is essential to judge progress on automatic text summarization. However, we conduct a survey on recent summarization system papers that reveals little agreement on how to perform such evaluation studies. We conduct two evaluation experiments on two aspects of summaries' linguistic quality (coherence and repetitiveness) to compare Likert-type and ranking annotations and show that best choice of evaluation method can vary from one aspect to another. In our survey, we also find that study parameters such as the overall number of annotators and distribution of annotators to annotation items are often not fully reported and that subsequent statistical analysis ignores grouping factors arising from one annotator judging multiple summaries. Using our evaluation experiments, we show that the total number of annotators can have a strong impact on study power and that current statistical analysis methods can inflate type I error rates up to eight-fold. In addition, we highlight that for the purpose of system comparison the current practice of eliciting multiple judgements per summary leads to less powerful and reliable annotations given a fixed study budget." @default.
- W3122359963 created "2021-02-01" @default.
- W3122359963 creator A5011795541 @default.
- W3122359963 creator A5082971401 @default.
- W3122359963 date "2021-01-27" @default.
- W3122359963 modified "2023-09-26" @default.
- W3122359963 title "How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluation" @default.
- W3122359963 cites W1520857482 @default.
- W3122359963 cites W1544827683 @default.
- W3122359963 cites W2016522586 @default.
- W3122359963 cites W2100175927 @default.
- W3122359963 cites W2102065370 @default.
- W3122359963 cites W2110921901 @default.
- W3122359963 cites W2136082655 @default.
- W3122359963 cites W2141845152 @default.
- W3122359963 cites W214995755 @default.
- W3122359963 cites W2153222072 @default.
- W3122359963 cites W2153804780 @default.
- W3122359963 cites W2154652894 @default.
- W3122359963 cites W2158751803 @default.
- W3122359963 cites W2250332012 @default.
- W3122359963 cites W2251023345 @default.
- W3122359963 cites W2251171258 @default.
- W3122359963 cites W2345613236 @default.
- W3122359963 cites W2606974598 @default.
- W3122359963 cites W2739751068 @default.
- W3122359963 cites W2741672218 @default.
- W3122359963 cites W2741935727 @default.
- W3122359963 cites W2773795499 @default.
- W3122359963 cites W2798935874 @default.
- W3122359963 cites W2804665012 @default.
- W3122359963 cites W2842624112 @default.
- W3122359963 cites W2889518897 @default.
- W3122359963 cites W2896780650 @default.
- W3122359963 cites W2897820187 @default.
- W3122359963 cites W2938002492 @default.
- W3122359963 cites W2951265142 @default.
- W3122359963 cites W2952245931 @default.
- W3122359963 cites W2962972512 @default.
- W3122359963 cites W2963047186 @default.
- W3122359963 cites W2963913216 @default.
- W3122359963 cites W2970785793 @default.
- W3122359963 cites W2970807214 @default.
- W3122359963 cites W2970886762 @default.
- W3122359963 cites W2971034336 @default.
- W3122359963 cites W2971289520 @default.
- W3122359963 cites W2987624052 @default.
- W3122359963 cites W2992347006 @default.
- W3122359963 cites W2995969307 @default.
- W3122359963 cites W2996614149 @default.
- W3122359963 cites W3035628162 @default.
- W3122359963 cites W3035729454 @default.
- W3122359963 cites W3045321166 @default.
- W3122359963 cites W3081757849 @default.
- W3122359963 cites W3100501376 @default.
- W3122359963 cites W3102195370 @default.
- W3122359963 cites W2756774204 @default.
- W3122359963 cites W3034360475 @default.
- W3122359963 hasPublicationYear "2021" @default.
- W3122359963 type Work @default.
- W3122359963 sameAs 3122359963 @default.
- W3122359963 citedByCount "0" @default.
- W3122359963 crossrefType "posted-content" @default.
- W3122359963 hasAuthorship W3122359963A5011795541 @default.
- W3122359963 hasAuthorship W3122359963A5082971401 @default.
- W3122359963 hasConcept C105776082 @default.
- W3122359963 hasConcept C105795698 @default.
- W3122359963 hasConcept C111472728 @default.
- W3122359963 hasConcept C138885662 @default.
- W3122359963 hasConcept C154945302 @default.
- W3122359963 hasConcept C170858558 @default.
- W3122359963 hasConcept C189430467 @default.
- W3122359963 hasConcept C204321447 @default.
- W3122359963 hasConcept C23123220 @default.
- W3122359963 hasConcept C2776321320 @default.
- W3122359963 hasConcept C2779530757 @default.
- W3122359963 hasConcept C2781181686 @default.
- W3122359963 hasConcept C2986587452 @default.
- W3122359963 hasConcept C33923547 @default.
- W3122359963 hasConcept C40696583 @default.
- W3122359963 hasConcept C41008148 @default.
- W3122359963 hasConcept C96608239 @default.
- W3122359963 hasConceptScore W3122359963C105776082 @default.
- W3122359963 hasConceptScore W3122359963C105795698 @default.
- W3122359963 hasConceptScore W3122359963C111472728 @default.
- W3122359963 hasConceptScore W3122359963C138885662 @default.
- W3122359963 hasConceptScore W3122359963C154945302 @default.
- W3122359963 hasConceptScore W3122359963C170858558 @default.
- W3122359963 hasConceptScore W3122359963C189430467 @default.
- W3122359963 hasConceptScore W3122359963C204321447 @default.
- W3122359963 hasConceptScore W3122359963C23123220 @default.
- W3122359963 hasConceptScore W3122359963C2776321320 @default.
- W3122359963 hasConceptScore W3122359963C2779530757 @default.
- W3122359963 hasConceptScore W3122359963C2781181686 @default.
- W3122359963 hasConceptScore W3122359963C2986587452 @default.
- W3122359963 hasConceptScore W3122359963C33923547 @default.
- W3122359963 hasConceptScore W3122359963C40696583 @default.
- W3122359963 hasConceptScore W3122359963C41008148 @default.
- W3122359963 hasConceptScore W3122359963C96608239 @default.
- W3122359963 hasLocation W31223599631 @default.