Matches in SemOpenAlex for { <https://semopenalex.org/work/W4378465274> ?p ?o ?g. }
Showing items 1 to 67 of
67
with 100 items per page.
- W4378465274 abstract "Research on automated text summarization relies heavily on human and automatic evaluation. While recent work on human evaluation mainly adopted intrinsic evaluation methods, judging the generic quality of text summaries, e.g. informativeness and coherence, our work focuses on evaluating the usefulness of text summaries with extrinsic methods. We carefully design three different downstream tasks for extrinsic human evaluation of summaries, i.e., question answering, text classification and text similarity assessment. We carry out experiments using system rankings and user behavior data to evaluate the performance of different summarization models. We find summaries are particularly useful in tasks that rely on an overall judgment of the text, while being less effective for question answering tasks. The results show that summaries generated by fine-tuned models lead to higher consistency in usefulness across all three tasks, as rankings of fine-tuned summarization systems are close across downstream tasks according to the proposed extrinsic metrics. Summaries generated by models in the zero-shot setting, however, are found to be biased towards the text classification and similarity assessment tasks, due to its general and less detailed summary style. We further evaluate the correlation of 14 intrinsic automatic metrics with human criteria and show that intrinsic automatic metrics perform well in evaluating the usefulness of summaries in the question-answering task, but are less effective in the other two tasks. This highlights the limitations of relying solely on intrinsic automatic metrics in evaluating the performance and usefulness of summaries." @default.
- W4378465274 created "2023-05-27" @default.
- W4378465274 creator A5006995666 @default.
- W4378465274 creator A5029568096 @default.
- W4378465274 creator A5055198364 @default.
- W4378465274 date "2023-05-24" @default.
- W4378465274 modified "2023-09-23" @default.
- W4378465274 title "Is Summary Useful or Not? An Extrinsic Human Evaluation of Text Summaries on Downstream Tasks" @default.
- W4378465274 doi "https://doi.org/10.48550/arxiv.2305.15044" @default.
- W4378465274 hasPublicationYear "2023" @default.
- W4378465274 type Work @default.
- W4378465274 citedByCount "0" @default.
- W4378465274 crossrefType "posted-content" @default.
- W4378465274 hasAuthorship W4378465274A5006995666 @default.
- W4378465274 hasAuthorship W4378465274A5029568096 @default.
- W4378465274 hasAuthorship W4378465274A5055198364 @default.
- W4378465274 hasBestOaLocation W43784652741 @default.
- W4378465274 hasConcept C103278499 @default.
- W4378465274 hasConcept C115961682 @default.
- W4378465274 hasConcept C121332964 @default.
- W4378465274 hasConcept C154945302 @default.
- W4378465274 hasConcept C162324750 @default.
- W4378465274 hasConcept C170858558 @default.
- W4378465274 hasConcept C187736073 @default.
- W4378465274 hasConcept C204321447 @default.
- W4378465274 hasConcept C21547014 @default.
- W4378465274 hasConcept C23123220 @default.
- W4378465274 hasConcept C2776207758 @default.
- W4378465274 hasConcept C2776436953 @default.
- W4378465274 hasConcept C2780451532 @default.
- W4378465274 hasConcept C2781181686 @default.
- W4378465274 hasConcept C41008148 @default.
- W4378465274 hasConcept C44291984 @default.
- W4378465274 hasConcept C62520636 @default.
- W4378465274 hasConceptScore W4378465274C103278499 @default.
- W4378465274 hasConceptScore W4378465274C115961682 @default.
- W4378465274 hasConceptScore W4378465274C121332964 @default.
- W4378465274 hasConceptScore W4378465274C154945302 @default.
- W4378465274 hasConceptScore W4378465274C162324750 @default.
- W4378465274 hasConceptScore W4378465274C170858558 @default.
- W4378465274 hasConceptScore W4378465274C187736073 @default.
- W4378465274 hasConceptScore W4378465274C204321447 @default.
- W4378465274 hasConceptScore W4378465274C21547014 @default.
- W4378465274 hasConceptScore W4378465274C23123220 @default.
- W4378465274 hasConceptScore W4378465274C2776207758 @default.
- W4378465274 hasConceptScore W4378465274C2776436953 @default.
- W4378465274 hasConceptScore W4378465274C2780451532 @default.
- W4378465274 hasConceptScore W4378465274C2781181686 @default.
- W4378465274 hasConceptScore W4378465274C41008148 @default.
- W4378465274 hasConceptScore W4378465274C44291984 @default.
- W4378465274 hasConceptScore W4378465274C62520636 @default.
- W4378465274 hasLocation W43784652741 @default.
- W4378465274 hasOpenAccess W4378465274 @default.
- W4378465274 hasPrimaryLocation W43784652741 @default.
- W4378465274 hasRelatedWork W1512698090 @default.
- W4378465274 hasRelatedWork W1605559518 @default.
- W4378465274 hasRelatedWork W2016908626 @default.
- W4378465274 hasRelatedWork W2151407063 @default.
- W4378465274 hasRelatedWork W2333425924 @default.
- W4378465274 hasRelatedWork W2747680751 @default.
- W4378465274 hasRelatedWork W3107290838 @default.
- W4378465274 hasRelatedWork W4295954116 @default.
- W4378465274 hasRelatedWork W1520020687 @default.
- W4378465274 hasRelatedWork W2742008479 @default.
- W4378465274 isParatext "false" @default.
- W4378465274 isRetracted "false" @default.
- W4378465274 workType "article" @default.