Matches in SemOpenAlex for { <https://semopenalex.org/work/W4387561255> ?p ?o ?g. }
Showing items 1 to 79 of
79
with 100 items per page.
- W4387561255 abstract "There is an emerging line of research on multimodal instruction tuning, and a line of benchmarks have been proposed for evaluating these models recently. Instead of evaluating the models directly, in this paper we try to evaluate the Vision-Language Instruction-Tuning (VLIT) datasets themselves and further seek the way of building a dataset for developing an all-powerful VLIT model, which we believe could also be of utility for establishing a grounded protocol for benchmarking VLIT models. For effective analysis of VLIT datasets that remains an open question, we propose a tune-cross-evaluation paradigm: tuning on one dataset and evaluating on the others in turn. For each single tune-evaluation experiment set, we define the Meta Quality (MQ) as the mean score measured by a series of caption metrics including BLEU, METEOR, and ROUGE-L to quantify the quality of a certain dataset or a sample. On this basis, to evaluate the comprehensiveness of a dataset, we develop the Dataset Quality (DQ) covering all tune-evaluation sets. To lay the foundation for building a comprehensive dataset and developing an all-powerful model for practical applications, we further define the Sample Quality (SQ) to quantify the all-sided quality of each sample. Extensive experiments validate the rationality of the proposed evaluation paradigm. Based on the holistic evaluation, we build a new dataset, REVO-LION (REfining VisiOn-Language InstructiOn tuNing), by collecting samples with higher SQ from each dataset. With only half of the full data, the model trained on REVO-LION can achieve performance comparable to simply adding all VLIT datasets up. In addition to developing an all-powerful model, REVO-LION also includes an evaluation set, which is expected to serve as a convenient evaluation benchmark for future research." @default.
- W4387561255 created "2023-10-12" @default.
- W4387561255 creator A5008121592 @default.
- W4387561255 creator A5029156526 @default.
- W4387561255 creator A5055774883 @default.
- W4387561255 creator A5061858149 @default.
- W4387561255 creator A5072861783 @default.
- W4387561255 creator A5087158377 @default.
- W4387561255 creator A5089966579 @default.
- W4387561255 date "2023-10-10" @default.
- W4387561255 modified "2023-10-13" @default.
- W4387561255 title "REVO-LION: Evaluating and Refining Vision-Language Instruction Tuning Datasets" @default.
- W4387561255 doi "https://doi.org/10.48550/arxiv.2310.06594" @default.
- W4387561255 hasPublicationYear "2023" @default.
- W4387561255 type Work @default.
- W4387561255 citedByCount "0" @default.
- W4387561255 crossrefType "posted-content" @default.
- W4387561255 hasAuthorship W4387561255A5008121592 @default.
- W4387561255 hasAuthorship W4387561255A5029156526 @default.
- W4387561255 hasAuthorship W4387561255A5055774883 @default.
- W4387561255 hasAuthorship W4387561255A5061858149 @default.
- W4387561255 hasAuthorship W4387561255A5072861783 @default.
- W4387561255 hasAuthorship W4387561255A5087158377 @default.
- W4387561255 hasAuthorship W4387561255A5089966579 @default.
- W4387561255 hasBestOaLocation W43875612551 @default.
- W4387561255 hasConcept C111472728 @default.
- W4387561255 hasConcept C119857082 @default.
- W4387561255 hasConcept C124101348 @default.
- W4387561255 hasConcept C138885662 @default.
- W4387561255 hasConcept C142724271 @default.
- W4387561255 hasConcept C144133560 @default.
- W4387561255 hasConcept C154945302 @default.
- W4387561255 hasConcept C162853370 @default.
- W4387561255 hasConcept C177264268 @default.
- W4387561255 hasConcept C185592680 @default.
- W4387561255 hasConcept C198531522 @default.
- W4387561255 hasConcept C199360897 @default.
- W4387561255 hasConcept C204787440 @default.
- W4387561255 hasConcept C2779530757 @default.
- W4387561255 hasConcept C2780385302 @default.
- W4387561255 hasConcept C41008148 @default.
- W4387561255 hasConcept C43617362 @default.
- W4387561255 hasConcept C71924100 @default.
- W4387561255 hasConcept C86251818 @default.
- W4387561255 hasConceptScore W4387561255C111472728 @default.
- W4387561255 hasConceptScore W4387561255C119857082 @default.
- W4387561255 hasConceptScore W4387561255C124101348 @default.
- W4387561255 hasConceptScore W4387561255C138885662 @default.
- W4387561255 hasConceptScore W4387561255C142724271 @default.
- W4387561255 hasConceptScore W4387561255C144133560 @default.
- W4387561255 hasConceptScore W4387561255C154945302 @default.
- W4387561255 hasConceptScore W4387561255C162853370 @default.
- W4387561255 hasConceptScore W4387561255C177264268 @default.
- W4387561255 hasConceptScore W4387561255C185592680 @default.
- W4387561255 hasConceptScore W4387561255C198531522 @default.
- W4387561255 hasConceptScore W4387561255C199360897 @default.
- W4387561255 hasConceptScore W4387561255C204787440 @default.
- W4387561255 hasConceptScore W4387561255C2779530757 @default.
- W4387561255 hasConceptScore W4387561255C2780385302 @default.
- W4387561255 hasConceptScore W4387561255C41008148 @default.
- W4387561255 hasConceptScore W4387561255C43617362 @default.
- W4387561255 hasConceptScore W4387561255C71924100 @default.
- W4387561255 hasConceptScore W4387561255C86251818 @default.
- W4387561255 hasLocation W43875612551 @default.
- W4387561255 hasOpenAccess W4387561255 @default.
- W4387561255 hasPrimaryLocation W43875612551 @default.
- W4387561255 hasRelatedWork W1490753184 @default.
- W4387561255 hasRelatedWork W1993948687 @default.
- W4387561255 hasRelatedWork W2059640416 @default.
- W4387561255 hasRelatedWork W2284465472 @default.
- W4387561255 hasRelatedWork W2291782699 @default.
- W4387561255 hasRelatedWork W2329895846 @default.
- W4387561255 hasRelatedWork W2341842940 @default.
- W4387561255 hasRelatedWork W2619091065 @default.
- W4387561255 hasRelatedWork W4238897586 @default.
- W4387561255 hasRelatedWork W435179959 @default.
- W4387561255 isParatext "false" @default.
- W4387561255 isRetracted "false" @default.
- W4387561255 workType "article" @default.