Matches in SemOpenAlex for { <https://semopenalex.org/work/W4367187268> ?p ?o ?g. }
- W4367187268 endingPage "69" @default.
- W4367187268 startingPage "58" @default.
- W4367187268 abstract "ObjectivesA major obstacle in deployment of models for automated quality assessment is their reliability. To analyze their calibration and selective classification performance.Study Design and SettingWe examine two systems for assessing the quality of medical evidence, EvidenceGRADEr and RobotReviewer, both developed from Cochrane Database of Systematic Reviews (CDSR) to measure strength of bodies of evidence and risk of bias (RoB) of individual studies, respectively. We report their calibration error and Brier scores, present their reliability diagrams, and analyze the risk–coverage trade-off in selective classification.ResultsThe models are reasonably well calibrated on most quality criteria (expected calibration error [ECE] 0.04–0.09 for EvidenceGRADEr, 0.03–0.10 for RobotReviewer). However, we discover that both calibration and predictive performance vary significantly by medical area. This has ramifications for the application of such models in practice, as average performance is a poor indicator of group-level performance (e.g., health and safety at work, allergy and intolerance, and public health see much worse performance than cancer, pain, and anesthesia, and Neurology). We explore the reasons behind this disparity.ConclusionPractitioners adopting automated quality assessment should expect large fluctuations in system reliability and predictive performance depending on the medical area. Prospective indicators of such behavior should be further researched." @default.
- W4367187268 created "2023-04-28" @default.
- W4367187268 creator A5008576336 @default.
- W4367187268 creator A5060332228 @default.
- W4367187268 creator A5084069668 @default.
- W4367187268 date "2023-07-01" @default.
- W4367187268 modified "2023-09-27" @default.
- W4367187268 title "Analysis of predictive performance and reliability of classifiers for quality assessment of medical evidence revealed important variation by medical area" @default.
- W4367187268 cites W1967935259 @default.
- W4367187268 cites W1984963575 @default.
- W4367187268 cites W1994466426 @default.
- W4367187268 cites W2025720061 @default.
- W4367187268 cites W2029045893 @default.
- W4367187268 cites W2073241381 @default.
- W4367187268 cites W2095601341 @default.
- W4367187268 cites W2095623039 @default.
- W4367187268 cites W2098923148 @default.
- W4367187268 cites W2106952837 @default.
- W4367187268 cites W2116485318 @default.
- W4367187268 cites W2125001590 @default.
- W4367187268 cites W2133218851 @default.
- W4367187268 cites W2137556846 @default.
- W4367187268 cites W2147947791 @default.
- W4367187268 cites W2165010366 @default.
- W4367187268 cites W2184378182 @default.
- W4367187268 cites W2271587360 @default.
- W4367187268 cites W2593758073 @default.
- W4367187268 cites W2774486220 @default.
- W4367187268 cites W2777686840 @default.
- W4367187268 cites W2788484292 @default.
- W4367187268 cites W2902105843 @default.
- W4367187268 cites W2953029151 @default.
- W4367187268 cites W2961191798 @default.
- W4367187268 cites W3003735349 @default.
- W4367187268 cites W3006673017 @default.
- W4367187268 cites W3036957975 @default.
- W4367187268 cites W3087185831 @default.
- W4367187268 cites W3130937151 @default.
- W4367187268 cites W3155544647 @default.
- W4367187268 cites W4281621725 @default.
- W4367187268 cites W4281827825 @default.
- W4367187268 doi "https://doi.org/10.1016/j.jclinepi.2023.04.006" @default.
- W4367187268 hasPubMedId "https://pubmed.ncbi.nlm.nih.gov/37120028" @default.
- W4367187268 hasPublicationYear "2023" @default.
- W4367187268 type Work @default.
- W4367187268 citedByCount "0" @default.
- W4367187268 crossrefType "journal-article" @default.
- W4367187268 hasAuthorship W4367187268A5008576336 @default.
- W4367187268 hasAuthorship W4367187268A5060332228 @default.
- W4367187268 hasAuthorship W4367187268A5084069668 @default.
- W4367187268 hasBestOaLocation W43671872681 @default.
- W4367187268 hasConcept C105795698 @default.
- W4367187268 hasConcept C111472728 @default.
- W4367187268 hasConcept C119857082 @default.
- W4367187268 hasConcept C121332964 @default.
- W4367187268 hasConcept C127413603 @default.
- W4367187268 hasConcept C138885662 @default.
- W4367187268 hasConcept C142724271 @default.
- W4367187268 hasConcept C163258240 @default.
- W4367187268 hasConcept C165838908 @default.
- W4367187268 hasConcept C19648533 @default.
- W4367187268 hasConcept C200601418 @default.
- W4367187268 hasConcept C204787440 @default.
- W4367187268 hasConcept C2779530757 @default.
- W4367187268 hasConcept C33923547 @default.
- W4367187268 hasConcept C35405484 @default.
- W4367187268 hasConcept C41008148 @default.
- W4367187268 hasConcept C43214815 @default.
- W4367187268 hasConcept C62520636 @default.
- W4367187268 hasConcept C71924100 @default.
- W4367187268 hasConceptScore W4367187268C105795698 @default.
- W4367187268 hasConceptScore W4367187268C111472728 @default.
- W4367187268 hasConceptScore W4367187268C119857082 @default.
- W4367187268 hasConceptScore W4367187268C121332964 @default.
- W4367187268 hasConceptScore W4367187268C127413603 @default.
- W4367187268 hasConceptScore W4367187268C138885662 @default.
- W4367187268 hasConceptScore W4367187268C142724271 @default.
- W4367187268 hasConceptScore W4367187268C163258240 @default.
- W4367187268 hasConceptScore W4367187268C165838908 @default.
- W4367187268 hasConceptScore W4367187268C19648533 @default.
- W4367187268 hasConceptScore W4367187268C200601418 @default.
- W4367187268 hasConceptScore W4367187268C204787440 @default.
- W4367187268 hasConceptScore W4367187268C2779530757 @default.
- W4367187268 hasConceptScore W4367187268C33923547 @default.
- W4367187268 hasConceptScore W4367187268C35405484 @default.
- W4367187268 hasConceptScore W4367187268C41008148 @default.
- W4367187268 hasConceptScore W4367187268C43214815 @default.
- W4367187268 hasConceptScore W4367187268C62520636 @default.
- W4367187268 hasConceptScore W4367187268C71924100 @default.
- W4367187268 hasFunder F4320307762 @default.
- W4367187268 hasFunder F4320334704 @default.
- W4367187268 hasLocation W43671872681 @default.
- W4367187268 hasLocation W43671872682 @default.
- W4367187268 hasOpenAccess W4367187268 @default.
- W4367187268 hasPrimaryLocation W43671872681 @default.
- W4367187268 hasRelatedWork W162860637 @default.
- W4367187268 hasRelatedWork W2033512842 @default.
- W4367187268 hasRelatedWork W2160599399 @default.