FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W2910932239> ?p ?o ?g. }

• W2910932239 abstract "Background: It is important for developers of automated scoring systems to ensure that their systems are as fair and valid as possible. This commitment means evaluating the performance of these systems in light of construct-irrelevant response strategies. The enhancement of systems to detect and deal with these kinds of strategies is often an iterative process, whereby as new strategies come to light they need to be evaluated and effective mechanisms built into the automated scoring systems to handle them. In this paper, we focus on the Babel system, which automatically generates semantically incohesive essays. We expect that these essays may unfairly receive high scores from automated scoring engines despite essentially being nonsense. Literature Review: We discuss literature related to gaming of automated scoring systems. One reason that Babel essays are so easy to identify as nonsense by human readers is that they lack any semantic cohesion. Therefore, we also discuss some literature related to cohesion and detecting semantic cohesion. Research Questions: This study addressed three research questions:Can we automatically detect essays generated by the Babel system?Can we integrate the detection of Babel-generated essays into an operational automated essay scoring system while making sure not to flag valid student responses?Does a general approach for detecting semantically incohesive essays also detect Babel-generated essays?Research Methodology: This article describes the creation of two corpora necessary to address the research questions: (1) a corpus of Babel-generated essays and (2) a corresponding corpus of good-faith essays. We built a classifier to distinguish Babel-generated essays from good-faith essays and investigated whether the classifier can be integrated into an automated scoring engine without adverse effects. We also developed a measure of lexical-semantic cohesion and examined its distribution in Babel and in good-faith essays.Results: We found that the classifier built on Babel-generated essays and good-faith essays and using features from the automated scoring engine can distinguish the Babel-generated essays from the good-faith ones with 100% accuracy. We also found that if we integrated this classifier into the automated scoring engine it flagged very few responses that were submitted as part of operational submissions (76 of 434,656). The responses that were flagged had previously been assigned a score of Null (non-scorable) or a score of 1 by human experts. The measure of lexical-semantic cohesion shows promise in being able to distinguish Babel-generated essays from good-faith essays.Conclusions: Our results show that it is possible to detect the kind of gaming strategy illustrated by the Babel system and add it to an automated scoring engine without adverse effects on essays seen during real high-stakes tests. We also show that a measure of lexical-semantic cohesion can separate Babel-generated essays from good-faith essays to a certain degree, depending on task. This points to future work that would generalize the capability to detect semantic incoherence in essays. Directions for Further Research: Babel-generated essays can be identified and flagged by an automated scoring system without any adverse effects on a large set of good-faith essays. However, this is just one type of gaming strategy. It is important for developers of automated scoring systems to continue to be diligent about expanding the construct coverage of their systems in order to prevent weaknesses that can be exploited by tools such as Babel. It is also important to focus on the underlying linguistic reasons that lead to nonsense sentences. Successful identification of such nonsense would lead to improved automated scoring and feedback." @default.
• W2910932239 created "2019-01-25" @default.
• W2910932239 creator A5041834124 @default.
• W2910932239 creator A5048408884 @default.
• W2910932239 creator A5066547058 @default.
• W2910932239 date "2018-01-01" @default.
• W2910932239 modified "2023-09-26" @default.
• W2910932239 title "Developing an e-rater Advisory to Detect Babel-generated Essays" @default.
• W2910932239 cites W1489616935 @default.
• W2910932239 cites W1532156013 @default.
• W2910932239 cites W1710422233 @default.
• W2910932239 cites W181196098 @default.
• W2910932239 cites W1977196161 @default.
• W2910932239 cites W2015933299 @default.
• W2910932239 cites W2018072436 @default.
• W2910932239 cites W2019413183 @default.
• W2910932239 cites W2045178568 @default.
• W2910932239 cites W2047867333 @default.
• W2910932239 cites W2054745812 @default.
• W2910932239 cites W2054929903 @default.
• W2910932239 cites W2065841724 @default.
• W2910932239 cites W2080322628 @default.
• W2910932239 cites W2093464183 @default.
• W2910932239 cites W2095577883 @default.
• W2910932239 cites W2118898541 @default.
• W2910932239 cites W2153371246 @default.
• W2910932239 cites W2153579005 @default.
• W2910932239 cites W2158240052 @default.
• W2910932239 cites W2187342979 @default.
• W2910932239 cites W2250189634 @default.
• W2910932239 cites W2250952041 @default.
• W2910932239 cites W2251159501 @default.
• W2910932239 cites W2251930319 @default.
• W2910932239 cites W2511531539 @default.
• W2910932239 cites W2806152843 @default.
• W2910932239 cites W2911964244 @default.
• W2910932239 doi "https://doi.org/10.37514/jwa-j.2018.2.1.08" @default.
• W2910932239 hasPublicationYear "2018" @default.
• W2910932239 type Work @default.
• W2910932239 sameAs 2910932239 @default.
• W2910932239 citedByCount "3" @default.
• W2910932239 countsByYear W29109322392020 @default.
• W2910932239 countsByYear W29109322392021 @default.
• W2910932239 crossrefType "journal-article" @default.
• W2910932239 hasAuthorship W2910932239A5041834124 @default.
• W2910932239 hasAuthorship W2910932239A5048408884 @default.
• W2910932239 hasAuthorship W2910932239A5066547058 @default.
• W2910932239 hasConcept C104054115 @default.
• W2910932239 hasConcept C154945302 @default.
• W2910932239 hasConcept C178790620 @default.
• W2910932239 hasConcept C185592680 @default.
• W2910932239 hasConcept C204321447 @default.
• W2910932239 hasConcept C23123220 @default.
• W2910932239 hasConcept C2522767166 @default.
• W2910932239 hasConcept C41008148 @default.
• W2910932239 hasConcept C95623464 @default.
• W2910932239 hasConceptScore W2910932239C104054115 @default.
• W2910932239 hasConceptScore W2910932239C154945302 @default.
• W2910932239 hasConceptScore W2910932239C178790620 @default.
• W2910932239 hasConceptScore W2910932239C185592680 @default.
• W2910932239 hasConceptScore W2910932239C204321447 @default.
• W2910932239 hasConceptScore W2910932239C23123220 @default.
• W2910932239 hasConceptScore W2910932239C2522767166 @default.
• W2910932239 hasConceptScore W2910932239C41008148 @default.
• W2910932239 hasConceptScore W2910932239C95623464 @default.
• W2910932239 hasLocation W29109322391 @default.
• W2910932239 hasOpenAccess W2910932239 @default.
• W2910932239 hasPrimaryLocation W29109322391 @default.
• W2910932239 hasRelatedWork W145835004 @default.
• W2910932239 hasRelatedWork W1480949871 @default.
• W2910932239 hasRelatedWork W1537134536 @default.
• W2910932239 hasRelatedWork W2097342019 @default.
• W2910932239 hasRelatedWork W2107417444 @default.
• W2910932239 hasRelatedWork W2111499444 @default.
• W2910932239 hasRelatedWork W2125436846 @default.
• W2910932239 hasRelatedWork W2213888577 @default.
• W2910932239 hasRelatedWork W2275056699 @default.
• W2910932239 hasRelatedWork W2289052906 @default.
• W2910932239 hasRelatedWork W2737290437 @default.
• W2910932239 hasRelatedWork W2793353517 @default.
• W2910932239 hasRelatedWork W2936074642 @default.
• W2910932239 hasRelatedWork W2946072937 @default.
• W2910932239 hasRelatedWork W2954574950 @default.
• W2910932239 hasRelatedWork W2995238850 @default.
• W2910932239 hasRelatedWork W30318985 @default.
• W2910932239 hasRelatedWork W3171929203 @default.
• W2910932239 hasRelatedWork W48501643 @default.
• W2910932239 hasRelatedWork W3092696361 @default.
• W2910932239 isParatext "false" @default.
• W2910932239 isRetracted "false" @default.
• W2910932239 magId "2910932239" @default.
• W2910932239 workType "article" @default.