Matches in SemOpenAlex for { <https://semopenalex.org/work/W4312777793> ?p ?o ?g. }
Showing items 1 to 72 of
72
with 100 items per page.
- W4312777793 abstract "Benchmark frameworks and datasets allow us to analyze and understand the knowledge that NLP models capture about the world they were trained on. Several Transformer models have recently been adapted to code-related tasks such as code search, in which the goal is to find the most semantically relevant code given a query written in natural language. To achieve satisfactory performance, the retrieval models heavily rely on the quality of the query. In this paper, we introduce the Natural Language Code Search Robustness Benchmark (COBE), which provides a more holistic evaluation of the state-of-the-art models considering several aspects of the retrieval models, such as: (i) retrieval capabilities measured in multiple ranking metrics; (ii) robustness to a plethora of input perturbations; (iii) efficiency in terms of training and retrieval times; and (iv) stability across fine-tuning runs. We shed a light over important questions showing that simply computing performance-based retrieval metrics does not suffice to evaluate this kind of model. The proposed benchmark introduces novel metrics and measurement strategies that allow a rigorous quantitative analysis of input-query robustness while providing an understanding of model generalization behavior. We perform an extensive set of experiments using state-of-the-art models such as CodeBert, GraphCodeBert, and CodeT5. Those models are fine-tuned over many different scenarios in six programming languages. Several models trained in this study outperform their state-of-the-art counterparts, which provides evidence that the standard fine-tuning approach used in code search related work is sub-optimal. The proposed benchmark is a powerful tool to evaluate code search models, providing insights on how they behave during fine-tuning and how they are interpreting the input queries." @default.
- W4312777793 created "2023-01-05" @default.
- W4312777793 creator A5018120863 @default.
- W4312777793 creator A5039629929 @default.
- W4312777793 creator A5076343915 @default.
- W4312777793 date "2022-07-18" @default.
- W4312777793 modified "2023-09-27" @default.
- W4312777793 title "COBE: A Natural Language Code Search Robustness Benchmark" @default.
- W4312777793 cites W2805788202 @default.
- W4312777793 cites W2964194820 @default.
- W4312777793 cites W2982223350 @default.
- W4312777793 cites W2997525715 @default.
- W4312777793 cites W2999343753 @default.
- W4312777793 cites W3035231859 @default.
- W4312777793 cites W3089869718 @default.
- W4312777793 cites W3098605233 @default.
- W4312777793 cites W3126095862 @default.
- W4312777793 cites W3198685994 @default.
- W4312777793 doi "https://doi.org/10.1109/ijcnn55064.2022.9892610" @default.
- W4312777793 hasPublicationYear "2022" @default.
- W4312777793 type Work @default.
- W4312777793 citedByCount "0" @default.
- W4312777793 crossrefType "proceedings-article" @default.
- W4312777793 hasAuthorship W4312777793A5018120863 @default.
- W4312777793 hasAuthorship W4312777793A5039629929 @default.
- W4312777793 hasAuthorship W4312777793A5076343915 @default.
- W4312777793 hasConcept C104317684 @default.
- W4312777793 hasConcept C119857082 @default.
- W4312777793 hasConcept C124101348 @default.
- W4312777793 hasConcept C13280743 @default.
- W4312777793 hasConcept C137293760 @default.
- W4312777793 hasConcept C154945302 @default.
- W4312777793 hasConcept C185592680 @default.
- W4312777793 hasConcept C185798385 @default.
- W4312777793 hasConcept C189430467 @default.
- W4312777793 hasConcept C199360897 @default.
- W4312777793 hasConcept C205649164 @default.
- W4312777793 hasConcept C41008148 @default.
- W4312777793 hasConcept C43126263 @default.
- W4312777793 hasConcept C55493867 @default.
- W4312777793 hasConcept C63479239 @default.
- W4312777793 hasConceptScore W4312777793C104317684 @default.
- W4312777793 hasConceptScore W4312777793C119857082 @default.
- W4312777793 hasConceptScore W4312777793C124101348 @default.
- W4312777793 hasConceptScore W4312777793C13280743 @default.
- W4312777793 hasConceptScore W4312777793C137293760 @default.
- W4312777793 hasConceptScore W4312777793C154945302 @default.
- W4312777793 hasConceptScore W4312777793C185592680 @default.
- W4312777793 hasConceptScore W4312777793C185798385 @default.
- W4312777793 hasConceptScore W4312777793C189430467 @default.
- W4312777793 hasConceptScore W4312777793C199360897 @default.
- W4312777793 hasConceptScore W4312777793C205649164 @default.
- W4312777793 hasConceptScore W4312777793C41008148 @default.
- W4312777793 hasConceptScore W4312777793C43126263 @default.
- W4312777793 hasConceptScore W4312777793C55493867 @default.
- W4312777793 hasConceptScore W4312777793C63479239 @default.
- W4312777793 hasLocation W43127777931 @default.
- W4312777793 hasOpenAccess W4312777793 @default.
- W4312777793 hasPrimaryLocation W43127777931 @default.
- W4312777793 hasRelatedWork W1786507113 @default.
- W4312777793 hasRelatedWork W2039826537 @default.
- W4312777793 hasRelatedWork W2911288319 @default.
- W4312777793 hasRelatedWork W2950197776 @default.
- W4312777793 hasRelatedWork W2977900939 @default.
- W4312777793 hasRelatedWork W3107474891 @default.
- W4312777793 hasRelatedWork W3138953784 @default.
- W4312777793 hasRelatedWork W4226470611 @default.
- W4312777793 hasRelatedWork W4296775963 @default.
- W4312777793 hasRelatedWork W4320505317 @default.
- W4312777793 isParatext "false" @default.
- W4312777793 isRetracted "false" @default.
- W4312777793 workType "article" @default.