Matches in SemOpenAlex for { <https://semopenalex.org/work/W1980214592> ?p ?o ?g. }
Showing items 1 to 74 of
74
with 100 items per page.
- W1980214592 abstract "Robots.txt non cooperating web crawlers are unwanted by any website as they can create serious negative impact in terms of denial of service, privacy and cost. Defense mechanisms such as automated content access protocol, captcha, web crawler trap, real time bot detection etc. have been proposed to protect websites from unwanted crawler access. Although, the extent of these mechanisms being practically applied against such crawlers is not known clearly. In this paper we present an investigation carried out to get insights about defense mechanisms used by websites against robots.txt non cooperating web crawlers. This investigation is limited only to search engine class of websites. MBot, a self-developed non cooperating web crawler is the primary tool used for investigation. On investigation we find that search engine websites do have defense mechanisms to prevent non cooperating crawler access on them. Although, absence of any kind of defense phenomena to prevent MBot's access is also observed on some of the investigated websites. Robustness in observed defense mechanisms to basic network and application parameters like proxy, port number, user agent, IP address etc. is also observed." @default.
- W1980214592 created "2016-06-24" @default.
- W1980214592 creator A5024884227 @default.
- W1980214592 creator A5060394290 @default.
- W1980214592 creator A5086231931 @default.
- W1980214592 date "2012-10-01" @default.
- W1980214592 modified "2023-09-28" @default.
- W1980214592 title "Defense response of search engine websites to non cooperating crawlers" @default.
- W1980214592 cites W183174002 @default.
- W1980214592 cites W2038311291 @default.
- W1980214592 cites W2052747824 @default.
- W1980214592 cites W2062284582 @default.
- W1980214592 cites W2094560935 @default.
- W1980214592 cites W2159185913 @default.
- W1980214592 cites W2102475166 @default.
- W1980214592 doi "https://doi.org/10.1109/wict.2012.6409078" @default.
- W1980214592 hasPublicationYear "2012" @default.
- W1980214592 type Work @default.
- W1980214592 sameAs 1980214592 @default.
- W1980214592 citedByCount "1" @default.
- W1980214592 countsByYear W19802145922014 @default.
- W1980214592 crossrefType "proceedings-article" @default.
- W1980214592 hasAuthorship W1980214592A5024884227 @default.
- W1980214592 hasAuthorship W1980214592A5060394290 @default.
- W1980214592 hasAuthorship W1980214592A5086231931 @default.
- W1980214592 hasConcept C110875604 @default.
- W1980214592 hasConcept C11392498 @default.
- W1980214592 hasConcept C136764020 @default.
- W1980214592 hasConcept C13743948 @default.
- W1980214592 hasConcept C173576120 @default.
- W1980214592 hasConcept C22735295 @default.
- W1980214592 hasConcept C38652104 @default.
- W1980214592 hasConcept C38822068 @default.
- W1980214592 hasConcept C41008148 @default.
- W1980214592 hasConcept C73340581 @default.
- W1980214592 hasConcept C97854310 @default.
- W1980214592 hasConceptScore W1980214592C110875604 @default.
- W1980214592 hasConceptScore W1980214592C11392498 @default.
- W1980214592 hasConceptScore W1980214592C136764020 @default.
- W1980214592 hasConceptScore W1980214592C13743948 @default.
- W1980214592 hasConceptScore W1980214592C173576120 @default.
- W1980214592 hasConceptScore W1980214592C22735295 @default.
- W1980214592 hasConceptScore W1980214592C38652104 @default.
- W1980214592 hasConceptScore W1980214592C38822068 @default.
- W1980214592 hasConceptScore W1980214592C41008148 @default.
- W1980214592 hasConceptScore W1980214592C73340581 @default.
- W1980214592 hasConceptScore W1980214592C97854310 @default.
- W1980214592 hasLocation W19802145921 @default.
- W1980214592 hasOpenAccess W1980214592 @default.
- W1980214592 hasPrimaryLocation W19802145921 @default.
- W1980214592 hasRelatedWork W1501777393 @default.
- W1980214592 hasRelatedWork W163011932 @default.
- W1980214592 hasRelatedWork W1965434526 @default.
- W1980214592 hasRelatedWork W1973341111 @default.
- W1980214592 hasRelatedWork W1996051673 @default.
- W1980214592 hasRelatedWork W2011761246 @default.
- W1980214592 hasRelatedWork W2043254368 @default.
- W1980214592 hasRelatedWork W2105123614 @default.
- W1980214592 hasRelatedWork W2128396626 @default.
- W1980214592 hasRelatedWork W2138393708 @default.
- W1980214592 hasRelatedWork W2159185913 @default.
- W1980214592 hasRelatedWork W2311576849 @default.
- W1980214592 hasRelatedWork W2567414736 @default.
- W1980214592 hasRelatedWork W2626977282 @default.
- W1980214592 hasRelatedWork W2805486984 @default.
- W1980214592 hasRelatedWork W2904382902 @default.
- W1980214592 hasRelatedWork W2970191066 @default.
- W1980214592 hasRelatedWork W3042540910 @default.
- W1980214592 hasRelatedWork W3117537110 @default.
- W1980214592 hasRelatedWork W3166571549 @default.
- W1980214592 isParatext "false" @default.
- W1980214592 isRetracted "false" @default.
- W1980214592 magId "1980214592" @default.
- W1980214592 workType "article" @default.