Results: 4049

IDRobotUser Agent
2(^| )sentry\/sentry/8.22.0 (
3[wW]getWget/1.14 (linux-gnu)
4[wW]getWGETbot/1.0 (+
5[wW]getWget/1.20.3 (linux-gnu)
6^Apache-HttpClientApache-HttpClient/4.2.3 (java 1.5)
7^Apache-HttpClientApache-HttpClient/4.3.3 (java 1.5)
8^Apache-HttpClientApache-HttpClient/4.5.3 (Java/1.8.0_121)
9^Apache-HttpClientApache-HttpClient/4.5.2 (Java/1.8.0_181)
10^Apache-HttpClientApache-HttpClient/4.3.1 (java 1.5)
11^Apache-HttpClientApache-HttpClient/4.3.5 (java 1.5)
12^Apache-HttpClientApache-HttpClient/4.4.1 (Java/1.8.0_65)
13^Apache-HttpClientApache-HttpClient/4.5.2 (Java/1.8.0_65)
14^Apache-HttpClientApache-HttpClient/4.5.2 (Java/1.8.0_151)
15^Apache-HttpClientApache-HttpClient/4.5.2 (Java/1.8.0_161)
16^Apache-HttpClientApache-HttpClient/4.2.5 (java 1.5)
17^Apache-HttpClientApache-HttpClient/4.5.10 (Java/1.8.0_201)
18^Apache-HttpClientApache-HttpClient/4.5.7 (Java/11.0.3)
19^Apache-HttpClientApache-HttpClient/4.5.3-SNAPSHOT (Java/1.8.0_152)
28^LCCLCC (+
29007ac9 CrawlerMozilla/5.0 (compatible; 007ac9 Crawler; (Germany) web directory link checking123spider-Bot (Version: 1.02, powered by - UK web directory192.comAgent
321st ZipCommander Net - IE based browser1st ZipCommander (Net) - (China) robot ( By +( A Real artificial intelligence search engine China)
342Bone online link checker2Bone_LinkChecker/1.0 libwww-perl/5.64 CMS Detector (
36360SpiderMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider
37361SpiderMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider(compatible; HaosouSpider;
38362SpiderMozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 QIHU 360SE; 360Spider
39363SpiderMozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; )Firefox/; 360Spider
40364SpiderMozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:; 360Spider
41365SpiderMozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv: Firefox/ 360Spider;
42366SpiderMozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv: Gecko/20070312 Firefox/; 360Spider
43367SpiderMozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider
44368SpiderMozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider;
45369SpiderMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 360Spider
464Anything LinkChecker v2.0
4780LegsMozilla/5.0 (compatible; 008/0.83; Gecko/2008032620
48A Passion for Jazz music related search robotBebopBot/2.5.1 ( crawler ) robot - now Jet2Web SearchA-Online Search
50A1 Keyword Research - search engine and keyword optimization softwareA1 Keyword Research/1.0.2 (+ miggibot/2007.03.27
52Aardvark web crawler for Sun's Blog recommendationsaardvark-crawler
53Abacho / robotAbachoBOT
54Abacho / robotAbachoBOT (Mozilla compatible)
55ABCdatos - Castilian program & tutorial directoryABCdatos BotLink/
56Aberja Hybridsuchmaschine (Germany) link checkingAberja Checkomat
57AboundexAboundex/0.3 (
58AboundexAboundex/0.2 (
59About robotAbout/0.1libwww-perl/5.47
60acapbotMozilla/5.0 (compatible;acapbot/0.1.;treat like Googlebot)
61acapbotMozilla/5.0 (compatible;acapbot/0.1;treat like Googlebot)
62Accela Technology RSS feed crawlerAccelatech RSSCrawler/0.4
63Accoona Search robotaccoona
64Accoona Search robotAccoona-AI-Agent/1.1.1 (crawler at accoona dot com)
65Accoona Search robotAccoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)
66Ace Explorer - IE based browserAce Explorer
67Ackerm search robotAck (
68Acoi picture finder robotAcoiRobot search (Germany) robotAcoon Robot v1.50.001 search (Germany) robotAcoon Robot v1.52 ( search (Germany) robotAcoon-Robot 4.0.x.[xx] ( search (Germany) robotAcoon-Robot v3.xx ( and
73Acorn Search ProjectAcorn/Nutch-0.9 (Non-Profit Search Engine;; acorn at isara dot org)
74Activeworlds 3D homepage browserActiveworlds
75Activeworlds 3D homepage browserActiveWorlds/3.xx (xxx)
76Ad Muncher - banner killerAd Muncher v4.xx.x
77Ad Muncher - banner killerAd Muncher v4x Build xxxxx
78adbeat_botMozilla/5.0 (compatible; adbeat_bot;;
79AddSearchBotMozilla/5.0 (compatible; AddSearchBot/0.9; +; robot
81ADmantXADmantX Platform Semantic Analyzer - ADmantX Inc. - -
82AdsBot GoogleMozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +
83AdsBot-Google-MobileMozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +
84AdsBot-Google-MobileMozilla/5.0 (Linux; Android 5.0; SM-G920A) AppleWebKit (KHTML, like Gecko) Chrome Mobile Safari (compatible; AdsBot-Google-Mobile; +
86AdsBot-Google([^-]|$)AdsBot-Google (+
87adscannerMozilla/5.0 (compatible; adscanner/)
89AdvBotMozilla/5.0 (compatible; AdvBot/2.0; +
90Aesop robotAESOP_com_SpiderMan
91Agada search (Germany) robotagadine/1.x.x (+
93Ahrefs(Bot|SiteAudit)Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +
94Ahrefs(Bot|SiteAudit)Mozilla/5.0 (compatible; AhrefsBot/5.2; +
95Ahrefs(Bot|SiteAudit)Mozilla/5.0 (compatible; AhrefsBot/6.1; News; +
96Ahrefs(Bot|SiteAudit)Mozilla/5.0 (compatible; AhrefsBot/6.1; +
97Ahrefs(Bot|SiteAudit)Mozilla/5.0 (compatible; AhrefsBot/5.2; News; +
98Ahrefs(Bot|SiteAudit)Mozilla/5.0 (compatible; AhrefsSiteAudit/5.2; +
99AideRss - Postrank RSS and Blog filteringAideRSS/1.0 (
100aiHitBotMozilla/5.0 (compatible; aiHitBot/2.9; +
101Akregator news feed reader for KDEAkregator/1.2.9; librss/remnants
102Aladin robotAladin/3.324
103Aleksika Danmark - Search engine optimization spiderAleksika Spider/1.0 (+
104Alertinfo - French version of Feedreader 3.xxAlertInfo 2.0 (Powered by Newsbrain)
105Alexaia_archiver (+;
106Alexa robotAllesklar/0.1 libwww-perl/5.46
108Alligator download managerAlligator 1.31 (
109AlphaBotMozilla/5.0 (compatible; AlphaBot/3.2; +
110Altavista robotAltaVista Intranet V2.0 AVS EVAL
111Altavista robotAltaVista Intranet V2.0 Compaq Altavista Eval
112Altavista robotAltaVista Intranet V2.0
113Altavista robotAltaVista V2.0B
114Altavista robotAVSearch-3.0(AltaVista/AVC)
115Altavista robot ??AV Fetch 1.0
116Amaya - W3C's Editor/Browseramaya/x.xx libwww/x.x.x
117Amazon CloudFrontAmazon CloudFront robot for checking their affiliate sitesamzn_assoc robot for checking their affiliate sitesaranhabot
120Amfibi Search robotAmfibiBOT
121Amfibi Search robotAmfibibot/0.06 (Amfibi Web Search;;
122Amfibi Search robotAmfibibot/0.07 (Amfibi Robot;;
123Amfibi Search robotCabot/Nutch-0.9 (Amfibi's web-crawling robot;;
124Amfibi Search robotCabot/Nutch-1.0-dev (Amfibi's web-crawling robot;;
125amibot - Amidalla search engine robot (62.241.33.xx)amibot
126Amiga Miami TCP StackAmiTCP Miami (AmigaOS 2.04)
127AndersPinkBotMozilla/5.0 (compatible; AndersPinkBot/1.0; +
128Annomille Italian historical oriented robotAnnoMille spider 0.1 alpha -
129annotate Google - Firefox extension for annotating Google search resultsannotate_google;
130AnswerBus natural language search using COLLATE technologyAnswerBus (
131AnswerChase search toolAnswerChase PROve x.0
132AnswerChase search toolAnswerChase x.0
133ANT Fresco BrowserANTFresco/x.xx
134Antibot (discontinued) robotantibot-V1.1.5/i586-linux-2.2
135AnyApexBot 1.0Mozilla/5.0 (compatible; AnyApexBot/1.0; +
136AnyEventMozilla/5.0 (compatible; U; AnyEvent-HTTP/2.24; +
137Anzwers (Yahoo) Australia robotAnzwersCrawl/2.0 (;Engine)
138AOLMozilla/5.0 (compatible; MSIE 9.0; AOL 9.7; AOLBuild 4343.19; Windows NT 6.1; WOW64; Trident/5.0; FunWebProducts)
139AOLMozilla/4.0 (compatible; MSIE 8.0; AOL 9.7; AOLBuild 4343.19; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E)
140AOLMozilla/4.0 (compatible; MSIE 7.0; AOL 9.7; AOLBuild 4343.19; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E)
141AOLMozilla/4.0 (compatible; MSIE 8.0; AOL 9.7; AOLBuild 4343.19; Windows NT 5.1; Trident/4.0; GTB7.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
142AOLMozilla/4.0 (compatible; MSIE 8.0; AOL 9.7; AOLBuild 4343.21; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E)
143AOLMozilla/4.0 (compatible; MSIE 8.0; AOL 9.7; AOLBuild 4343.27; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
144AperciteMozilla/5.0 (compatible; Apercite; +
145Apexoo Search spiderApexoo Spider 1.x
146APIs-GoogleAPIs-Google (+
147Aport robotAport
148AppEngine-GoogleAppEngine-Google; (+; appid: example)
149AppInsightsMozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; AppInsights)
150ApplebotMozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; +
151ApplebotMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Applebot/0.3; +
152ApplebotMozilla/5.0 (compatible; Applebot/0.3; +
153ApplebotMozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +
154ApplebotMozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
155Araby search - ArabiaArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4;;)
156archive.org_botMozilla/5.0 (compatible; special_archiver/3.1.1 +
157archive.org_botMozilla/5.0 (compatible; archive.org_bot +
158archive.org_botMozilla/5.0 (compatible; archive.org_bot +
159archive.org_botMozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 +
160archive.org_botMozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +
161archive.org_botMozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +
162ArchiveBotArchiveTeam ArchiveBot/20170106.02 (wpull 2.0.2)
163Arikus inContext search engine softwareArikus_Spider
164Art of Computing blog search projectblogsearchbot-pumpkin-3
165Asaha search robot (Turkey)ASAHA Search Engine Turkey V.001 (
166ASAHINA Antenna information detecting agentAsahina-Antenna/1.x
167ASAHINA Antenna information detecting agentAsahina-Antenna/1.x ( ;
168Ask 24x Info
169Ask About Oil - Petroleum related search ( using NutchAskAboutOil/0.06-rcp (Nutch;;
170askEd! / Inferret search (Japan) robot using Nutchasked/Nutch-0.8 (web crawler;; epicurus at gmail dot com)
171ASPSeek search engine softwareASPSeek/1.2.x
172ASPSeek search engine softwareASPSeek/1.2.xa
173ASPSeek search engine softwareASPseek/1.2.xx
174ASPSeek search engine softwareASPSeek/1.2.xxpre
175ASPSeek search engine software -Yahoo-Inc. / Telecom Canada robotASPSeek/1.2.5
176Associative Sort robotASSORT/0.10
177AT&T/Fast Search robot for FirstGov (U.S.Government) Search -
178Atlocal local business search robotAtlocalBot/1.1 +(
179Atomic Email Hunter email extracing and harvestingAtomic_Email_Hunter/4.0
180Atomz robotAtomz/1.0
181atSpider (ceased) email harvester / spambotatSpider/1.0
182Attentio social media monitoring and analysingAttentio/Nutch-0.9-dev (Attentio's beta blog crawler;;
183Audiovox SMT5600 (AT&T) Smartphone mobile phone browserAUDIOVOX-SMT5600
184Augurnet Swiss (was search robotaugurfind
185Augurnet Swiss (was search robotaugurnfind V-1.x
186Auto Email Pro Email harvesterautoemailspider
187Autowebdir - The Automatically Generated Web Directoryautowebdir 1.1 (
188Avant Browser - IE based browserAdvanced Browser (
189Avant Browser - IE based browserAvant Browser (
190AWeb Amiga browserAmiga-AWeb/3.4.167SE
191AWeb Amiga browserAWeb
192awesomecrawlerMozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.5 Safari/537.22 +awesomecrawler
193Axada search Germany robotaxadine/ (Axadine Crawler;; )
195Axmo search robotAxmoRobot - Crawling your site for better indexing on search engine.
196Ay-Up geo sync search robotFastBug
197Ayell Euronet business directory robotEurobot/1.0 (
198Azureus Java BitTorrent ClientAzureus 2.x.x.x
199B2B BotB2B Bot
200Babaloo search robot (Slovenia)BabalooSpider/1.3 (BabalooSpider;;
201BaBoom Web Portal (ODP) robot (66.98.254.xx)BaboomBot/1.x.x (+
202BackStreet Browser - Offline browser / website downloaderBackStreet Browser 3.x
203Baidu search (Japan) crawlerBaiduspider+(+
204Baidu search (Japan) image crawlerBaiduImagespider+(+
205Baidu spidering engine - used by diff. IPsBaiDuSpider
206Baidu spidering engine - used by diff. IPsBaiduspider+(+
207Baidu spidering engine - used by diff. IPsBaiduspider+(+
214BaiduspiderMozilla/5.0 (compatible; Baiduspider-render/2.0; +
216BaiduspiderMozilla/5.0 (compatible; Baiduspider/2.0; +
217Balihoo - Search Engine for Advertising MediaBloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots meta tags ;; robot at balihoo dot com)
218Balihoo - Vertical search engine crawler (beta)Balihoo/Nutch-1.0-dev (Crawler for search engine - obeys robots.txt and robots meta tags ;; robot at balihoo dot com)
219Barca Pro email & PIM softwareBarca/2.0.xxxx
220Barca Pro email & PIM softwareBarcaPro/1.4.xxxx
221Bark[rR]owlerBarkrowler/0.5.1 (experimenting / debugging - sorry for your logs ) - -- based on BuBiNG
222Barrahome crawlerBarraHomeCrawler (
223Basic Web Hacking 3 fake user-agent from Hellbound Hackers challengesbwh3_user_agent
224BazQuxMozilla/5.0 (compatible; BazQux/2.4; +; 1 subscribers)
225BB4 network monitoringBigBrother/1.6e
226BD-Brandprotect copyright infringement crawlerBPImageWalker/2.0 (
227BDCbotMozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; + AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
228Bdncentral Sitesearch robotBDNcentral Crawler v2.3 [en] ( (X11; I; Linux 2.0.44 i686)
229beammachine web directory (Germany) link checkingBeamMachine/0.5 (dead link remover of
230Beauty robot for Cosmoty - German beauty and wellness searchbeautybot/1.0 (+
231BehloolBotMozilla/5.0 (compatible; BehloolBot/beta; +
232Big Brother link checking toolBig Brother (
233BigClique Search robotBigCliqueBOT/1.03-dev (bigclicbot;;
235Biglotron search (France) robotBIGLOTRON (Beta 2;GNU/Linux) search ( Internet Spider;;
237Bilbo - web frontend for the Nessus Security ScannerBilbo/2.3b-UNIX (Beta) search robot - TurkeyBilgiBetaBot/0.8-dev ( (Beta) ;; (Beta) search robot - TurkeyBilgiBot/1.0(beta) (; bilgi at bilgi dot com)
240BingbotMozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +
241BingbotMozilla/5.0 (compatible; adidxbot/2.0; +
242bingbotMozilla/5.0 (compatible; bingbot/2.0;
243bingbotMozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +
244bingbotMozilla/5.0 (seoanalyzer; compatible; bingbot/2.0; +
245bingbotMozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +
246bingbotMozilla/5.0 (compatible; bingbot/2.0; +
247bingbotMozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0;
248BingbotMozilla/5.0 (compatible; bingbot/2.0
249bingbotMozilla/5.0 (compatible; bingbot/2.0; +
250bingbotMozilla/5.0 (compatible; adidxbot/2.0; +
251BingbotMozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko BingPreview/1.0b
252bingbotMozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0;
253bingbotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; + Safari/537.36
254Bingbotmsnbot-media/1.1 (+
255bingbotMozilla/5.0 (compatible; adidxbot/2.0;
256bingbotMozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +
257BingbotMozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +
258BingbotMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
259BingbotMozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +
260Bingbotmsnbot/2.0b (+
261bingbotMozilla/5.0 (compatible; bingbot/2.0; + SitemapProbe
262BingbotMozilla/5.0 (compatible; bingbot/2.0; +
263BingbotMozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; bingbot/2.0; +
264BingbotMozilla/5.0 (compatible; bingbot/2.0 +
265BingPreview\/Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
266BingPreview\\/Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;WOW64;Trident/5.0;BingPreview/1.0b)
267BingPreview\\/Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko
268BingPreview\\/Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0;WOW64;Trident/6.0;BingPreview/1.0b)
269BingPreview\\/Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b
270BingPreview\\/Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
271binlarbinlar_2.6.3 binlar2.6.3@unspecified.mail
272binlarbinlar_2.6.3 binlar_2.6.3@unspecified.mail
273binlarbinlar_2.6.3 larbin2.6.3@unspecified.mail
275binlarbinlar_2.6.3 test@mgmt.mic
276Bisnisseek (was robotCustom Spider /1.0
277Bitacle Blog Search Archive robotBitacle bot/1.1
278Bitacle Blog Search Archive robotBitacle Robot (V:1.0;) (
281bitlybotbitlybot/3.0 (+
282BitTorrent Search Engine btbot robotBTbot/0.x (+
283Biyubi Navigator - Mexican browser for Fenix OSBiyubi/x.x (Sistema Fenix; G11; Familia Toledo; es-mx)
284Blackberry Wireless Internet browser via Google WAP ProxyBlackBerry7520/4.0.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/ UP.Link/ (Google WAP Proxy/1.0)
285BlackboardBlackboard Safeassign
286BlackWidow web site scanner / downloading toolBlackWidow
287Blaiz Enterprises RawGrunt searchBlaiz-Bee/1.0 (+
288Blaiz Enterprises RawGrunt searchBlaiz-Bee/2.00.8222 (BE Internet Search Engine
289Blaiz Enterprises RawGrunt searchBlaiz-Bee/2.00.xxxx (+
290BLEXBot\/Mozilla/5.0 (compatible; BLEXBot/1.0; +
291BLEXBot\\/Mozilla/5.0 (compatible; BLEXBot/1.0; +
292Blitzsuche Germany
293Blitzsuche Germany (Mozilla compatible)
294BlockNote web page editorBlockNote.Net
295BlogBridge RSS readerBlogBridge 2.13 (
296blogdex robot from MIT.eduBlogBot/1.x
297Bloglines article searchBloglines Title Fetch/1.0 (
298Bloglines graphics crawlerBloglines-Images/0.1 (
299Bloglines news crawlerBloglines/3.1 (
300blogmuraBotblogmuraBot (+
301BlogsNow realtime link tracker robotBlogsNowBot, V 2.01 (+
302BlogVipe news and Blog crawler (Netherlands)BlogVibeBot-v1.1 (
303blogWatcher robot from Okumura Group Tokyo ( (
304Blooby search (beta) robotBloobyBot
306Bluefish HTML-editor for Linuxbluefish 0.6 HTML editor
307bnf.fr_botMozilla/5.0 (compatible; bnf.fr_bot; +
308Bobby web accessibility desktop testing toolBobby/4.0.x RPT-HTTPClient/0.3-3E
309Boitho search (Norway) robot via 80.202.212.xx / (
310Boitho search (Norway) robot via 80.202.212.xx /
311Boitho search (Norway) robot via 80.202.212.xx / (
312BomboraBotMozilla/5.0 (compatible; BomboraBot/1.0; +
313Bookdog - Mac bookmark managerBookdog/x.x
314Bookmark Base bookmark managerBookmarkBase(2/;
315Bookmark Buddy - favorite bookmark managerBookmark Buddy bookmark checker (
316Bookmark Express bookmark managerBMLAUNCHER (+
319botifyMozilla/5.0 (compatible; botify;
320BoxcarBotMozilla/5.0 (compatible; BoxcarBot/1.1;
321Brandimensions Brand Protection robotBDFetch
322BrandVerityMozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:41.0) Gecko/20100101 Firefox/55.0 BrandVerity/1.0 (
323BravoBrian bSTOP parental controlBravoBrian
324BravoBrian bSTOP parental Agent Detector
325BrightCloud web filtering for classifying websitesBrightCrawler (
327BUbiNGBUbiNG (+
328Buck\/Buck/2.2; (+
329BuildCMS crawler - market monitoring project of BuildCMSBuildCMS crawler (
330Bulkfeeds: RSS directory link checkingBulkfeeds/r1752 (
331BullsEye/Intelliseek robotBullsEye
332BurstFind search crawler (64.34.172.xx)BurstFindCrawler/1.1 (;;
333Buscaplus (Spain) robotBuscaplus Robi/1.0 ( robotbdcindexer_2.6.2 (research@bdc)
335BuzzbotBuzzbot/1.0 (Buzzbot;;
336buzzbotBuzzbot/1.0 (Buzzbot;;
337BytespiderMozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.4141.1682 Mobile Safari/537.36; Bytespider
338BytespiderMozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2988.1545 Mobile Safari/537.36; Bytespider
339BytespiderMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.5527.1507 Mobile Safari/537.36; Bytespider
340BytespiderMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.6023.1635 Mobile Safari/537.36; Bytespider
341BytespiderMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.4944.1981 Mobile Safari/537.36; Bytespider
342BytespiderMozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.4454.1745 Mobile Safari/537.36; Bytespider
343BytespiderMozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.7990.1979 Mobile Safari/537.36; Bytespider
344BytespiderMozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.3478.1649 Mobile Safari/537.36; Bytespider
345BytespiderMozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.5267.1259 Mobile Safari/537.36; Bytespider
346BytespiderMozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.7597.1164 Mobile Safari/537.36; Bytespider;
347BytespiderMozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.2268.1523 Mobile Safari/537.36; Bytespider
348BytespiderMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2576.1836 Mobile Safari/537.36; Bytespider
349BytespiderMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.9681.1227 Mobile Safari/537.36; Bytespider
350BytespiderMozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.3754.1902 Mobile Safari/537.36; Bytespider
351BytespiderMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.3248.1547 Mobile Safari/537.36; Bytespider
352BytespiderMozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.9038.1080 Mobile Safari/537.36; Bytespider
353BytespiderMozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.5216.1326 Mobile Safari/537.36; Bytespider
354BytespiderMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.4022.1033 Mobile Safari/537.36; Bytespider
355BytespiderMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.3613.1739 Mobile Safari/537.36; Bytespider
356CaliperbotCaliperbot/1.0 (+ Favorites checking toolFavorites Checking (
358CapsuleCheckerCapsuleChecker (
359Carnegie Mellon School robot/link checking ?billbot
360Carnegie Mellon University WebBOTCarnegie_Mellon_University_Research_WebBOT-->PLEASE READ-->
361Carnegie Mellon University WebBOTCarnegie_Mellon_University_WebCrawler search & web directory (Germany)Catall Spider
363CC Metadata ScaperCC Metadata Scaper
364CCBotCCBot/2.0 (
365CCBotCCBot/2.0 (
366CERN Line Mode BrowserCERN-LineMode/2.15
367cg-eye CGI checkercg-eye interactive
368changedetectionMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; )
369changedetectionMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; )
370Charon Amiga download managerCharon/1.x (Amiga)
371Chat Catcher blog monitoring robotChat Catcher/1.0
372check_httpcheck_http/v2.2.1 (nagios-plugins 2.2.1)
373Checkbot link validationCheckbot/1.xx LWP/5.xx
374Checklinks - Perl link checkerCheckLinks/1.x.x
375CheckMarkNetwork\/CheckMarkNetwork/1.0 (+
376CheckWeb link validationCheckWeb
377Chilkat HTTP component user-agentChilkat/1.0.0 (+
378Chitika Inc. Blog advertisingChitika ContentHit 1.0 Christcrawler (was 2.0
380Chrome-LighthouseMozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3694.0 Mobile Safari/537.36 Chrome-Lighthouse
381Chrome-LighthouseMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3694.0 Safari/537.36 Chrome-Lighthouse
382Chrome-LighthouseMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36(KHTML, like Gecko) Chrome/69.0.3464.0 Safari/537.36 Chrome-Lighthouse
383Chrome-LighthouseMozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36(KHTML, like Gecko) Chrome/69.0.3464.0 Mobile Safari/537.36 Chrome-Lighthouse
384CHttpClient - C++ class using WinInetCHttpClient by Open Text Corporation
385CipinetBot -Cipinet Search Engine Web CrawlerCipinetBot (
386Cisco Content EngineCE-Preload
387Cityreview regional search (Germany) link checkingCityreview Robot (+
388CJB Net anonymous socks proxy service (216.194.70.x)CJB.NET Proxy
389Claria (ex Gator) SearchScout robot (64.152.73.xx)ClariaBot/1.0
390Claria (ex Gator) SearchScout robot (64.152.73.xx)Diamond/x.0
391Claria (ex Gator) SearchScout robot (64.152.73.xx)DiamondBot
392Claymont Search
393ClickagyClickagy Intelligence Bot v2
394Cliqzbot\/Mozilla/5.0 (compatible; Cliqzbot/2.0; +
395Cliqzbot\\/Mozilla/5.0 (compatible; Cliqzbot/2.0; +
396Cliqzbot\\/Mozilla/5.0 (compatible; Cliqzbot/0.1 +
397Cliqzbot\\/Cliqzbot/0.1 (+
398Cliqzbot\\/Cliqzbot/0.1 (+
399Cliqzbot\\/Mozilla/5.0 (compatible; Cliqzbot/1.0 +
400CloudFlare-AlwaysOnlineMozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; + AppleWebKit/534.34
401Clush clustered search robotClushbot/2.x (+
402Clush clustered search robotClushbot/3.x-BinaryFury (+
403Clush clustered search robotClushbot/3.xx-Ajax (+
404Clush clustered search robotClushbot/3.xx-Hector (+
405Clush clustered search robotClushbot/3.xx-Peleus (+
406COAST Webmaster - Web management and maintenance softwareCOAST WebMaster Pro/4.x.x.xx (Windows NT)
407coccocMozilla/5.0 (compatible; coccocbot-image/1.0; +
408coccocMozilla/5.0 (compatible; coccoc/1.0; +
409coccocMozilla/5.0 (compatible; coccoc/1.0; +
410coccocMozilla/5.0 (compatible; imagecoccoc/1.0; +
411coccocMozilla/5.0 (compatible; image.coccoc/1.0; +
412coccocMozilla/5.0 (compatible; imagecoccoc/1.0; +
413coccocMozilla/5.0 (compatible; coccocbot-web/1.0; +
414coccoccoccoc/1.0 (
415coccoccoccoc/1.0 (
416coccoccoccoc/1.0 ()
417Cocoa (social bookmarks manager) client for Mac OS (v36) (Mac OS X;
418Cogent Search Bot from Cogent Software Solutions for unknown purposesCogentbot/1.X (+
419Cold Fusion server used by Bookmark Tracker - online favourites managmentColdFusion (
420Cold Fusion server used by various IPs i.e.: - NetWORLD web catalogue link checkingColdFusion
421collection@infegy.comMozilla/5.0 (compatible) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36
422Combine harvesting & indexing robotcombine/0.0
423Combine harvesting & indexing robotCombine/2.0
424Combine harvesting & indexing robotCombine/3
425Combine harvesting & indexing robotCombine/x.0
426cometrics Web Content Mining solution - Germanycometrics-bot,
427Cometsystems Comet Search robot via Findwhat (now Miva)Crawler (
428Commision Junction link checking spiderCJ Spider/
429CommonCrawl Foundation search crawlerCCBot/1.0 (+
430Companybook-CrawlerCompanybook-Crawler (+
431Comperio Web Miner based on Fast ESPFAST Enterprise Crawler 6 used by Comperio AS (
432ComRite Chinese Search Engine for Oversea Web Sites (69.248.26.xx)Comrite/0.7.1 (Nutch;;
433ContentSmartz e-mail harvesting toolsContentSmartz
434ContextAd BotMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;.NET CLR 1.0.3705; ContextAd Bot 1.0)
435contxbotMozilla/5.0 (compatible;contxbot/1.0)
436converaConveraCrawler/0.9e (+
437Converas RetrievalWare Internet SpiderCrawlConvera0.1 (
438Converas RetrievalWare Internet Spider (63.241.61.x)Convera Internet Spider V6.x
439Converas RetrievalWare Internet Spider (63.241.61.x)ConveraCrawler/0.2
440Converas RetrievalWare Internet Spider (63.241.61.x)ConveraCrawler/0.9d (+
441Converas RetrievalWare Internet Spider (63.241.61.x)ConveraMultiMediaCrawler/0.1 (+
442Cooliris photo and video browserCooliris/1.5 CFNetwork/459 Darwin/10.0.0d3
443Coral Content Distribution NetworkCoralWebPrx/0.1.1x (See
444CoreMedia player on Apple iPhoneApple iPhone v1.1.4 CoreMedia v1.0.0.4A102
445Cosmix project crawler (204.14.48.x /
446Cosmix project crawler (204.14.48.x / - local French directory link checkingCoteoNutchCrawler/Nutch-0.9 (info [at] coteo [dot] com) robotCougarSearch/0.x (+
449Covac Arachnid Web CrawlerArachBot
450Covac Arachnid Web CrawlerCovac TexAs Arachbot
451CoverScout for iTunes - CD cover search toolCoverScout%203/3.0.1 CFNetwork/339.5 Darwin/9.5.0 (i386) (iMac5,1) / Abacho robotCrawler / Abacho robotCrawler V 0.2.x
454crawler4jcrawler4j (
455crawler4jcrawler4j (
456Crawlly Beta search - Germany ( (Crawllybot; +;
457Creatice Commons using Nutch open source robotCreativeCommons/0.06-dev (Nutch;;
458Cricket A100 cell phone browserCricket-A100/1.0 UP.Browser/ (GUI) MMP/2.0
459Croccrawler robotCrocCrawler vx.3 [en] ( (X11; I; Linux 2.0.44 i686)
460CrunchBotCrunchBot/1.0 (+
461CSE HTML Validator for WindowsCSE HTML Validator Professional (
462Cuam - IE based browserCuam Ver0.050bx
463Cuasar (Spain) music / ringtone search spiderCuasarbot/0.9b
464Curl file transferring toolcurl/7.10.x (i386-redhat-linux-gnu) libcurl/7.10.x OpenSSL/0.9.7a ipv6 zlib/1.1.4
465Curl file transferring toolcurl/7.7.x (i386--freebsd4.3) libcurl 7.7.x (SSL 0.9.6) (ipv6 enabled)
466Curl file transferring toolcurl/7.8 (i686-pc-linux-gnu) libcurl 7.8 (OpenSSL 0.9.6)
467Curl file transferring toolcurl/7.9.x (win32) libcurl 7.9.x
468CurryGuide UK link check robotCurryGuide SiteScan 1.1
469Custo web site spidering tool (link checking)Custo x.x (
470cXensebotcXensebot/1.1a user robot / faked user agent ?Bot
472Cyberdog Mac Browser (was (Macintosh; 68k)
473CyberPatrolCyberPatrol SiteCat Webbot (
474CyberPatrol LLC robot for web filtering softwareCyberPatrol SiteCat Webbot (
475CyberSpyder Link Test softwareCyberSpyder Link Test/2.1.12 (
476Cydral image & site search spiderCydralSpider/1.x (Cydral Web Image Search;
477Cydral image & site search spiderCydralSpider/3.0 (Cydral Image Search;
478DareBoostMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36 DareBoost
479DataBecker Bilder Sauger (discontinued) web graphics downloaderDer große BilderSauger 2.00u
481datagnionbotdatagnionbot (+
482DatanyzeMozilla/5.0 (X11; Datanyze; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
483DataparkSearch open source search engineDataparkSearch/4.47 (+
484DataparkSearch open source search engineDataparkSearch/4.xx (
485Dataprovider.comMozilla/5.0 (compatible;
486DataSpear Directory robot (24.109.29.xx)DataSpear/1.0 (Spider;;
487DataSpear Directory robot (24.109.29.xx)DataSpearSpiderBot/0.2 (DataSpear Spider Bot;;
488Daum\/Mozilla/5.0 (compatible; Daum/4.1; +
489DAUMOA - RSS search robot of DaumELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +
490Daypop blog - weblog - online mag search spiderdaypopbot/0.x
491dbDig search enginedbDig(
492DC-Sakura download managerDC-Sakura/x.xx
493dcrawldcrawl/1.0 XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36 XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
498deepak-USC/ISI robot from USC/Information Science Institutedeepak-USC/ISI
499Deepindex robotDeepIndex
500Deepindex robotDeepIndex (
501Deepindex robot (via Canada) picture robot for thumbnail preview via Mozilla/5.0 (compatible; Konqueror/3.4; FreeBSD) KHTML/3.4.2 (like Gecko)
503DeleGate application level gateway / proxy serverDeleGate/9.0.5-fix1
504Denmex Websearch robot/link checkingDenmex websearch (
505DeuSu\/Mozilla/5.0 (compatible; DeuSu/0.1.0; +
506DeuSu\\/Mozilla/5.0 (compatible; DeuSu/0.1.0; +
507DeuSu\\/Mozilla/5.0 (compatible; DeuSu/5.0.2; +
508DevComponents HTMLDocument Class Library for Visual HtmlDocument Object
509DiaGem Japan web crawlerDiaGem/1.1 (
510Die Kraehe Meta-Search-Engine (Germany) link checking-DIE-KRAEHE- META-SEARCH-ENGINE/1.1
511Diffbot\/Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +
512Diffbot\\/Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +
513Digg DeeperDigg Deeper/v1 (
514Diggit! robotDigger/1.0 JDK/1.3.0rc3
515Digincore botMozilla/5.0 (compatible; Digincore bot; for rules and instructions.)
516Dillo Web BrowserDillo/0.8.5-i18n-misc
517Dillo Web BrowserDillo/0.x.x
518disastrous - a link checker based on Pythondisastrous/1.0.5 (running with Python 2.5.1;;
519DISCo Pump offline browser / website ripperDISCo Pump x.x
520discobotMozilla/5.0 (compatible; discobot/2.0; +
521discobotMozilla/5.0 (compatible; discobot/1.0; +
522discobotmozilla/5.0 (compatible; discobot/1.1; +
523DiscordbotMozilla/5.0 (compatible; Discordbot/2.0; +
525Ditto picture search robotDittoSpyder Movie Find robotFlickBot 2.0 RPT-HTTPClient/0.3-3
527DLC Perl HTTP link checkingDeadLinkCheck/0.4.0 libwww-perl/5.xx
528DNS Right - Online DNS WebBot Link Ckeck Tool. Report abuse to:
529DNS Right - Online DNS Web Server IIS vs Apache Survey. See Results at
530DnyzBotMozilla/5.0 (compatible; DnyzBot/1.0)
531DocZilla - Mozilla-based SGML/XML/HTML- browserDocZilla/1.0 (Windows; U; WinNT4.0; en-US; rv:1.0.0) Gecko/20020804
532Domain Re-Animator BotDomain Re-Animator Bot ( -
533domaincrawlerCipaCrawler/3.0 (;
534DomainStatsBotDomainStatsBot/1.0 (
535Donut P - Japanese IE based browserDonutP; Windows98SE
536dotbotMozilla/5.0 (compatible; DotBot/1.1;,
538Download Accelerator Plus download managerDAP x.x
539Download Demon/Netzip download managerDownload Demon/3.x.x.x
540Download Direct download managerDownloadDirect.1.0
541Download Druid IE plugin download managerDownload Druid 2.x
542Download Express download managerDownload Express 1.0
543Download Master download managerDownload Master
544Download Ninja download manager (Japan)Download Ninja 3.0
545Download Wonder download managerDownload Wonder
546Download-Tipp Germany link checkingDownload-Tipp Linkcheck (
547Download-Tipp Germany robotEyeCatcher (
548download.exe .NET based downloading toolDownload.exe(1.1) (+
549Downloadaccelerator download managerDA 3.5 (
550Downloadaccelerator download managerDA 4.0
551Downloadaccelerator download managerDA 4.0 (
552Downloadaccelerator download managerDA 5.0
553Downloadaccelerator download managerDA 7.0
554Dr.WEB online virus scannerDr.Web (R) online scanner:
555Dragonfly CMS - Open Source content management systemDragonfly File Reader
556DreamCast DreamPassport browser(DreamPassport/3.0; isao/MyDiGiRabi)
557Drecom Japan ( - Blog search ??Drecombot/1.0 (
558Drew Curtis' link link verifier
560Drupal - open source content management platformDrupal (+
561dt Search Spider softwaredtSearchSpider
562DTAAgent Java object for data collectingDTAAgent
563Duck Duck Go search crawlerDuckDuckBot/1.0; (+
564DuckDuckBotDuckDuckBot/1.0; (+
565DuckDuckBotMozilla/5.0 (compatible; DuckDuckBot-Https/1.1;
566DuckDuckBotDuckDuckBot/1.1; (+
567DuckDuckGo-Favicons-BotMozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; + robotDumbot(version 0.1 beta - robotDumbot(version 0.1 beta - robotDumbot(version 0.1 beta)
571e-Society Project (Japan) crawler (133.9.238.xx)e-SocietyRobot(
572e-Style ISP search (Russia) robot ( 4 (compatible; MSIE 6.0; Windows NT 5.0)
573Earthcom (Czech Republic) search robot (194.108.39.xx) []
574Earthcom (Czech Republic) search robot (194.108.39.xx) []
576eCatch (now Wysigot) offline browsereCatch/3.0 robotEchO!/2.0
578Egoto Search robotEgotoBot/4.8 (+
579eJupiter searcg robot (206.191.49.xx)
580electricmonkMozilla/5.0 (compatible; electricmonk/3.2.0 +
581Elftales crawler for uChoose theme based search (Germany)elfbot/1.0 (+
582ELinks text mode browserELinks (0.x.x; Linux 2.4.20 i586; 132x60)
583ELinks text mode browserELinks/0.x.x (textmode; NetBSD 1.6.2 sparc; 132x43)
584EmailSpider E-mail harvesting softwareEmailSpider
586EmeraldShield spam and web filtration WebBot
587EmeraldShield spam and web filtration WebBot (
588Empas search Korea robot (
589Empyreum Ccubee (Czech) search engine solutionccubee/x.x
590Enabot - Enaball semantic search project crawlerEnaBot/1.x (
591endo - Mac news site and blog aggregatorendo/1.0 (Mac OS X; ppc i386;
592Enfish Personal search toolEnfish Tracker
593Enterprise Search engine software (
594Enterprise Search engine software (
595Enterprise Search engine software (;MSSQL (
596Enterprise Search engine software ( (
597Envolk Web Search robotenvolk/1.7 (+
598Envolk Web Search robotenvolk[ITS]spider/1.6(+
599epicbotMozilla/5.0 (compatible; epicbot; +
600erightMozilla/5.0 (compatible; eright/1.0;
601EroCrawler adult search robotEroCrawler
602ESI Smart-Spider toolkitESISmartSpider - European Internet Portal robotEuripBot/0.x (+ GetFile - European Internet Portal robotEuripBot/0.x (+ GetRobots - European Internet Portal robotEuripBot/0.x (+ PreCheck
606europarchive.orgMozilla/5.0 (compatible; MSIE 7.0 +
607Euroseek spiderArachnoidea (
608Evaal Search Engine robotEvaalSE -
609Eventax event search (Germany)eventax/1.3 (eventax;;
610EveryoneSocialBotMozilla/5.0 (compatible; EveryoneSocialBot/1.0;
611Evolution integrated mail solution Camel TCP stream classCamelHttpStream/1.0
612ExabotMozilla/5.0 (compatible; Exabot/3.0; +
613ExabotMozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
614ExabotMozilla/5.0 (compatible; Exabot/3.0;
615ExabotMozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +
616ExabotMozilla/5.0 (compatible; Alexabot/1.0; +;
617ExabotMozilla/5.0 (compatible; Exabot-Images/3.0; +
618ExabotMozilla/5.0 (compatible; Exabot PyExalead/3.0; +
619exabotMozilla/5.0 (compatible; Alexabot/1.0; +;
620eXact Search Bar for IEExactSearch
621ExactSEEK (Jayde Online) roboteseek-larbin_2.6.2 (
622ExactSEEK (Jayde Online) robotExactSeek Crawler/0.1
623ExactSEEK (Jayde Online) robotexactseek-crawler-2.63 (
624ExactSEEK (Jayde Online) robotexactseek-pagereaper-2.63 (
625ExactSEEK (Jayde Online) robot (
626Exalead (France) search robot (193.47.80.xx)Exabot-Test/1.0
627Exalead (France) search robot (193.47.80.xx)Exabot/2.0
628Exalead (France) search robot (193.47.80.xx)Exabot/3.0
629Exalead (France) search robot (193.47.80.xx)Exalead NG/MimeLive Client (convert/http/0.120)
630Exalead Websearch image crawler (193.47.80.xx)Exabot-Images/1.0
631Excalibur (now Convera) spider softwareExcalibur Internet Spider V6.5.4
632Excite spiderArchitextSpider
633Execrawl software search using nutchExecrawl/1.0 (Execrawl;;
634Exisoftware image grabber and downloading toolExtreme Picture Finder
635exooba crawler for exooba search pre-alpha developmentexooba crawler/exooba crawler (crawler for;; info at exooba dot com)
636exooba crawler for exooba search pre-alpha developmentexooba/exooba crawler (exooba; exooba)
638Expired Domain Sleuth domain name toolExpired Domain Sleuth
639Express Web Pictures image browserExpress WebPictures (
640ExtLinksBotMozilla/5.0 (compatible; ExtLinksBot/1.5 +
641Extractor Pro e-mail collectorExtractorPro
642EZIDEZID (EZID link checker;
643ezoomsMozilla/5.0 (compatible; Ezooms/1.0;
645Facebookfacebookexternalhit/1.1 (+
646Facebookfacebookexternalhit/1.0 (+
647facebookexternalhitfacebookexternalhit/1.0 (+
649Factbites search robotFactbot 1.09 (see
650Factbites search robotfactbot :
651FaEdit Professional - Japanese bookmark managerFaEdit/2.0.x
652FairAd userFairAd Client
653faked user agentAnonymizer/1.1
654FAST Enterprise CrawlerFAST Enterprise Crawler 6 used by Schibsted (
655FAST Enterprise CrawlerFAST Enterprise Crawler 6 / Scirus;
656Fast Enterprise Crawler (66.151.181.xx) for Scirus scienctific information searchFAST Enterprise Crawler 6 / Scirus;
657FAST Enterprise Crawler used by WWU (University of Muenster - Germany)FAST Enterprise Crawler 6 used by WWU (
659FAST-WebCrawlerFAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;
660FAST-WebCrawlerFAST-WebCrawler/3.7 (atw-crawler at fast dot no;
661FAST-WebCrawlerFAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;
662Fast/Alltheweb crawlerFAST-WebCrawler/2.2.6 (;
663Fast/Alltheweb crawlerFAST-WebCrawler/2.2.7 (;
664Fast/Alltheweb crawlerFAST-WebCrawler/2.2.8 (;
665Fast/Alltheweb crawlerFAST-WebCrawler/3.2 test
666Fast/Alltheweb crawlerFAST-WebCrawler/3.3 (;
667Fast/Alltheweb crawlerFAST-WebCrawler/3.4/Nirvana (;
668Fast/Alltheweb crawlerFAST-WebCrawler/3.4/PartnerSite (;
669Fast/Alltheweb crawlerFAST-WebCrawler/3.5 (atw-crawler at fast dot no;
670Fast/Alltheweb crawlerFAST-WebCrawler/3.6 (atw-crawler at fast dot no;
671Fast/Alltheweb crawlerFAST-WebCrawler/3.6/FirstPage (;
672Fast/Alltheweb crawlerFAST-WebCrawler/3.7 (atw-crawler at fast dot no;
673Fast/Alltheweb crawlerFAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;
674Fast/Alltheweb crawlerFAST-WebCrawler/3.8 (atw-crawler at fast dot no;
675Fast/Alltheweb crawlerFAST-WebCrawler/3.8/Fresh (atw-crawler at fast dot no;
676Fast/Alltheweb crawler (66.151.181.xx)Fast Crawler Gold Edition
677Fast/Alltheweb crawler (66.151.181.xx)FAST Enterprise Crawler 6 (Experimental)
678Fast/Alltheweb crawler (66.151.181.xx)FAST Enterprise Crawler 6 used by Cobra Development (
679Fast/Alltheweb crawler (66.151.181.xx)FAST Enterprise Crawler 6 used by FAST (FAST)
680Fast/Alltheweb crawler (66.151.181.xx)FAST Enterprise Crawler/6 (
681Fast/Alltheweb crawler (66.151.181.xx)FAST Enterprise Crawler/6.4 (helpdesk at
682Fast/Alltheweb crawler (66.151.181.xx)FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0)
683Fast/Alltheweb crawler (66.151.181.xx)FAST MetaWeb Crawler (helpdesk at fastsearch dot com)
684Fast/Alltheweb crawler (66.151.181.xx)Fast PartnerSite Crawler
685Fast/Alltheweb crawler (66.151.181.xx)FastCrawler 3.0.1 (
686Fast/Alltheweb crawler for Australian search (66.151.181.xx)FAST Enterprise Crawler 6 used by Web Crawler (search_comments\at\sensis\dot\com\dot\au)
687Fast/Alltheweb crawler used by SuperPages.comFastSearch Web Crawler for Verizon SuperPages (
688Fast/Alltheweb multimedia crawlerFAST-WebCrawler/2.2.10 (Multimedia Search) (;
689Fast/Alltheweb multimedia crawlerFAST-WebCrawler/3.x Multimedia
690Fast/Alltheweb multimedia crawlerFAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)
691Fastbot search Germany crawler ( crawler beta 2.0 (+
692Favcollector Favicon collecting robotFavcollector/2.0 (
693FavIconizer - IE favorites icons refreshing toolFavIconizer (Germany) search crawler/0.6 (
695Favorites Sweeper bookmark checkerFavorites Sweeper v.2.03
696Favourites managing programBookmark Renewal Check Agent []
697Favourites managing programBookmark Renewal Check Agent [] (Version 2.0beta)
698FaXo Search robot (69.152.89.xx)Faxobot/1.0
700Feed::Find - Syndication feed (RSS/Atom) auto-discoveryFeed::Find/0.0x
701Feed24 news feed and blog
702Feedable beta web based RSS serviceFeedable/0.1 (compatible; MSIE 6.0; Windows NT 5.1)
703FeedDemon RSS readerFeedDemon/2.7 (; Microsoft Windows XP)
704Feedfetcher-GoogleFeedfetcher-Google; (+; 1 subscribers; feed-id=728742641706423)
705FeedForAll RSS feed robotFeedForAll rss2html.php v2
706FeedHub news feed personalization engine powered by mSpokeFeedHub FeedDiscovery/1.0 (
707FeedHub news feed personalization engine powered by mSpokeFeedHub MetaDataFetcher/1.0 (
708Feedjit news feed service favicon crawlerFeedjit Favicon Crawler 1.0
709FeedlyFeedlyBot/1.0 (
710FeedlyFeedly/1.0 (+; like FeedFetcher-Google)
711FeedMap / BlogMap geo coding serviceBlogMap (
712FeedShow online RSS feed readerFeedshow/x.0 (; 1 subscriber)
713FeedShow online RSS feed readerFeedshowOnline (
714FeedspotMozilla/5.0 (compatible; Feedspotbot/1.0; +
715FeedspotbotMozilla/5.0 (compatible; Feedspot/1.0 (+; like FeedFetcher-Google)
716FeedspotbotMozilla/5.0 (compatible; Feedspotbot/1.0; +
718FeedZcollector - Feed (RSS, ATOM and RDF) capturing softwareFeedZcollector v1.x (Platinum)
719FemtosearchBotMozilla/5.0 (compatible; FemtosearchBot/1.0;
720Fetch\/Fetch/2.0a (CMS Detection/Web/SEO analysis tool, see
721Fetch\\/Fetch/2.0a (CMS Detection/Web/SEO analysis tool, see
722FeverFever/1.38 (Feed Parser;; Allow like Gecko)
723Filangy search and bookmark serviceFilangy/0.01-beta (Filangy;;
724Filangy search and bookmark serviceFilangy/1.0x (Filangy;;
725Filangy search and bookmark serviceFilangy/1.0x (Filangy;;
726File Boost Network link (+
727FileHound download managerFileHound x.x\/crawlerMozilla/5.0 (compatible; oBot/2.3.1; +
729filtrbox media content (news) monitoringFiltrbox/1.0
730Find An ISP robotFindAnISP.com_ISP_Finder_v99a
731Find the Best search robotFTB-Bot
732findlinkfindlinks/1.1.6-beta1 (+
733findlinkfindlinks/1.1.5-beta7 (+
734findlinkfindlinks/1.1.3-beta9 (+
735findlinkfindlinks/1.1.3-beta8 (+
736findlinkfindlinks/1.0 (+
737findlinkfindlinks/2.0.4 (+
738findlinkfindlinks/2.0.2 (+
739findlinkfindlinks/2.0.1 (+
740findlinkfindlinks/2.0 (+
741findlinkfindlinks/1.1.6-beta6 (+
742findlinkfindlinks/1.1.6-beta5 (+
743findlinkfindlinks/1.1.6-beta1 (+; YaCy 0.1;
744findlinkfindlinks/1.1.6-beta4 (+
745findlinkfindlinks/1.1.6-beta3 (+
746findlinkfindlinks/1.1.6-beta2 (+
747findlinkfindlinks/2.2 (+
748findlinkfindlinks/2.1.5 (+
749findlinkfindlinks/2.1.3 (+
750findlinkfindlinks/2.0.9 (+
751findlinkfindlinks/2.1 (+
752findlinkfindlinks/2.5 (+
753findlinkfindlinks/2.0.5 (+
754findlinkfindlinks/2.6 (+ (site is offline) / Digital-Integrity robotDIIbot/1.2
756findxbotMozilla/5.0 (compatible; Findxbot/1.0; +
757Finesearch robotFineBot
758Finjan Vital Security Web Appliance security solutionFinjan-prefetch robotFirefly/1.0 robotFirefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)
761Firstsfind Germany robot / link checkingfirstsbot
762Flamingo_SearchEngineFlamingo_SearchEngine (+
763Flaptor information retrieval solutions robotFlapbot/0.7.2 (Flaptor Crawler;; crawler at flaptor period com)
764FlatArts Favorites Icon ToolFLATARTS_FAVICO search serviceFlexum spider search serviceFlexum/2.0
767FlipboardProxyMozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +
768FlipboardProxyMozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +
769FlipboardProxyMozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; +
770FlipboardProxyMozilla/5.0 (compatible; FlipboardProxy/1.2; +
771FlipboardProxyMozilla/5.0 (compatible; FlipboardProxy/1.1; + robot (link ckecking ??)Ceramic Tile Installation Guide (
773FLY Index Metasearch link 1.0/
774Fnoole news crawlerFnooleBot/2.5.2 (+ social search Spider/0.1 beta 1 (
776FollowSite robot - website monitoringFollowSite Bot ( )
777FollowSite robot - website ( )
778Fooky search Scorpionbots robot (;
779Fourelle Venturi proxy serverDual Proxy
780fr-crawlerMozilla/5.0 (compatible; fr-crawler/1.1)
781Free Download Manager (FDM) download acceleratorFDM 1.x
782Free Download Manager (FDM) download acceleratorFDM 2.x
783Free Downloads shareware directory link download-link validator /0.1
784FreeBSD download toolfetch libfetch/2.0 (;
786Frelics backlink checking bot (beta)Frelicbot/1.0 +
787Frequent Finders spider via Sitefusion.comFFC Trap Door Spider
788Fresh Download download managerFreshDownload/x.xx
789FreshNotes - music related artist search (72.3.225.xx)FreshNotes crawler< report problems to crawler-at-freshnotes-dot-com
790FriendicaFriendica 'The Tazmans Flax-lily' 2019.01-1293;
791FS Consulting (was Black Widow web crawlerBlackWidow
793FuseBulb searchFuseBulb.Com
794FyberSearch FyberSpider robotFyberSpider (+
796G2 Web ServicesG2 Web Services/1.0 (built with StormCrawler Archetype 1.8;;
797g2reader-botg2reader-bot/1.0 (+
798GarlikCrawlerGarlikCrawler/1.2 (,
799GenieoMozilla/5.0 (compatible; Genieo/1.0
802GigabotGigabot/2.0 (
803GingerCrawlerGingerCrawler/1.0 (Language Assistant for Dyslexics;; support at ginger software dot com)
804Global Opto's eApollo Flash based search engine - TaiwaneApolloBot/2.0 (compatible; heritrix/2.0.0-SNAPSHOT-20071024.170148 +
805Gluten Free Crawler\/Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +
806GnowitNewsbotMozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0 / GnowitNewsbot / Contact information at
809Google (216.239.39.x) proxy serverDoCoMo/1.0/P502i/c10 (Google CHTML Proxy/1.0)
810Google FaviconMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
811Google Feedfetcher - RSS and Atom feed crawlerFeedfetcher-Google; (+
812Google news feed feetcher for iGoogle gadgetsFeedfetcher-Google-iGoogleGadgets; (+
813Google Web PreviewMozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/4.0 Mobile Safari/537.36
814Google-Adwords-InstantGoogle-Adwords-Instant (+
815Google-PhysicalWebMozilla/5.0 (Google-PhysicalWeb)
816Google-Site-VerificationMozilla/5.0 (compatible; Google-Site-Verification/1.0)
817Google-Structured-Data-Testing-ToolMozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +
819GooglebotGooglebot/2.1 (+
821GooglebotGooglebot/2.1 (+
822GooglebotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; + Safari/537.36
823GooglebotMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +
824GooglebotMozilla/5.0 (compatible; Googlebot/2.1; +
825GooglebotSAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/ (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +
826GooglebotMozilla/5.0 (iPhone; U; CPU iPhone OS) (compatible; Googlebot-Mobile/2.1;
828GooglebotAdsBot-Google (+
829GooglebotMozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +
830GooglebotDoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +
833Googlebot-MobileSAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/ (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +
834Googlebot-MobileNokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +
835Googlebot-MobileMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +
836Googlebot-MobileMozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +
837Googlebot-MobileDoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +
840Googlebot\/Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; + Safari/537.36
841Googlebot\/Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +
842Googlebot\/Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Mobile Safari/537.36 (compatible; Googlebot/2.1; +
843Googlebot\/Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +
844Googlebot\/Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +
845Googlebot\/Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +
846Googlebot\/Mozilla/5.0 (compatible; Googlebot/2.1; +
847Googlebot\/Googlebot/2.1 (+ link popularity checkingBig Fish v1.0
849GowikibotMozilla/5.0 (compatible; Gowikibot/1.0; +
850GrapeshotCrawlerMozilla/5.0 (compatible; GrapeshotCrawler/2.0; +
851GrobbotMozilla/5.0 (compatible; Grobbot/2.2; +
852GroupHigh\/Mozilla/5.0 (compatible; GroupHigh/1.0; +
853grub.orgMozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with
854grub.orgMozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with
855grub.orgMozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with
856grub.orgMozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with
857grub.orgMozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with
858grub.orgMozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with
859grub.orgMozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with
860grub.orgMozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with
861grub.orgMozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with
862grub.orgMozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with
863grub.orgMozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with
864HatenaHatena::Fetcher/0.01 (master) Furl/3.13
865HatenaHatenaBookmark/4.0 (Hatena::Bookmark; Analyzer)
867HatenaHatena-Favicon/2 (
869HatenaHatena Antenna/0.3
870HeadlessChromeMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/76.0.3803.0 Safari/537.36
871HeadlessChromeMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/69.0.3494.0 Safari/537.36
872HeadlessChromeMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/74.0.3729.169 Safari/537.36
873heritrixMozilla/5.0 (compatible; heritrix/3.1.1 +
874heritrixMozilla/5.0 (compatible; heritrix/3.1.1 +
875heritrixMozilla/5.0 (compatible; heritrix/3.1.0-RC1 +
876heritrixMozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV +
877heritrixMozilla/5.0 (compatible; heritrix/3.0.0-SNAPSHOT-20091120.021634 +
878heritrixMozilla/5.0 (compatible; heritrix/2.0.2 +
879heritrixMozilla/5.0 (compatible; heritrix/2.0.2 +
880heritrixMozilla/5.0 (compatible; heritrix/1.14.4 +
881heritrixMozilla/5.0 (compatible; heritrix/1.14.4 +
882heritrixMozilla/5.0 (compatible; heritrix/1.14.3.r6601 +
883heritrixMozilla/5.0 (compatible; heritrix/1.14.3 +
884heritrixMozilla/5.0 (compatible; heritrix/1.14.3 +
885heritrixMozilla/5.0 (compatible; heritrix/1.14.2 +
886heritrixMozilla/5.0 (compatible; heritrix/1.14.2 +
887heritrixMozilla/5.0 (compatible; heritrix/1.12.1b +
888heritrixMozilla/5.0 (compatible; heritrix/1.12.1 +
889heritrixMozilla/5.0 (compatible; heritrix/1.14.3 +
890heritrixMozilla/5.0 (compatible; sukibot_heritrix/3.1.1 +
891heritrixMozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20160309-0050; UniLeipzigASV +
892heritrixMozilla/5.0 (compatible; heritrix/3.2.0 +
893heritrixMozilla/5.0 (compatible; heritrix/3.2.0 +
894heritrixMozilla/5.0 (compatible; heritrix/3.2.0 +
895Heritrix - The Internet Archive's open-source crawler (
898HttpUrlConnectionJersey/2.25.1 (HttpUrlConnection 1.8.0_141)
899HTTrackMozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)
900HubSpotHubSpot Connect 2.0 ( - BizOpsCompanies-Tq2-BizCoDomainValidationAudit
901HubSpotMozilla/5.0 (compatible; HubSpot Crawler;
902HubSpotMozilla/5.0 (X11; Linux x86_64; HubSpot Single Page link check; AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
903HubSpotMozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler -
904Hungarian Academy of Sciences data mining searchComputer_and_Automation_Research_Institute_Crawler
906ia_archiveria_archiver (+;
907IAS crawlerIAS crawler (ias_crawler;
908IBM's Almaden Research robotFocusedSampler/1.0
909ICBot\/Mozilla/5.0 (compatible; ICBot/0.1; +
910ICC-CrawlerICC-Crawler/2.0 (Mozilla-compatible; ;
911IceRocket Web search robotBlogSearch/1.x +
912IceRocket Web search robotBlogzIce/1.0 (+;
913IceRocket Web search robotBlogzIce/1.0 +
914ichiroichiro/5.0 (
915ichiroichiro/3.0 (
916ichiroichiro/2.0 (
917ichiroKDDI-CA31 UP.Browser/ (GUI) MMP/2.0 (compatible; ichiro/mobile goo;+
918ichiroKDDI-CA31 UP.Browser/ (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +
919ichiroDoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo;+
920ichiroDoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +
921ichiroichiro/4.0 (
922ichiroichiro/3.0 (
923ichiroichiro/3.0 (
924ichiroichiro/2.0 (
925ichiroKDDI-CA31 UP.Browser/ (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +
926ichiroDoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+
927ichiroDoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +
928ichiroDoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +
929imrbotMozilla/5.0 (compatible; imrbot/1.10.8 +
930IndeedBotMozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0 (IndeedBot 1.1)
931INFOMINE/iVia Scholary Internet Resource Collections robotDataFountains/DMOZ Downloader
932INFOMINE/iVia Scholary Internet Resource Collections robotDataFountains/Dmoz Downloader (
933INFOMINE/iVia Scholary Internet Resource Collections robotDataFountains/DMOZ Feature Vector Corpus Creator (
934Inktomi (Hotbot-Lycos NBCi) robotBSDSeek/1.0
936Intelliseek's BlogPulse blog searchBlogpulse (
937Intelliseek's BlogPulse blog searchBlogPulseLive (
938Internet Explorer versions 4.x and 5 plugin contentcontype
939IPS AgentMozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1
940IPS AgentMozilla/5.0 (X11; U; Linux i686; en-US; rv:; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4 Firefox/3.5.3
941IPS AgentMozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7
942IPS AgentBlackBerry9000/ Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent
943IPS AgentMozilla/5.0 (X11; U; Linux i686; en-US; rv:; ips-agent) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24
944ips-agentMozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1
945ips-agentMozilla/5.0 (X11; U; Linux i686; en-US; rv:; ips-agent) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24
946ips-agentMozilla/5.0 (X11; U; Linux i686; en-US; rv:; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4Firefox/3.5.3
947ips-agentMozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7
948ips-agentBlackBerry9000/ Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent
949iskanieiskanie (+
950IstellaBotMozilla/5.0 (compatible; IstellaBot/1.23.15 +
952it2media-domain-crawlerit2media-domain-crawler/1.0 on
953James BOTMozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/20070725 Firefox/ - James BOT - WebCrawler
954Jamie's SpiderJamie's Spider (
955JavaOS app. for Sanyo Internet-TVAplix_SANYO_browser/1.x (Japanese)
956JavaOS app. for SEGA Saturn InternetAplix_SEGASATURN_browser/1.x (Japanese)
957JavaOS app. for SEGA Saturn Internet and Sanyo Internet-TVAplix HTTP/1.0.1
958JetCar/FlashGet download managerFlashGet
959JetslideMozilla/5.0 (compatible; Jetslide; +
961JooblebotMozilla/5.0 (compatible; Jooblebot/2.0; Windows NT 6.1; WOW64; + AppleWebKit/537.36 (KHTML, like Gecko) Safari/537.36
962jpg-newsbotjpg-newsbot/2.0; (+
963Jugendschutzprogramm-CrawlerJugendschutzprogramm-Crawler; Info:
964K7MLWCBotK7MLWCBot/1.0 (+
965KemvibotKemvibot/1.0 (,
966Keywen Encyclopedia BotEasyDL/3.xx
967Keywen Encyclopedia BotEasyDL/3.xx
968KosmioBotMozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36 (compatible; KosmioBot/1.0; +
970LaserlikebotMozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Laserlikebot/0.1)
971LeikibotMozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; + Active Bookmark HTML page creatorActiveBookmark 1.x
973libwww-perl2Bone_LinkChecker/1.0 libwww-perl/6.03
974libwww-perl2Bone_LinkChkr/1.0 libwww-perl/6.03
975libwww-perlamibot - - libwww-perl/5.831
976Linguee BotLinguee Bot (
977Linguee BotLinguee Bot (;
978linkapediabotlinkapediabot (+
979LinkArchiver@LinkArchiver twitter bot
980LinkcheckerBeepware (site is down) web directory link checkingBeebwareDirectory/v0.01
981linkdexMozilla/5.0 (compatible; linkdexbot/2.1; +
982linkdexMozilla/5.0 (compatible; linkdexbot/2.0; +
983linkdexMozilla/5.0 (compatible; linkdexbot/2.0; +
984linkdexlinkdexbot/Nutch-1.0-dev (; crawl at linkdex dot com)
986linkdexMozilla/5.0 (compatible; linkdexbot/2.2; +
987linkdexMozilla/5.0 (compatible; linkdexbot/2.1; +
988LinkedInBotLinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +
989LinkedInBotLinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.3 +
990LinkedInBotLinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +
991Linkomatic submission verifierAgentName/0.1 libwww-perl/5.48
992Linkscan tool from Elsopautohttp
993lipperheyMozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +
994lipperheyMozilla/5.0 (compatible; Lipperhey Site Explorer;
995lipperheyMozilla/5.0 (compatible; Lipperhey SEO Service;
996lipperheyMozilla/5.0 (compatible; Lipperhey Link Explorer;
998Livelap[bB]otLivelapBot/0.2 (
999Logitech Desktop Managers (LDM) Backweb (BW) update checkBW-C-2.0
1000ltx71ltx71 - (
1001Luminator-robotsMozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.13 (KHTML, like Gecko) Chrome/30.0.1599.66 Safari/537.13 Luminator-robots/2.0
1002magpie-crawlermagpie-crawler/1.1 (U; Linux amd64; en-GB; +
1003Mail.RU_BotMozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +
1004Mail.RU_BotMozilla/5.0 (compatible; Mail.RU_Bot/2.0; +
1005Mail.RU_BotMozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +
1006Mail.RU_BotMozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +
1007mappydataMozilla/5.0 (compatible; Mappy/1.0; +
1008Mastodonhttp.rb/2.2.2 (Mastodon/1.5.1; +
1009MauiBotMauiBot (
1010MaxOS X CoreFoundation CFNetwork APICFNetwork/x.x
1011MBCrawlerMBCrawler/1.0 (
1012Mediapartners-GoogleMozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +
1013Mediapartners-GoogleMozilla/5.0 (iPhone; U; CPU iPhone OS 10_0 like Mac OS X; en-us) AppleWebKit/602.1.38 (KHTML, like Gecko) Version/10.0 Mobile/14A5297c Safari/602.1 (compatible; Mediapartners-Google/2.1; +
1014Mediapartners-GoogleMozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0 (Following Mediapartners-Google)
1016MediatoolkitbotMediatoolkitbot (
1017MegaIndexMozilla/5.0 (compatible;; +
1018Megaproxy userAnonymized by ProxyOS:
1020memorybotMozilla/5.0 (compatible; memorybot/1.21.14 +
1021Metacarta / Cogent robotflunky
1022MetaJobBotMozilla/5.0 (compatible; MetaJobBot;
1023MetaURIMetaURI API/2.0
1024Microsoft server information robot (see link)BunnySlippers
1025MiggiBot website crawler engine - A1 Sitemap GeneratorA1 Sitemap Generator/1.0 (+ miggibot/2006.01.24
1026mindUpBotmindUpBot (
1027Miniflux\/Mozilla/5.0 (compatible; Miniflux/2.0.x-dev; +
1028Miniflux\\/Mozilla/5.0 (compatible; Miniflux/2.0.12; +
1029Miniflux\\/Mozilla/5.0 (compatible; Miniflux/3b6e44c; +
1030Miniflux\\/Mozilla/5.0 (compatible; Miniflux/ae1dc1a; +
1031Miniflux\\/Mozilla/5.0 (compatible; Miniflux/2.0.11; +
1032Miniflux\\/Mozilla/5.0 (compatibl$; Miniflux/2.0.x-dev; +
1033Miniflux\\/Mozilla/5.0 (compatible; Miniflux/2.0.10; +
1034Miniflux\\/Mozilla/5.0 (compatible; Miniflux/2.0.7; +
1035Miniflux\\/Mozilla/5.0 (compatible; Miniflux/2.0.3; +
1036Miniflux\\/Mozilla/5.0 (compatible; Miniflux/2.0.x-dev; +
1037Mirago UK RobotExperimentalHenrytheMiragoRobot
1038MixCat robotFelix - Mixcat Crawler (+
1040MJ12botMozilla/5.0 (compatible; MJ12bot/v1.3.3;
1041MJ12botMozilla/5.0 (compatible; MJ12bot/v1.3.2;
1042MJ12botMozilla/5.0 (compatible; MJ12bot/v1.3.1;
1043MJ12botMozilla/5.0 (compatible; MJ12bot/v1.3.0;
1044MJ12botMozilla/5.0 (compatible; MJ12bot/v1.2.5;
1045MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.8;
1046MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.7;
1047MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.7;
1048MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.6;
1049MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.5;
1050MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.4;
1051MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.2;
1052MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.0;
1053MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.1;
1054MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.3;
1055MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.4 (domain ownership verifier);
1056MJ12botMozilla/5.0 (compatible; MJ12bot/v1.2.3;
1057MJ12botMozilla/5.0 (compatible; MJ12bot/v1.2.1;
1058MJ12botMJ12bot/v1.2.0 (
1059MJ12botMozilla/5.0 (compatible; MJ12bot/v1.2.4;
1060mlbotMLBot (
1061MlbotMLBot (
1062moatbotMozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36 moatbot
1063MojeekBot\/MojeekBot/0.2 (archi;
1064MojeekBot\\/Mozilla/5.0 (compatible; MojeekBot/0.6;
1065MojeekBot\\/Mozilla/5.0 (compatible; MojeekBot/0.6; +
1066MojeekBot\\/Mozilla/5.0 (compatible; MojeekBot/0.5;
1067MojeekBot\\/Mozilla/5.0 (compatible; MojeekBot/0.2;
1068MojeekBot\\/Mozilla/5.0 (compatible; MojeekBot/0.2;
1069MojeekBot\\/MojeekBot/0.2 (archi;
1071MoreoverMozilla/5.0 Moreover/5.1 (+
1072msnbotmsnbot/2.0b (+
1073msnbotmsnbot/1.1 (+
1074msnbotmsnbot/2.0b (+
1075msnbotadidxbot/2.0 (+
1076msnbotadidxbot/1.1 (+
1077msnbotmsnbot/2.0b (+
1078msnbotmsnbot/1.0 (+
1079msnbotmsnbot-media/2.0b (+
1080msnbotmsnbot-media/1.1 (+
1081msnbotmsnbot-media/1.0 (+
1082msnbotmsnbot-UDiscovery/2.0b (+
1083msnbotmsnbot-NewsBlogs/2.0b (+
1084msnbotlibrabot/2.0 (+
1085msnbotlibrabot/1.0 (+
1086MuckRackMozilla/5.0 (compatible; MuckRack/1.0; +
1087MultiviewbotMozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Multiviewbot
1088Nameprotect copyright search robot (24.177.134.x)abot/0.1 (abot;;
1089Nameprotect copyright search robot (24.177.134.x)aipbot/1.0 (aipbot;;
1090Nameprotect copyright search robot (24.177.134.x)aipbot/2-beta (aipbot dev;;
1091National Research Council Canada robotAVSearch-1.0(
1092Naver Japan / Korea robotCowbot-0.1 (NHN Corp. / +82-2-3011-1954 /
1093Naver Japan / Korea robotCowbot-0.1.x (NHN Corp. / +82-2-3011-1954 /
1094Naver Japan / Korea robotdloader(NaverRobot)/1.0
1095neofonie search robot Germany:robot/1.0 (linux) ( admin e-mail: undefined )
1096Neomo Search (Germany) robot ( (
1097NerdByNature.BotMozilla/5.0 (compatible; NerdByNature.Bot;
1099NetcraftSurveyAgentMozilla/5.0 (compatible; NetcraftSurveyAgent/1.0;
1100netEstate NE CrawlernetEstate NE Crawler (+
1101NetvibesNetvibes (crawler;
1102NetvibesNetvibes (crawler/bot;
1103Newsbrain Feedreader3Feedreader 3.xx (Powered by Newsbrain)
1104newsharecountsMozilla/5.0 (compatible;; +
1111NextLinks - German vocabulary and hyperlink searchfindlinks/ (+
1112NF embedded browser on Amois SkypephoneAmoi 8512/R21.0 NF-Browser/3.3
1113Nimbostratus-BotMozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2;
1115Nmap Scripting EngineMozilla/5.0 (compatible; Nmap Scripting Engine;
1116Nordic semantic search engineCentiverseBot
1117Nordic semantic search engineCentiverseBot - investigator
1118Nordic semantic search engineCentiverseBot/3.0 (
1119NTL user agentCheckUrl
1120NTT DoCoMo (Japan) proxy server ( SH901iS(c100;TB;W24H12),gzip(gfe) (via
1121NTT DoCoMo (Japan) robotDoCoMo/1.0/Nxxxi/c10
1122NTT DoCoMo (Japan) robotDoCoMo/1.0/Nxxxi/c10/TB
1123NTT DoCoMo (Japan) robotDoCoMo/2.0 P900iV(c100;TB;W24H11)
1124nutchNutchCVS/0.7.1 (Nutch;;
1133omgiliomgili/0.5 +
1134online-webceo-botMozilla/5.0 (compatible; online-webceo-bot/1.0; +
1135OpenHoseBotMozilla/5.0 (compatible; OpenHoseBot/2.1; +
1136OpenPortal4U robotDigOut4U
1137OrangeBot\/Mozilla/5.0 (compatible; OrangeBot/2.0;
1138outbrainMozilla/5.0 (Java) outbrain
1139OutclicksBotOutclicksBot/2 +
1140OutclicksBotOutclicksBot/2 +
1141OutclicksBotOutclicksBot/2 +
1142OutclicksBotOutclicksBot/2 +
1143page2rssMozilla/5.0 (compatible;Page2RSS/0.7; +
1144page2rssMozilla/5.0 (compatible; Page2RSS/0.7; +
1145Pages Jaunes business search (France) robot using Fast Enterprise CrawlerFAST Enterprise Crawler 6 used by Pages Jaunes (
1147PaperLiBotMozilla/5.0 (compatible; PaperLiBot/2.1;
1148Part of ButtMan remote access toolBMCLIENT
1150PhantomJSMozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1 bl.uk_lddc_renderbot/2.0.0 (+ UP.Browser for mobiles on Alcatel cellphoneAlcatel-BG3/1.0 UP.Browser/
1153pingdomMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/59.0.3071.109 Chrome/59.0.3071.109 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; +
1154Pinpoint WAP search robotCrawlerBoy (compatible; Pinterestbot/1.0; +
1156PiplBotPiplBot (+
1157PocketParserPocketParser/2.0 (+
1158postrankPostRank/2.0 (
1159postrankPostRank/2.0 (; 1 subscribers)
1160PowerTCP ActiveX control toolDart Communications PowerTCP
1161PR-CY.RUMozilla/5.0 (compatible; PR-CY.RU; +
1162PrimalbotMozilla/5.0 (compatible; Primalbot; +;)
1163PrivacyAwareBotMozilla/5.0 (compatible; PrivacyAwareBot/1.1; +
1164Privoxy web proxy(Privoxy/1.0)
1165Probably E-Mail harvesting robot - same as LMQueueBotContactBot/0.2
1166Project BanBots Perl script robotBanBots/1.2 (
1167Proposed Content-Based Image Tracking System (CoBITS) P2P crawlerCoBITSProbe
1168ProximicMozilla/5.0 (compatible; proximic; +
1169proximicMozilla/5.0 (compatible; proximic; +
1170ProximicMozilla/5.0 (compatible; proximic; +
1171ProximicMozilla/5.0 (compatible; proximic; +
1172psbotpsbot-page (+
1173psbotpsbot-image (+
1174psbotpsbot/0.1 (+
1175PubSub - Mac OS X utility for managing RSS/Atom subscriptions via the PubSub frameworkApple-PubSub/65.1.1
1176PulsepointPulsepoint XT3 web scraper
1195QwantifyMozilla/5.0 (compatible; Qwantify/2.0n; +*
1196QwantifyMozilla/5.0 (compatible; Qwantify/Bleriot/1.1; +
1197QwantifyMozilla/5.0 (compatible; Qwantify/2.4w; +
1198QwantifyMozilla/5.0 (compatible; Qwantify/Bleriot/1.2.1; +
1199RankActiveLinkBotMozilla/5.0 (compatible; RankActiveLinkBot; +
1200redditbotMozilla/5.0 (compatible; redditbot/1.0; +
1201RegionStuttgartBotMozilla/5.0 (compatible; RegionStuttgartBot/1.0; +
1202Relevare Portal software
1203RetrevoPageAnalyzerMozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; RetrevoPageAnalyzer; +
1204RivvaMozilla/5.0 (compatible; Rivva;
1205Robot for BravoBrian bSTOPBravoBrian SpiderEngine MarcoPolo
1206rogerbotrogerbot/1.2 (,
1207rogerbotrogerbot/1.0 (,
1208rogerbotMozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler;
1209rogerbotrogerbot/1.0 (,
1210rogerbotrogerbot/1.1 (,
1211rogerbotrogerbot/1.1 (,
1212rogerbotrogerbot/1.1 (,
1213rogerbotrogerbot/1.0 (,
1214rogerbotrogerbot/1.0 (,
1215rogerbotrogerbot/1.0 (,
1216rogerbotrogerbot/1.0 (,
1217rogerbotrogerbot/1.0 (,
1218rogerbotrogerbot/1.0 (,
1219RSS Feed Seeker bot ( Seeker Bot (RSS Feed Seeker
1220RSSingBotRSSingBot (
1221S[eE][mM]rushBotMozilla/5.0 (compatible; SemrushBot/6~bl; +
1223S[eE][mM]rushBotMozilla/5.0 (compatible; SemrushBot/0.98~bl; +
1224S[eE][mM]rushBotMozilla/5.0 (compatible; SemrushBot-BA; +
1225S[eE][mM]rushBotMozilla/5.0 (compatible; SemrushBot/3~bl; +
1226S[eE][mM]rushBotMozilla/5.0 (compatible; SemrushBot-SI/0.97; +
1227S[eE][mM]rushBotMozilla/5.0 (compatible; SemrushBot-SA/0.97; + web filtering serviceFANGCrawl/0.01
1229SafeDNSBotSafeDNSBot (
1230SafeSearch microdata crawlerSafeSearch microdata crawler (,
1231Samsung SPH-A660 phone with Sprint softwareAU-MIC/2.0 MMP/2.0
1233ScoutJetMozilla/5.0 (compatible; ScoutJet; +
1234ScrapyScrapy/1.0.3 (+
1235Screaming Frog SEO SpiderScreaming Frog SEO Spider/5.1 (Germany) submission checking / robot (84.73.57.xx)de.searchengine.comBot 1.2 (
1237SearchEngineWorld's robots.txt validator( Robots.txt Validator ) robot (
1239seekbotSeekbot/1.0 ( RobotsTxtFetcher/1.2
1240Seekport CrawlerMozilla/5.0 (compatible; Seekport Crawler;
1242semanticbotsemanticbot (
1243SemanticScholarBotSemanticScholarBot/1.0 (+
1244SEOkicksMozilla/5.0 (compatible; SEOkicks; +
1245seoscannersMozilla/5.0 (compatible;;
1246serpstatbot\/serpstatbot/1.0 (advanced backlink tracking bot;;
1247Seznam Search (Czech Republic) robotCloakDetect/0.9 (+
1248seznambotMozilla/5.0 (compatible; SeznamBot/3.2; +
1249seznambotMozilla/5.0 (compatible; SeznamBot/3.2-test4; +
1250seznambotMozilla/5.0 (compatible; SeznamBot/3.2-test1-1; +
1251seznambotMozilla/5.0 (compatible; SeznamBot/3.2-test2; +
1252seznambotMozilla/5.0 (compatible; SeznamBot/3.2-test1; +
1253SharewarePlaza link checkingAgent-SharewarePlazaFileCheckBot/2.0+(+
1254Sicher-durchs-Netz German security related search (
1256SimpleScraperMozilla/5.0 (compatible; SimpleScraper)
1257Simpy Bookmarklet crawler (69.55.233.xx)Argus/1.1 (Nutch;; feedback at simpy dot com)
1258Singingfish media spider ( via AOL searchasterias/2.0
1259sitebotMozilla/5.0 (compatible; Whoiswebsitebot/0.1; +
1260siteexplorer.infoMozilla/5.0 (compatible; SiteExplorer/1.0b; +
1261siteexplorer.infoMozilla/5.0 (compatible; SiteExplorer/1.1b; +
1262Siteimprove.comMozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) LinkCheck by
1263Siteimprove.comMozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by
1264Siteimprove.comMozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by
1265Siteimprove.comMozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) Match by search (Germany) (+Have Good Day)
1267SkypeUriPreviewMozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5
1268Slack-ImgProxySlack-ImgProxy 1.106 (+
1269Slack-ImgProxySlack-ImgProxy 0.59 (+
1270Slack-ImgProxySlack-ImgProxy 149 (+
1271Slack-ImgProxySlack-ImgProxy 1.138 (+
1272Slack-ImgProxySlack-ImgProxy 0.66 (+
1273Slack-ImgProxySlack-ImgProxy (+
1274SlackbotSlackbot-LinkExpanding (+
1275SlurpMozilla/5.0 (compatible; Yahoo! Slurp/3.0;
1276smtbotMozilla/5.0 (compatible; SMTBot/1.0; +
1277smtbotMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0;
1278smtbotSMTBot (
1279smtbotMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; +
1280smtbotMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +
1281SnacktoryMozilla/5.0 (compatible; Snacktory; +
1283SogouSogou Pic Spider/3.0(
1284SogouSogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
1285SogouSogou News Spider/4.0(+
1286SogouSogou Orion spider/3.0(
1287SogouSogou head spider/3.0(
1288some (website) downloading toolDDD
1289some download agentdlman
1290Some site scanning tool from ( Bot DOT 16b
1291Some site scanning tool from ( Web Bot 2816B
1292Some site scanning tool from 68.154.96.xx ( Bot Z 16b
1293Some site scanning tool from diff. IPs i.e.: - 66.28.240.xx ( - 68.5.174.xx ( Web Bot 0416B
1294Some site scanning tool i.e. from - 68.154.96.xx ( Web Bot 0516B
1295Some site scanning tool via diff. IPsESurf15a 15
1296Some site scanning tool via diff. IPsFSurf15a 01
1297Some site scanning tool via diff. IPs i.e.: - ( 71
1298Some site scanning tool via diff. IPs i.e.: - ( Search VxB
1299Some site scanning tool via diff. IPs i.e.: - ( - ( 01
1300Some site scanning tool via diff. IPs i.e.: - ( - ( VA
1301Some site scanning tool via diff. IPs i.e.: - ( 1.4d
1302Some site scanning tool via diff. IPs i.e.: - ( 1.4b
1303Some site scanning tool via diff. IPs i.e.: - ( 81
1304Some site scanning tool via diff. IPs i.e.: - ( - ( 1.4b
1305Some spam botFranklin Locator 1.8
1306Some user agentCancer Information and Support International;
1307SonicMozzila/5.0 (compatible; Sonic/1.0;
1308SonicMozilla/5.0 (compatible; Sonic/1.0;
1309SonicMozilla/5.0 (compatible; RankSonicSiteAuditor/1.0; +
1310Sonic E-mail collectorEmailSiphon
1311Sootle Web Directory deep link checkerDeep Link Calculator v1.0
1312SorgouSogou web spider/4.0(+
1313spbotMozilla/5.0 (compatible; spbot/4.0.7; + )
1314spbotMozilla/5.0 (compatible; spbot/4.4.1; + )
1315spbotMozilla/5.0 (compatible; spbot/4.4.2; + )
1316spbotMozilla/5.0 (compatible; spbot/5.0.1; + )
1317spbotMozilla/5.0 (compatible; spbot/5.0.3; + )
1318spbotMozilla/5.0 (compatible; spbot/1.0; + )
1319spbotMozilla/5.0 (compatible; spbot/4.0a; + )
1320spbotMozilla/5.0 (compatible; spbot/5.0; + )
1321spbotMozilla/5.0 (compatible; spbot/4.4.0; + )
1322spbotMozilla/5.0 (compatible; spbot/4.3.0; + )
1323spbotMozilla/5.0 (compatible; spbot/4.0; + )
1324spbotMozilla/5.0 (compatible; spbot/4.0.5; + )
1325spbotMozilla/5.0 (compatible; spbot/4.0.3; + )
1326spbotMozilla/5.0 (compatible; spbot/4.0.2; + )
1327spbotMozilla/5.0 (compatible; spbot/3.1; + )
1328spbotMozilla/5.0 (compatible; spbot/3.0; + )
1329spbotMozilla/5.0 (compatible; spbot/2.0; + )
1330spbotMozilla/5.0 (compatible; spbot/2.0.4; + )
1331spbotMozilla/5.0 (compatible; spbot/2.0.1; + )
1332spbotMozilla/5.0 (compatible; spbot/1.2; + )
1333spbotMozilla/5.0 (compatible; spbot/5.0.2; + )
1334spbotMozilla/5.0 (compatible; spbot/4.2.0; + )
1335spbotMozilla/5.0 (compatible; spbot/4.1.0; + )
1336spbotMozilla/5.0 (compatible; spbot/4.0b; + )
1337spbotMozilla/5.0 (compatible; spbot/4.0.9; + )
1338spbotMozilla/5.0 (compatible; spbot/4.0.8; + )
1339spbotMozilla/5.0 (compatible; spbot/4.0.7; + )
1340spbotMozilla/5.0 (compatible; spbot/4.0.6; + )
1341spbotMozilla/5.0 (compatible; spbot/4.0.4; + )
1342spbotMozilla/5.0 (compatible; spbot/4.0.1; + )
1343spbotMozilla/5.0 (compatible; spbot/2.1; + )
1344spbotMozilla/5.0 (compatible; spbot/2.0.3; + )
1345spbotMozilla/5.0 (compatible; spbot/2.0.2; + )
1346spbotMozilla/5.0 (compatible; spbot/1.1; + )
1347speedyMozilla/5.0 (compatible; Speedy Spider;
1348speedyMozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider for SpeedyAds (
1349speedySpeedy Spider (
1350speedySpeedy Spider (Entireweb; Beta/1.2;
1351speedyMozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (
1352SPH Search - Singapore related search using Fast crawlerFAST Enterprise Crawler 6 used by Singapore Press Holdings (
1353StorygizeBotMozilla/5.0 (compatible; StorygizeBot;
1354Streamline3Bot\/Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) Streamline3Bot/1.0
1355Suchmaschine21 (Germany) robotCoolBot
1356summifySummify (Summify/1.0.1; +
1357SurveyBotMozilla/5.0 (Windows; U; Windows NT 5.1; en; rv: Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)
1358SWIMGBotMozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36 SWIMGBot robot using AspseekASPseek/1.2.9d
1360Sygol Search (Italy) robot
1361Sync2It bookmark management & clustering engine!Susie (
1362SysomosMozilla/5.0 (compatible; Sysomos/1.0; +; Sysomos)
1363TangibleeBotTangibleeBot/ (
1364TelegramBotTelegramBot (like TwitterBot)
1365TeomaMozilla/5.0 (compatible; Ask Jeeves/Teoma; +
1366TeomaMozilla/2.0 (compatible; Ask Jeeves/Teoma; +
1367TeomaMozilla/2.0 (compatible; Ask Jeeves/Teoma)
1368TeomaMozilla/2.0 (compatible; Ask Jeeves/Teoma; +
1369theoldreader.comMozilla/5.0 (compatible;
1370ThinklabThinklab (
1371TimelyWeb web page monitoring toolEldoS TimelyWeb/3.x
1372Timewe mobile browser (WAP) simulator (Japan)CDR/1.7.1 Simulator/0.7(+ Profile/MIDP-1.0 Configuration/CLDC-1.0
1373TinEyeMozilla/5.0 (compatible; TinEye-bot/1.31; +
1374Tiny Tiny RSSTiny Tiny RSS/1.15.3 (
1375Tomba project: the Portuguese web archiveArquivo-web-crawler (compatible; heritrix/1.12.1 +
1376ToutiaoSpiderMozilla/5.0 (compatible; ToutiaoSpider/1.0;;)
1378tracemyfileMozilla/5.0 (compatible; tracemyfile/1.0;
1379Trellian EMailWolf E-mail collectorEmailWolf 1.00
1380trendictionbotMozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search;; please let us know of any problems; web at Gecko/20071127 Firefox/
1381TrendsmapResolverMozilla/5.0 (compatible; TrendsmapResolver/0.1)
1382TurnitinBotTurnitinBot (
1383tweetedtimesMozilla/5.0 (compatible; +
1384TweetmemeBotMozilla/5.0 (TweetmemeBot/4.0; + Gecko/20100101 Firefox/31.0
1385TwinglyMozilla/5.0 (compatible; Twingly Recon;
1388TwurlyRuby, Twurly v1.1 (
1389UCLA Complex Networks Groups Complex Network Analysiscomplex_network_group/Nutch-0.9-dev (discovering the structure of the world-wide-web;;
1390UIUCs Cazoodle search based on MetaQuerierCazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler;;
1391um-LNMozilla/5.0 (compatible; um-LN/1.0; mailto:
1392UnCHAOS search robot UnChaos From Chaos To Order Hybrid Web Search Engine.(
1393UnCHAOS search robot UnChaos Bot Hybrid Web Search Engine. (
1394UnCHAOS search robot UnChaosBot From Chaos To Order UnChaos Hybrid Web Search Engine at (
1396unknownfly/6.01 libwww/4.0D
1397Unknown bot from Kornet Korea ( scans for Perl AwstatsDataCha0s/2.0
1398unknown divine/ robotdCSbot/1.1
1399Unknown guestbook spamming or harvesting tool from diff. IPs8484 Boston Project v 1.0
1400Unknown robot from 205.203.108.xx (
1401unknown robot from AGMLAB Information Technologies (Information retrieval system ?)dodgebot/experimental
1402Unknown robot from Allrati.comAllrati/1.1 (+)
1403Unknown robot from Czech Technical University Prague (147.32.141.xx)Firefox (
1404Unknown robot from Czech Technical University Prague (147.32.141.xx)Firefox_1.0.6 (
1405Unknown robot from douban search (China) - maybe image crawlingDoubanbot/1.0 (
1406Unknown robot from the University of Tokyo (157.82.157.xx)FeedChecker/0.01
1407unknown robot via Level3.netCrawler
1408Unknown spam bot from ( Local Browse 2.6
1409Unknown UA looking for Awstats Perl componentsDataCha0s/2.0
1410Unknown user agentdds explorer v1.0 beta
1411Unknown user agent from MicrosoftbCentral Billing Post-Process
1412Unknown Yahoo robot*/Nutch-0.9-dev
1413Unkown robot from Discovery Engine Corp.disco/Nutch-0.9 (experimental crawler;;
1414Unkown robot from Discovery Engine Corp.disco/Nutch-1.0-dev (experimental crawler;;
1416Uptimebot\ - Free website monitoring
1417UptimeRobotMozilla/5.0+(compatible; UptimeRobot/2.0;
1418urlappendbotMozilla/5.0 (compatible; URLAppendBot/1.0; +
1420vebidoobotMozilla/5.0 (compatible; vebidoobot/1.0; +
1421VelenPublicWebCrawlerVelenPublicWebCrawler (
1422VeoozbotMozilla/5.0 (compatible; Veoozbot/1.0; +
1423Vestris robotAlkalineBOT/1.3
1424Vestris robotAlkalineBOT/1.4 (1.4.0326.0 RTM)
1425Vigiltech e-Sense user research robot (website is offline)e-sense 1.0 ea(
1426vkShareMozilla/5.0 (compatible; vkShare; +
1427voilabotMozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (
1428voilabotMozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 (
1429Voyager - Amiga browserAmigaVoyager/3.4.4 (MorphOS/PPC native)
1430Vulcan Inc. Everest crawler (in development)Everest-Vulcan Inc./0.1 (R&D project; host=e-1-24;
1431Vulcan Inc. Everest crawler (in development)Everest-Vulcan Inc./0.1 (R&D project;
1432W3C_CSS_ValidatorJigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0
1436W3C-checklinkW3C-checklink/4.5 [4.154] libwww-perl/5.823
1437W3C-checklinkW3C-checklink/4.3 [4.42] libwww-perl/5.820
1438W3C-checklinkW3C-checklink/4.2 [4.20] libwww-perl/5.803
1439W3C-checklinkW3C-checklink/2.90 libwww-perl/5.64
1440W3C-checklinkW3C-checklink/4.3 [4.42] libwww-perl/5.805
1441W3C-checklinkW3C-checklink/4.2.1 [4.21] libwww-perl/5.803
1442W3C-checklinkW3C-checklink/4.5 [4.160] libwww-perl/5.823
1443W3C-checklinkW3C-checklink/4.3 [4.42] libwww-perl/5.808
1444W3C-checklinkW3C-checklink/ libwww-perl/5.64
1446Walhello Internet Search robotappie 1.1 (
1447Web mining project from CSCI 659 (computer science course) at Indiana Univeritycsci_b659/0.13
1448Webarchive Project Bruinbot crawlerBruinBot (+
1449WebCollage Syndicator graphics crawler/collectorcollage.cgi/1.xx
1450WebDataStatsMozilla/5.0 (compatible; WebDataStats/1.0 ; +
1451website directory adaxas link checkingAdaxas Spider (
1452WeSEE:SearchWeSEE:Search/0.1 (Alpha,
1454WhatsAppWhatsApp/2.19.308 A
1455WhatsAppWhatsApp/2.19.244 A
1456WhatsAppWhatsApp/2.19.92 i
1458WhatsAppWhatsApp/2.12.555 A
1459WhatsAppWhatsApp/2.12.510 A
1460WhatsAppWhatsApp/2.12.449 A
1462WhatsAppWhatsApp/0.3.4941 N
1463WhatsAppWhatsApp/0.3.4679 N
1464WhatsAppWhatsApp/0.3.4479 N
1465WhatsAppWhatsApp/2.16.42 A
1466WhatsAppWhatsApp/2.12.548 A
1467WhatsAppWhatsApp/2.19.330 A
1468WhatsAppWhatsApp/2.19.258 A
1469WhatsAppWhatsApp/2.19.175 A
1470WhatsAppWhatsApp/2.16.57 A
1471WhatsAppWhatsApp/2.16.13 A
1472WhatsAppWhatsApp/2.12.556 A
1473WhatsAppWhatsApp/2.12.540 A
1474WhatsAppWhatsApp/2.12.453 A
1479Wholeweb robotDaviesBot/1.7 (
1480Wildsoft Germany (closed) Internet client system user-agent (was Browser Center
1483WoriobotMozilla/5.0 (compatible; woriobot support [at] zite [dot] com +
1484woriobotMozilla/5.0 (compatible; woriobot +
1485WoriobotMozilla/5.0 (compatible; woriobot +
1486wotboxWotbox/2.01 (+
1487wotboxWotbox/2.0 (;
1488www\.uptime\.comMozilla/5.0 (compatible; Uptimebot/1.0; +
1489Xdefine text search engine robot - based on Egothor open source crawleregothor/3.0a (+
1490Xenu Link SleuthXenu Link Sleuth/1.3.8
1491xovibotMozilla/5.0 (compatible; XoviBot/2.0; +
1492Xyleme SA France robotcosmos/0.8_(
1493Xyleme SA France robotcosmos/0.9_(
1494Y!JMozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT;
1495Y!JMozilla/5.0 (iPhone; Y!J-BRY/YATSH crawler;
1496Y!JY!J-BRW/1.0 crawler (
1497Y!JY!J-ASR/0.1 crawler (
1498Y!JY!J-BRJ/YATS crawler (
1499Y!JY!J-PSC/1.0 crawler (
1500yacybotyacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.7.0_79; Europe/en)
1501yacybotyacybot (/global; amd64 Linux 2.6.32-573.3.1.el6.x86_64; java 1.7.0_85; Europe/en)
1502yacybotyacybot (/global; amd64 Linux 2.6.32-042stab094.8; java 1.7.0_79; America/en)
1503yacybotyacybot (-global; amd64 FreeBSD 9.2-RELEASE-p10; java 1.7.0_65; Europe/en)
1504yacybotyacybot (/global; amd64 FreeBSD 10.3-RELEASE; java 1.8.0_77; GMT/en)
1505yacybotyacybot (-global; amd64 Linux 5.2.11-Jinsol; java 12.0.2; Europe/en)
1506yacybotyacybot (-global; amd64 Linux 5.2.9-Jinsol; java 12.0.2; Europe/en)
1507yacybotyacybot (-global; amd64 Linux 5.2.8-Jinsol; java 12.0.2; Europe/en)
1508yacybotyacybot (-global; amd64 Windows 8 6.2; java 1.7.0_55; Europe/de)
1509yacybotyacybot (-global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de)
1510yacybotyacybot (-global; amd64 Linux 4.4.0-57-generic; java 9-internal; Europe/en)
1511yacybotyacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en)
1512yacybotyacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en)
1513yacybotyacybot (-global; amd64 Linux 3.19.0-15-generic; java 1.8.0_45-internal; Europe/de)
1514yacybotyacybot (/global; amd64 Linux 3.16-0.bpo.2-amd64; java 1.7.0_65; Europe/en)
1515yacybotyacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_111; Europe/en)
1516yacybotyacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.8.0_45; Europe/en)
1517yacybotyacybot (-global; amd64 Linux 2.6.32-042stab116.1; java 1.7.0_79; Europe/en)
1518yacybotyacybot (-global; amd64 Linux 2.6.32-042stab111.11; java 1.7.0_79; Europe/en)
1519yacybotyacybot (/global; amd64 Linux 2.6.32-042stab108.8; java 1.7.0_91; America/en)
1520yacybotyacybot (/global; amd64 FreeBSD 10.3-RELEASE-p7; java 1.7.0_95; GMT/en)
1521yacybotyacybot (/global; amd64 Linux 2.6.32-042stab093.4; java 1.7.0_65; Etc/en)
1522yacybotyacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_95; Europe/en)
1523yacybotyacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_91; Europe/de)
1524yacybotyacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/en)
1525yacybotyacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de)
1526yacybotyacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en)
1527yacybotyacybot (-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en)
1528yacybotyacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; America/en)
1529yacybotyacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_111; Europe/de)
1530yacybotyacybot (-global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.8.0_111; Europe/de)
1531yacybotyacybot (/global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.7.0_75; Europe/en)
1532yacybotyacybot (/global; amd64 Linux 3.14-0.bpo.1-amd64; java 1.7.0_55; Europe/de)
1533yacybotyacybot (/global; amd64 Linux 3.13.0-88-generic; java 1.7.0_101; Europe/en)
1534yacybotyacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_95; Europe/en)
1535yacybotyacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_101; Europe/en)
1536yacybotyacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/en)
1537yacybotyacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/de)
1538yacybotyacybot (/global; amd64 Linux 3.13.0-74-generic; java 1.7.0_91; Europe/en)
1539yacybotyacybot (-global; amd64 Linux 3.13.0-61-generic; java 1.7.0_79; Europe/en)
1540yacybotyacybot (/global; amd64 Linux 3.13.0-45-generic; java 1.7.0_75; Europe/en)
1541yacybotyacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/en)
1542yacybotyacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/de)
1543yacybotyacybot (/global; amd64 Linux 3.12.1; java 1.7.0_65; Europe/en)
1544yacybotyacybot (/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; America/en)
1545yacybotyacybot (/global; amd64 Linux 3.10.0-327.22.2.el7.x86_64; java 1.7.0_101; Etc/en)
1546yacybotyacybot (/global; amd64 Linux 3.10.0-229.7.2.el7.x86_64; java 1.8.0_45; Europe/en)
1547YahooMozilla/5.0 (compatible; Yahoo! Slurp China;
1548YahooMozilla/5.0 (compatible; Yahoo! Slurp/3.0;
1549YahooMozilla/5.0 (compatible; Yahoo! Slurp;
1550Yahoo Link PreviewMozilla/5.0 (compatible; Yahoo Link Preview;
1551Yahoo Search Japan robot ( SH902i (compatible; Y!J-SRD/1.0;
1552Yahoo Search Japan robot ( (compatible; Y!J-SRD/1.0;
1553YaK\/Mozilla/5.0 (compatible; YaK/1.0;;
1554YandexMozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +
1555YandexMozilla/5.0 (compatible; YandexVertis/3.0; +
1556YandexMozilla/5.0 (compatible; YandexAntivirus/2.0; +
1557YandexMozilla/5.0 (compatible; YandexMetrika/3.0; +
1558YandexMozilla/5.0 (compatible; YandexSitelinks; Dyatel; +
1559YandexMozilla/5.0 (compatible; YandexCalendar/1.0; +
1560YandexMozilla/5.0 (compatible; YaDirectFetcher/1.0; Dyatel; +
1561YandexMozilla/5.0 (compatible; YandexImageResizer/2.0; +
1562YandexMozilla/5.0 (compatible; YandexPagechecker/1.0; +
1563YandexMozilla/5.0 (compatible; YandexWebmaster/2.0; +
1564YandexMozilla/5.0 (compatible; YandexFavicons/1.0; +
1565YandexMozilla/5.0 (compatible; YandexBlogs/0.99; robot; +
1566YandexMozilla/5.0 (compatible; YandexMedia/3.0; +
1567YandexMozilla/5.0 (compatible; YandexVideo/3.0; +
1568YandexMozilla/5.0 (compatible; YandexImages/3.0; +
1569YandexMozilla/5.0 (compatible; YandexDirectDyn/1.0; +
1570YandexMozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +
1571YandexMozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +
1572YandexMozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +
1573YandexMozilla/5.0 (compatible; YandexBot/3.0; +
1574YandexAccessibilityBotMozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +
1575YandexBotMozilla/5.0 (compatible; YandexBot/3.0; +
1576YandexImagesMozilla/5.0 (compatible; YandexImages/3.0; +
1577YandexMobileBotMozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +
1578YangaYanga WorldSearch Bot v1.1/beta (
1579yangaYanga WorldSearch Bot v1.1/beta ( business search (Norway) via FindexaFindexa Crawler (
1581YetiMozilla/5.0 (compatible; Yeti/1.1; +
1583yoozBotMozilla/5.0 (compatible; yoozBot-2.2;;
1585ZD's FavOrg favourites managing programFavOrg
1586Zetbot search Belgium (213.41.128.xx) using Deepindex robotDeepIndex ( )
1587zgrabMozilla/5.0 zgrab/0.x
1588ZoomBotZoomBot (Linkbot 1.0
1589ZoominfoBotZoominfoBot (zoominfobot at zoominfo dot com)
1590ZumBotMozilla/5.0 (compatible; ZumBot/1.0;
1591ZuperlistBot\/Mozilla/5.0 (compatible; ZuperlistBot/1.0)
1592ZuperlistBot\\/Mozilla/5.0 (compatible; ZuperlistBot/1.0)
1593µTorrent BitTorrent ClientBTWebClient/180B(9704)
1594\[ZSEBOT\]ZoomSpider - [ZSEBOT]
1595!Susie ( bookmark management & clustering engine
1596 UnChaos From Chaos To Order Hybrid Web Search Engine.( search robot
1597 UnChaos Bot Hybrid Web Search Engine. ( search robot
1598 UnChaosBot From Chaos To Order UnChaos Hybrid Web Search Engine at ( search robot
1599 http://www.sygol.comSygol Search (Italy) robot
1600( Robots.txt Validator )SearchEngineWorld's robots.txt validator
1601(DreamPassport/3.0; isao/MyDiGiRabi)DreamCast DreamPassport browser
1602(Privoxy/1.0)Privoxy web proxy
1603*/Nutch-0.9-devUnknown Yahoo robot (+Have Good Day) search (Germany) robot
1605-DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.deDie Kraehe Meta-Search-Engine (Germany) link checking
1606123spider-Bot (Version: 1.02, powered by (Germany) web directory link checking - UK web directory
16081st ZipCommander (Net) - ZipCommander Net - IE based browser
16092Bone_LinkChecker/1.0 libwww-perl/5.642Bone online link checker LinkChecker v2.04Anything robot
16118484 Boston Project v 1.0Unknown guestbook spamming or harvesting tool from diff. IPs
1612:robot/1.0 (linux) ( admin e-mail: undefined )neofonie search robot Germany
1613A-Online robot - now Jet2Web Search
1614A1 Keyword Research/1.0.2 (+ miggibot/2007.03.27A1 Keyword Research - search engine and keyword optimization software
1615A1 Sitemap Generator/1.0 (+ miggibot/2006.01.24MiggiBot website crawler engine - A1 Sitemap Generator
1616aardvark-crawlerAardvark web crawler for Sun's Blog recommendations
1617AbachoBOTAbacho / robot
1618AbachoBOT (Mozilla compatible)Abacho / robot
1619ABCdatos BotLink/ - Castilian program & tutorial directory
1620Aberja CheckomatAberja Hybridsuchmaschine (Germany) link checking
1621abot/0.1 (abot;; copyright search robot (24.177.134.x)
1622About/0.1libwww-perl/5.47About robot
1623Accelatech RSSCrawler/0.4Accela Technology RSS feed crawler
1624accoonaAccoona Search robot
1625Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)Accoona Search robot
1626Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)Accoona Search robot
1627Ace ExplorerAce Explorer - IE based browser
1628Ack ( search robot
1629AcoiRobotAcoi picture finder robot
1630Acoon Robot search (Germany) robot
1631Acoon Robot v1.52 ( search (Germany) robot
1632Acoon-Robot 4.0.x.[xx] ( search (Germany) robot
1633Acoon-Robot v3.xx ( and search (Germany) robot
1634Acorn/Nutch-0.9 (Non-Profit Search Engine;; acorn at isara dot org)Acorn Search Project
1635ActiveBookmark Active Bookmark HTML page creator
1636ActiveworldsActiveworlds 3D homepage browser
1637ActiveWorlds/3.xx (xxx)Activeworlds 3D homepage browser
1638Ad Muncher v4.xx.xAd Muncher - banner killer
1639Ad Muncher v4x Build xxxxxAd Muncher - banner killer
1640Adaxas Spider ( directory adaxas link checking
1641Advanced Browser ( Browser - IE based browser
1642AESOP_com_SpiderManAesop robot
1643agadine/1.x.x (+ search (Germany) robot
1644Agent-SharewarePlazaFileCheckBot/2.0+(+ link checking
1645AgentName/0.1 libwww-perl/5.48Linkomatic submission verifier
1646AIBOT/2.1 By +( A Real artificial intelligence search engine China) (China) robot (
1647AideRSS/1.0 ( - Postrank RSS and Blog filtering
1648aipbot/1.0 (aipbot;; copyright search robot (24.177.134.x)
1649aipbot/2-beta (aipbot dev;; copyright search robot (24.177.134.x)
1650Akregator/1.2.9; librss/remnantsAkregator news feed reader for KDE
1651Aladin/3.324Aladin robot
1652Alcatel-BG3/1.0 UP.Browser/ UP.Browser for mobiles on Alcatel cellphone
1653Aleksika Spider/1.0 (+ Danmark - Search engine optimization spider
1654AlertInfo 2.0 (Powered by Newsbrain)Alertinfo - French version of Feedreader 3.xx
1655AlkalineBOT/1.3Vestris robot
1656AlkalineBOT/1.4 (1.4.0326.0 RTM)Vestris robot
1657Allesklar/0.1 libwww-perl/ robot
1658Alligator 1.31 ( download manager
1659Allrati/1.1 (+)Unknown robot from
1660AltaVista Intranet V2.0 AVS EVAL search@freeit.comAltavista robot
1661AltaVista Intranet V2.0 Compaq Altavista Eval sveand@altavista.netAltavista robot
1662AltaVista Intranet V2.0 crawler@evreka.comAltavista robot
1663AltaVista V2.0B crawler@evreka.comAltavista robot
1664amaya/x.xx libwww/x.x.xAmaya - W3C's Editor/Browser
1665AmfibiBOTAmfibi Search robot
1666Amfibibot/0.06 (Amfibi Web Search;; Search robot
1667Amfibibot/0.07 (Amfibi Robot;; Search robot
1668amibotamibot - Amidalla search engine robot (62.241.33.xx)
1669Amiga-AWeb/3.4.167SEAWeb Amiga browser
1670AmigaVoyager/3.4.4 (MorphOS/PPC native)Voyager - Amiga browser
1671AmiTCP Miami (AmigaOS 2.04)Amiga Miami TCP Stack
1672Amoi 8512/R21.0 NF-Browser/3.3NF embedded browser on Amois Skypephone robot for checking their affiliate sites
1674AnnoMille spider 0.1 alpha - http://www.annomille.itAnnomille Italian historical oriented robot
1675annotate_google; Google - Firefox extension for annotating Google search results
1676Anonymized by ProxyOS: http://www.megaproxy.comMegaproxy user
1677Anonymizer/1.1faked user agent
1678AnswerBus ( natural language search using COLLATE technology
1679AnswerChase PROve x.0AnswerChase search tool
1680AnswerChase x.0AnswerChase search tool
1681ANTFresco/x.xxANT Fresco Browser
1682antibot-V1.1.5/i586-linux-2.2Antibot (discontinued) robot
1683AnzwersCrawl/2.0 (;Engine)Anzwers (Yahoo) Australia robot
1684Apexoo Spider 1.xApexoo Search spider
1685Aplix HTTP/1.0.1JavaOS app. for SEGA Saturn Internet and Sanyo Internet-TV
1686Aplix_SANYO_browser/1.x (Japanese)JavaOS app. for Sanyo Internet-TV
1687Aplix_SEGASATURN_browser/1.x (Japanese)JavaOS app. for SEGA Saturn Internet
1688AportAport robot
1689appie 1.1 ( Internet Search robot
1690Apple iPhone v1.1.4 CoreMedia v1.0.0.4A102CoreMedia player on Apple iPhone
1691Apple-PubSub/65.1.1PubSub - Mac OS X utility for managing RSS/Atom subscriptions via the PubSub framework
1692ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4;;)Araby search - Arabia
1693ArachBotCovac Arachnid Web Crawler
1694Arachnoidea ( spider robot for checking their affiliate sites
1696ArchitextSpiderExcite spider
1697archive.org_botHeritrix - The Internet Archive's open-source crawler (
1698Argus/1.1 (Nutch;; feedback at simpy dot com)Simpy Bookmarklet crawler (69.55.233.xx)
1699Arikus_SpiderArikus inContext search engine software
1700Arquivo-web-crawler (compatible; heritrix/1.12.1 + project: the Portuguese web archive
1701ASAHA Search Engine Turkey V.001 ( search robot (Turkey)
1702Asahina-Antenna/1.xASAHINA Antenna information detecting agent
1703Asahina-Antenna/1.x ( ; Antenna information detecting agent
1704ask.24x.infoAsk 24x Info robot
1705AskAboutOil/0.06-rcp (Nutch;; About Oil - Petroleum related search ( using Nutch
1706asked/Nutch-0.8 (web crawler;; epicurus at gmail dot com)askEd! / Inferret search (Japan) robot using Nutch
1707ASPSeek/1.2.5ASPSeek search engine software -Yahoo-Inc. / Telecom Canada robot
1708ASPseek/ robot using Aspseek
1709ASPSeek/1.2.xASPSeek search engine software
1710ASPSeek/1.2.xaASPSeek search engine software
1711ASPseek/1.2.xxASPSeek search engine software
1712ASPSeek/1.2.xxpreASPSeek search engine software
1713ASSORT/0.10Associative Sort robot
1714asterias/2.0Singingfish media spider ( via AOL search
1715AtlocalBot/1.1 +( local business search robot
1716Atomic_Email_Hunter/4.0Atomic Email Hunter email extracing and harvesting
1717Atomz/1.0Atomz robot
1718atSpider/1.0atSpider (ceased) email harvester / spambot
1719Attentio/Nutch-0.9-dev (Attentio's beta blog crawler;; social media monitoring and analysing
1720AU-MIC/2.0 MMP/2.0Samsung SPH-A660 phone with Sprint software
1721AUDIOVOX-SMT5600Audiovox SMT5600 (AT&T) Smartphone mobile phone browser
1722augurfindAugurnet Swiss (was search robot
1723augurnfind V-1.xAugurnet Swiss (was search robot
1724autoemailspiderAuto Email Pro Email harvester
1725autohttpLinkscan tool from Elsop
1726autowebdir 1.1 ( - The Automatically Generated Web Directory
1727AV Fetch 1.0Altavista robot ??
1728Avant Browser ( Browser - IE based browser
1729AVSearch-1.0( Research Council Canada robot
1730AVSearch-2.0-fusionIdx-14-CompetitorWebSitesUnknown robot from 205.203.108.xx (
1731AVSearch-3.0(AltaVista/AVC)Altavista robot
1732AWebAWeb Amiga browser
1733axadine/ (Axadine Crawler;; )Axada search Germany robot
1734AxmoRobot - Crawling your site for better indexing on search engine.Axmo search robot
1735Azureus 2.x.x.xAzureus Java BitTorrent Client
1736BabalooSpider/1.3 (BabalooSpider;; search robot (Slovenia)
1737BaboomBot/1.x.x (+ Web Portal (ODP) robot (66.98.254.xx)
1738BackStreet Browser 3.xBackStreet Browser - Offline browser / website downloader
1739BaiduImagespider+(+ search (Japan) image crawler
1740BaiDuSpiderBaidu spidering engine - used by diff. IPs
1741Baiduspider+(+ spidering engine - used by diff. IPs
1742Baiduspider+(+ spidering engine - used by diff. IPs
1743Baiduspider+(+ search (Japan) crawler
1744Balihoo/Nutch-1.0-dev (Crawler for search engine - obeys robots.txt and robots meta tags ;; robot at balihoo dot com)Balihoo - Vertical search engine crawler (beta)
1745BanBots/1.2 ( BanBots Perl script robot
1746Barca/2.0.xxxxBarca Pro email & PIM software
1747BarcaPro/1.4.xxxxBarca Pro email & PIM software
1748BarraHomeCrawler ( crawler
1749bCentral Billing Post-ProcessUnknown user agent from Microsoft
1750bdcindexer_2.6.2 (research@bdc) robot
1751BDFetchBrandimensions Brand Protection robot
1752BDNcentral Crawler v2.3 [en] ( (X11; I; Linux 2.0.44 i686)Bdncentral Sitesearch robot
1753BeamMachine/0.5 (dead link remover of web directory (Germany) link checking
1754beautybot/1.0 (+ robot for Cosmoty - German beauty and wellness search
1755BebopBot/2.5.1 ( crawler )A Passion for Jazz music related search robot
1756BeebwareDirectory/v0.01LinkcheckerBeepware (site is down) web directory link checking
1757Big Brother ( Brother link checking tool
1758Big Fish link popularity checking
1759BigBrother/1.6eBB4 network monitoring
1760BigCliqueBOT/1.03-dev (bigclicbot;; Search robot
1761BIGLOTRON (Beta 2;GNU/Linux)Biglotron search (France) robot ( Internet Spider;; search robot
1763Bilbo/2.3b-UNIXBilbo - web frontend for the Nessus Security Scanner
1764BilgiBetaBot/0.8-dev ( (Beta) ;; (Beta) search robot - Turkey
1765BilgiBot/1.0(beta) (; bilgi at bilgi dot com) (Beta) search robot - Turkey
1766billbot wjj@cs.cmu.eduCarnegie Mellon School robot/link checking ?
1767Bitacle bot/1.1Bitacle Blog Search Archive robot
1768Bitacle Robot (V:1.0;) ( Blog Search Archive robot
1769Biyubi/x.x (Sistema Fenix; G11; Familia Toledo; es-mx)Biyubi Navigator - Mexican browser for Fenix OS
1770BlackBerry7520/4.0.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/ UP.Link/ (Google WAP Proxy/1.0)Blackberry Wireless Internet browser via Google WAP Proxy
1771BlackWidowFS Consulting (was Black Widow web crawler
1772BlackWidowBlackWidow web site scanner / downloading tool
1773Blaiz-Bee/1.0 (+ Enterprises RawGrunt search
1774Blaiz-Bee/2.00.8222 (BE Internet Search Engine Enterprises RawGrunt search
1775Blaiz-Bee/2.00.xxxx (+ Enterprises RawGrunt search
1776BlitzBOT@tricus.netBlitzsuche Germany robot (Mozilla compatible)Blitzsuche Germany robot
1778BlockNote.NetBlockNote web page editor
1779BlogBot/1.xblogdex robot from
1780BlogBridge 2.13 ( RSS reader
1781Bloglines Title Fetch/1.0 ( article search
1782Bloglines-Images/0.1 ( graphics crawler
1783Bloglines/3.1 ( news crawler
1784BlogMap ( / BlogMap geo coding service
1785Blogpulse ('s BlogPulse blog search
1786BlogPulseLive ('s BlogPulse blog search
1787BlogSearch/1.x + Web search robot
1788blogsearchbot-pumpkin-3Art of Computing blog search project
1789BlogsNowBot, V 2.01 (+ realtime link tracker robot
1790BlogVibeBot-v1.1 ( news and Blog crawler (Netherlands)
1791blogWatcher_Spider/0.1 ( robot from Okumura Group Tokyo (
1792BlogzIce/1.0 (+; Web search robot
1793BlogzIce/1.0 + Web search robot
1794BloobyBotBlooby search (beta) robot
1795Bloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots meta tags ;; robot at balihoo dot com)Balihoo - Search Engine for Advertising Media
1796bluefish 0.6 HTML editorBluefish HTML-editor for Linux
1797BMCLIENTPart of ButtMan remote access tool
1798BMLAUNCHERBookmark Express bookmark manager
1799Bobby/4.0.x RPT-HTTPClient/0.3-3EBobby web accessibility desktop testing tool ( search (Norway) robot via 80.202.212.xx / 80.80.111.xx search (Norway) robot via 80.202.212.xx / 80.80.111.xx ( search (Norway) robot via 80.202.212.xx / 80.80.111.xx
1803Bookdog/x.xBookdog - Mac bookmark manager
1804Bookmark Buddy bookmark checker ( Buddy - favorite bookmark manager
1805Bookmark Renewal Check Agent []Favourites managing program
1806Bookmark Renewal Check Agent [] (Version 2.0beta)Favourites managing program
1807BookmarkBase(2/; Base bookmark manager
1808Bot user robot / faked user agent ?
1809BPImageWalker/2.0 ( copyright infringement crawler
1810BravoBrian bstop.bravobrian.itBravoBrian bSTOP parental control
1811BravoBrian SpiderEngine MarcoPoloRobot for BravoBrian bSTOP
1812BrightCrawler ( web filtering for classifying websites
1813BruinBot (+ Webarchive Project Bruinbot crawler
1814BSDSeek/1.0Inktomi (Hotbot-Lycos NBCi) robot Agent DetectorBravoBrian bSTOP parental control
1816BTbot/0.x (+ Search Engine btbot robot
1817BTWebClient/180B(9704)µTorrent BitTorrent Client
1818BuildCMS crawler ( crawler - market monitoring project of BuildCMS
1819Bulkfeeds/r1752 ( RSS directory link checking
1820BullsEyeBullsEye/Intelliseek robot
1821bumblebee@relevare.comRelevare Portal software robot
1822BunnySlippersMicrosoft server information robot (see link)
1823BurstFindCrawler/1.1 (;; search crawler (64.34.172.xx)
1824Buscaplus Robi/1.0 ( (Spain) robot
1825BW-C-2.0Logitech Desktop Managers (LDM) Backweb (BW) update check
1826bwh3_user_agentBasic Web Hacking 3 fake user-agent from Hellbound Hackers challenges
1827Cabot/Nutch-0.9 (Amfibi's web-crawling robot;; Search robot
1828Cabot/Nutch-1.0-dev (Amfibi's web-crawling robot;; Search robot
1829CamelHttpStream/1.0Evolution integrated mail solution Camel TCP stream class
1830Cancer Information and Support International;Some user agent
1831carleson/1.0Cosmix project crawler (204.14.48.x /
1832Carnegie_Mellon_University_Research_WebBOT-->PLEASE READ--> Mellon University WebBOT
1833Carnegie_Mellon_University_WebCrawler Mellon University WebBOT
1834Catall search & web directory (Germany)
1835CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler;; Cazoodle search based on MetaQuerier
1836CCBot/1.0 (+ Foundation search crawler
1837ccubee/x.xEmpyreum Ccubee (Czech) search engine solution
1838CDR/1.7.1 Simulator/0.7(+ Profile/MIDP-1.0 Configuration/CLDC-1.0Timewe mobile browser (WAP) simulator (Japan)
1839CE-PreloadCisco Content Engine
1840CentiverseBotNordic semantic search engine
1841CentiverseBot - investigatorNordic semantic search engine
1842CentiverseBot/3.0 ( semantic search engine
1843Ceramic Tile Installation Guide ( robot (link ckecking ??)
1844CERN-LineMode/2.15CERN Line Mode Browser
1845cfetch/1.0Cosmix project crawler (204.14.48.x /
1846CFNetwork/x.xMaxOS X CoreFoundation CFNetwork API
1847cg-eye interactivecg-eye CGI checker
1848Charon/1.x (Amiga)Charon Amiga download manager
1849Chat Catcher/1.0Chat Catcher blog monitoring robot
1850Checkbot/1.xx LWP/5.xxCheckbot link validation
1851CheckLinks/1.x.xChecklinks - Perl link checker
1852CheckUrlNTL user agent
1853CheckWebCheckWeb link validation
1854Chilkat/1.0.0 (+ HTTP component user-agent
1855China Local Browse 2.6Unknown spam bot from (
1856Chitika ContentHit 1.0Chitika Inc. Blog advertising
1857ChristCRAWLER Christcrawler (was
1858CHttpClient by Open Text CorporationCHttpClient - C++ class using WinInet
1859CipinetBot ( -Cipinet Search Engine Web Crawler
1860Cityreview Robot (+ regional search (Germany) link checking
1861CJ Spider/Commision Junction link checking spider
1862CJB.NET ProxyCJB Net anonymous socks proxy service (216.194.70.x)
1863ClariaBot/1.0Claria (ex Gator) SearchScout robot (64.152.73.xx)
1864Claymont.comClaymont Search robot
1865CloakDetect/0.9 (+ Search (Czech Republic) robot
1866Clushbot/2.x (+ clustered search robot
1867Clushbot/3.x-BinaryFury (+ clustered search robot
1868Clushbot/3.xx-Ajax (+ clustered search robot
1869Clushbot/3.xx-Hector (+ clustered search robot
1870Clushbot/3.xx-Peleus (+ clustered search robot
1871COAST WebMaster Pro/4.x.x.xx (Windows NT)COAST Webmaster - Web management and maintenance software
1872CoBITSProbeProposed Content-Based Image Tracking System (CoBITS) P2P crawler (v36) (Mac OS X; (social bookmarks manager) client for Mac OS X
1874Cogentbot/1.X (+ Search Bot from Cogent Software Solutions for unknown purposes
1875ColdFusionCold Fusion server used by various IPs i.e.: - NetWORLD web catalogue link checking
1876ColdFusion ( Fusion server used by Bookmark Tracker - online favourites managment
1877collage.cgi/1.xxWebCollage Syndicator graphics crawler/collector
1878combine/0.0Combine harvesting & indexing robot
1879Combine/2.0 harvesting & indexing robot
1880Combine/3 harvesting & indexing robot
1881Combine/x.0Combine harvesting & indexing robot
1882cometrics-bot, http://www.cometrics.decometrics Web Content Mining solution - Germany
1883Commerce Browser CenterWildsoft Germany (closed) Internet client system user-agent (was
1884complex_network_group/Nutch-0.9-dev (discovering the structure of the world-wide-web;; Complex Networks Groups Complex Network Analysis
1885Computer_and_Automation_Research_Institute_Crawler crawler@ilab.sztaki.huHungarian Academy of Sciences data mining search
1886Comrite/0.7.1 (Nutch;; Chinese Search Engine for Oversea Web Sites (69.248.26.xx)
1888ContactBot/0.2Probably E-Mail harvesting robot - same as LMQueueBot
1889ContentSmartzContentSmartz e-mail harvesting tools
1890contypeInternet Explorer versions 4.x and 5 plugin content
1891Convera Internet Spider V6.xConveras RetrievalWare Internet Spider (63.241.61.x)
1892ConveraCrawler/0.2Converas RetrievalWare Internet Spider (63.241.61.x)
1893ConveraCrawler/0.9d (+ RetrievalWare Internet Spider (63.241.61.x)
1894ConveraMultiMediaCrawler/0.1 (+ RetrievalWare Internet Spider (63.241.61.x)
1895CoolBotSuchmaschine21 (Germany) robot
1896Cooliris/1.5 CFNetwork/459 Darwin/10.0.0d3Cooliris photo and video browser
1897CoralWebPrx/0.1.1x (See Content Distribution Network
1898cosmos/0.8_( SA France robot
1899cosmos/0.9_( SA France robot
1900CoteoNutchCrawler/Nutch-0.9 (info [at] coteo [dot] com) - local French directory link checking
1901CougarSearch/0.x (+ robot
1902Covac TexAs ArachbotCovac Arachnid Web Crawler
1903CoverScout%203/3.0.1 CFNetwork/339.5 Darwin/9.5.0 (i386) (iMac5,1)CoverScout for iTunes - CD cover search tool
1904Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / Japan / Korea robot
1905Cowbot-0.1.x (NHN Corp. / +82-2-3011-1954 / Japan / Korea robot
1906CrawlConvera0.1 ( RetrievalWare Internet Spider
1907Crawlerunknown robot via
1908Crawler ( Comet Search robot via Findwhat (now Miva)
1909Crawler / Abacho robot
1910Crawler V 0.2.x / Abacho robot
1911crawler@alexa.comAlexa crawler
1912CrawlerBoy Pinpoint.comPinpoint WAP search robot
1913Crawllybot/0.1 (Crawllybot; +; Beta search - Germany (
1914CreativeCommons/0.06-dev (Nutch;; Commons using Nutch open source robot
1915Cricket-A100/1.0 UP.Browser/ (GUI) MMP/2.0Cricket A100 cell phone browser
1916CrocCrawler vx.3 [en] ( (X11; I; Linux 2.0.44 i686)Croccrawler robot
1917csci_b659/0.13Web mining project from CSCI 659 (computer science course) at Indiana Univerity
1918CSE HTML Validator Professional ( HTML Validator for Windows
1919Cuam Ver0.050bxCuam - IE based browser
1920Cuasarbot/0.9b (Spain) music / ringtone search spider
1921curl/7.10.x (i386-redhat-linux-gnu) libcurl/7.10.x OpenSSL/0.9.7a ipv6 zlib/1.1.4Curl file transferring tool
1922curl/7.7.x (i386--freebsd4.3) libcurl 7.7.x (SSL 0.9.6) (ipv6 enabled)Curl file transferring tool
1923curl/7.8 (i686-pc-linux-gnu) libcurl 7.8 (OpenSSL 0.9.6)Curl file transferring tool
1924curl/7.9.x (win32) libcurl 7.9.xCurl file transferring tool
1925CurryGuide SiteScan 1.1CurryGuide UK link check robot
1926Custo x.x ( web site spidering tool (link checking)
1927Custom Spider /1.0Bisnisseek (was robot
1928Cyberdog/2.0 (Macintosh; 68k)Cyberdog Mac Browser (was
1929CyberPatrol SiteCat Webbot ( LLC robot for web filtering software
1930CyberSpyder Link Test/2.1.12 ( Link Test software
1931CydralSpider/1.x (Cydral Web Image Search; image & site search spider
1932CydralSpider/3.0 (Cydral Image Search; image & site search spider
1933DA 3.5 ( download manager
1934DA 4.0Downloadaccelerator download manager
1935DA 4.0 ( download manager
1936DA 5.0Downloadaccelerator download manager
1937DA 7.0Downloadaccelerator download manager
1938DAP x.xDownload Accelerator Plus download manager
1939Dart Communications PowerTCPPowerTCP ActiveX control tool
1940DataCha0s/2.0Unknown bot from Kornet Korea ( scans for Perl Awstats
1941DataCha0s/2.0Unknown UA looking for Awstats Perl components
1942DataFountains/DMOZ DownloaderINFOMINE/iVia Scholary Internet Resource Collections robot
1943DataFountains/Dmoz Downloader ( Scholary Internet Resource Collections robot
1944DataFountains/DMOZ Feature Vector Corpus Creator ( Scholary Internet Resource Collections robot
1945DataparkSearch/4.47 (+ open source search engine
1946DataparkSearch/4.xx ( open source search engine
1947DataSpear/1.0 (Spider;; Directory robot (24.109.29.xx)
1948DataSpearSpiderBot/0.2 (DataSpear Spider Bot;; Directory robot (24.109.29.xx)
1949DatenBot( German security related search (
1950DaviesBot/1.7 ( robot
1951daypopbot/0.xDaypop blog - weblog - online mag search spider
1952dbDig( search engine
1953DBrowse 1.4bSome site scanning tool via diff. IPs i.e.: - ( - (
1954DBrowse 1.4dSome site scanning tool via diff. IPs i.e.: - (
1955DC-Sakura/x.xxDC-Sakura download manager
1956dCSbot/1.1unknown divine/ robot
1957DDDsome (website) downloading tool
1958dds explorer v1.0 betaUnknown user agent
1959de.searchengine.comBot 1.2 ( (Germany) submission checking / robot (84.73.57.xx)
1960DeadLinkCheck/0.4.0 libwww-perl/5.xxDLC Perl HTTP link checking
1961Deep Link Calculator v1.0Sootle Web Directory deep link checker
1962deepak-USC/ISIdeepak-USC/ISI robot from USC/Information Science Institute
1963DeepIndexDeepindex robot
1964DeepIndex ( )Zetbot search Belgium (213.41.128.xx) using Deepindex robot
1965DeepIndex ( robot
1966DeepIndexer.caDeepindex robot (via Canada) Mozilla/5.0 (compatible; Konqueror/3.4; FreeBSD) KHTML/3.4.2 (like Gecko) picture robot for thumbnail preview via Yahoo
1968DeleGate/9.0.5-fix1DeleGate application level gateway / proxy server
1969Demo Bot DOT 16bSome site scanning tool from (
1970Demo Bot Z 16bSome site scanning tool from 68.154.96.xx (
1971Denmex websearch ( Websearch robot/link checking
1972Der große BilderSauger 2.00uDataBecker Bilder Sauger (discontinued) web graphics downloader robot ( HtmlDocument ObjectDevComponents HTMLDocument Class Library for Visual
1975DiaGem/1.1 ( Japan web crawler
1976Diamond/x.0Claria (ex Gator) SearchScout robot (64.152.73.xx)
1977DiamondBotClaria (ex Gator) SearchScout robot (64.152.73.xx)
1978Digger/1.0 JDK/1.3.0rc3Diggit! robot
1979DigOut4UOpenPortal4U robot
1980DIIbot/ (site is offline) / Digital-Integrity robot
1981Dillo/0.8.5-i18n-miscDillo Web Browser
1982Dillo/0.x.xDillo Web Browser
1983disastrous/1.0.5 (running with Python 2.5.1;; - a link checker based on Python
1984DISCo Pump x.xDISCo Pump offline browser / website ripper
1985disco/Nutch-0.9 (experimental crawler;; robot from Discovery Engine Corp.
1986disco/Nutch-1.0-dev (experimental crawler;; robot from Discovery Engine Corp.
1987DittoSpyderDitto picture search robot
1988dlmansome download agent
1989dloader(NaverRobot)/1.0Naver Japan / Korea robot WebBot Link Ckeck Tool. Report abuse to: dnsr@dnsright.comDNS Right - Online DNS tools
1991DoCoMo/1.0/Nxxxi/c10NTT DoCoMo (Japan) robot
1992DoCoMo/1.0/Nxxxi/c10/TBNTT DoCoMo (Japan) robot
1993DoCoMo/1.0/P502i/c10 (Google CHTML Proxy/1.0)Google (216.239.39.x) proxy server
1994DoCoMo/2.0 P900iV(c100;TB;W24H11)NTT DoCoMo (Japan) robot
1995DoCoMo/2.0 SH901iS(c100;TB;W24H12),gzip(gfe) (via DoCoMo (Japan) proxy server (
1996DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; Search Japan robot (
1997DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0; Search Japan robot (
1998DocZilla/1.0 (Windows; U; WinNT4.0; en-US; rv:1.0.0) Gecko/20020804DocZilla - Mozilla-based SGML/XML/HTML- browser
1999dodgebot/experimentalunknown robot from AGMLAB Information Technologies (Information retrieval system ?)
2000DonutP; Windows98SEDonut P - Japanese IE based browser
2001Doubanbot/1.0 ( robot from douban search (China) - maybe image crawling
2002Download Demon/3.x.x.xDownload Demon/Netzip download manager
2003Download Druid 2.xDownload Druid IE plugin download manager
2004Download Express 1.0Download Express download manager
2005Download MasterDownload Master download manager
2006Download Ninja 3.0Download Ninja download manager (Japan)
2007Download WonderDownload Wonder download manager
2008Download-Tipp Linkcheck ( Germany link checking
2009Download.exe(1.1) (+ .NET based downloading tool
2010DownloadDirect.1.0Download Direct download manager
2011Dr.Web (R) online scanner: online virus scanner
2012Dragonfly File ReaderDragonfly CMS - Open Source content management system
2013Drecombot/1.0 ( Japan ( - Blog search ??
2014Drupal (+ - open source content management platform
2015DSurf15a 01Some site scanning tool via diff. IPs i.e.: - ( - (
2016DSurf15a 71Some site scanning tool via diff. IPs i.e.: - (
2017DSurf15a 81Some site scanning tool via diff. IPs i.e.: - (
2018DSurf15a VASome site scanning tool via diff. IPs i.e.: - ( - (
2019DTAAgentDTAAgent Java object for data collecting
2020dtSearchSpiderdt Search Spider software
2021Dual ProxyFourelle Venturi proxy server
2022DuckDuckBot/1.0; (+ Duck Go search crawler
2023Dumbot(version 0.1 beta - robot
2024Dumbot(version 0.1 beta - robot
2025Dumbot(version 0.1 beta) robot
2026e-sense 1.0 ea( e-Sense user research robot (website is offline)
2027e-SocietyRobot( Project (Japan) crawler (133.9.238.xx)
2028eApolloBot/2.0 (compatible; heritrix/2.0.0-SNAPSHOT-20071024.170148 + Opto's eApollo Flash based search engine - Taiwan []Earthcom (Czech Republic) search robot (194.108.39.xx) []Earthcom (Czech Republic) search robot (194.108.39.xx)
2031EasyDL/3.xxKeywen Encyclopedia Bot
2032EasyDL/3.xx Encyclopedia Bot
2033EBrowse 1.4bSome site scanning tool via diff. IPs i.e.: - (
2034eCatch/3.0eCatch (now Wysigot) offline browser
2035EchO!/ robot
2036Educate Search VxBSome site scanning tool via diff. IPs i.e.: - (
2037egothor/3.0a (+ text search engine robot - based on Egothor open source crawler
2038EgotoBot/4.8 (+ Search robot
2039ejupiter.comeJupiter searcg robot (206.191.49.xx)
2040EldoS TimelyWeb/3.xTimelyWeb web page monitoring tool
2041elfbot/1.0 (+ crawler for uChoose theme based search (Germany)
2042ELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; + - RSS search robot of Daum
2043ELinks (0.x.x; Linux 2.4.20 i586; 132x60)ELinks text mode browser
2044ELinks/0.x.x (textmode; NetBSD 1.6.2 sparc; 132x43)ELinks text mode browser
2045EmailSiphonSonic E-mail collector
2046EmailSpiderEmailSpider E-mail harvesting software
2047EmailWolf 1.00Trellian EMailWolf E-mail collector WebBotEmeraldShield spam and web filtration services WebBot ( spam and web filtration services
2050EMPAS_ROBOTEmpas search Korea robot (
2051EnaBot/1.x ( - Enaball semantic search project crawler
2052endo/1.0 (Mac OS X; ppc i386; - Mac news site and blog aggregator
2053Enfish TrackerEnfish Personal search tool
2054Enterprise_Search/1.0Enterprise Search engine software (
2055Enterprise_Search/1.0.xxxEnterprise Search engine software (
2056Enterprise_Search/;MSSQL ( Search engine software (
2057envolk/1.7 (+ Web Search robot
2058envolk[ITS]spider/1.6(+ Web Search robot
2059EroCrawlerEroCrawler adult search robot
2060ES.NET_Crawler/2.0 ( Search engine software (
2061eseek-larbin_2.6.2 ( (Jayde Online) robot
2062ESISmartSpiderESI Smart-Spider toolkit
2063eStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0)e-Style ISP search (Russia) robot (
2064ESurf15a 15Some site scanning tool via diff. IPs
2065EuripBot/0.x (+ - European Internet Portal robot
2066EuripBot/0.x (+ - European Internet Portal robot
2067EuripBot/0.x (+ - European Internet Portal robot
2068Eurobot/1.0 ( Euronet business directory robot
2069EvaalSE - bot@evaal.comEvaal Search Engine robot
2070eventax/1.3 (eventax;; event search (Germany)
2071Everest-Vulcan Inc./0.1 (R&D project; host=e-1-24; Inc. Everest crawler (in development)
2072Everest-Vulcan Inc./0.1 (R&D project; Inc. Everest crawler (in development)
2073Exabot-Images/1.0Exalead Websearch image crawler (193.47.80.xx)
2074Exabot-Test/1.0Exalead (France) search robot (193.47.80.xx)
2075Exabot/2.0Exalead (France) search robot (193.47.80.xx)
2076Exabot/3.0Exalead (France) search robot (193.47.80.xx)
2077ExactSearcheXact Search Bar for IE
2078ExactSeek Crawler/0.1ExactSEEK (Jayde Online) robot
2079exactseek-crawler-2.63 ( (Jayde Online) robot
2080exactseek-pagereaper-2.63 ( (Jayde Online) robot
2081exactseek.comExactSEEK (Jayde Online) robot (
2082Exalead NG/MimeLive Client (convert/http/0.120)Exalead (France) search robot (193.47.80.xx)
2083Excalibur Internet Spider V6.5.4Excalibur (now Convera) spider software
2084Execrawl/1.0 (Execrawl;; software search using nutch
2085exooba crawler/exooba crawler (crawler for;; info at exooba dot com)exooba crawler for exooba search pre-alpha development
2086exooba/exooba crawler (exooba; exooba)exooba crawler for exooba search pre-alpha development
2087ExperimentalHenrytheMiragoRobotMirago UK Robot
2088Expired Domain SleuthExpired Domain Sleuth domain name tool
2089Express WebPictures ( Web Pictures image browser
2090ExtractorProExtractor Pro e-mail collector
2091Extreme Picture FinderExisoftware image grabber and downloading tool
2092EyeCatcher ( Germany robot
2093Factbot 1.09 (see search robot
2094factbot : search robot
2095FaEdit/2.0.xFaEdit Professional - Japanese bookmark manager
2096FairAd ClientFairAd user
2097FANGCrawl/ web filtering service link verifierDrew Curtis' link checking
2099Fast Crawler Gold EditionFast/Alltheweb crawler (66.151.181.xx)
2100FAST Enterprise Crawler 6 (Experimental)Fast/Alltheweb crawler (66.151.181.xx)
2101FAST Enterprise Crawler 6 / Scirus; Enterprise Crawler (66.151.181.xx) for Scirus scienctific information search
2102FAST Enterprise Crawler 6 used by Cobra Development ( crawler (66.151.181.xx)
2103FAST Enterprise Crawler 6 used by Comperio AS ( Web Miner based on Fast ESP
2104FAST Enterprise Crawler 6 used by FAST (FAST)Fast/Alltheweb crawler (66.151.181.xx)
2105FAST Enterprise Crawler 6 used by Pages Jaunes ( Jaunes business search (France) robot using Fast Enterprise Crawler
2106FAST Enterprise Crawler 6 used by Web Crawler (search_comments\at\sensis\dot\com\dot\au)Fast/Alltheweb crawler for Australian search (66.151.181.xx)
2107FAST Enterprise Crawler 6 used by Singapore Press Holdings ( Search - Singapore related search using Fast crawler
2108FAST Enterprise Crawler 6 used by WWU ( Enterprise Crawler used by WWU (University of Muenster - Germany)
2109FAST Enterprise Crawler/6 ( crawler (66.151.181.xx)
2110FAST Enterprise Crawler/6.4 (helpdesk at crawler (66.151.181.xx)
2111FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0)Fast/Alltheweb crawler (66.151.181.xx)
2112FAST MetaWeb Crawler (helpdesk at fastsearch dot com)Fast/Alltheweb crawler (66.151.181.xx)
2113Fast PartnerSite CrawlerFast/Alltheweb crawler (66.151.181.xx)
2114FAST-WebCrawler/2.2.10 (Multimedia Search) (; multimedia crawler
2115FAST-WebCrawler/2.2.6 (; crawler
2116FAST-WebCrawler/2.2.7 (; crawler
2117FAST-WebCrawler/2.2.8 (; crawler
2118FAST-WebCrawler/3.2 testFast/Alltheweb crawler
2119FAST-WebCrawler/3.3 (; crawler
2120FAST-WebCrawler/3.4/Nirvana (; crawler
2121FAST-WebCrawler/3.4/PartnerSite (; crawler
2122FAST-WebCrawler/3.5 (atw-crawler at fast dot no; crawler
2123FAST-WebCrawler/3.6 (atw-crawler at fast dot no; crawler
2124FAST-WebCrawler/3.6/FirstPage (; crawler
2125FAST-WebCrawler/3.7 (atw-crawler at fast dot no; crawler
2126FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no; crawler
2127FAST-WebCrawler/3.8 (atw-crawler at fast dot no; crawler
2128FAST-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; crawler
2129FAST-WebCrawler/3.x MultimediaFast/Alltheweb multimedia crawler
2130FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)Fast/Alltheweb multimedia crawler
2131fastbot crawler beta 2.0 (+ search Germany crawler (
2132FastBug http://www.ay-up.comAy-Up geo sync search robot
2133FastCrawler 3.0.1 ( crawler (66.151.181.xx)
2134FastSearch Web Crawler for Verizon SuperPages ( crawler used by
2135Favcollector/2.0 ( Favicon collecting robot
2136FavIconizerFavIconizer - IE favorites icons refreshing tool crawler/0.6 ( (Germany) search robot
2138FavOrgZD's FavOrg favourites managing program
2139Favorites Checking ( Favorites checking tool
2140Favorites Sweeper v.2.03Favorites Sweeper bookmark checker
2141Faxobot/1.0FaXo Search robot (69.152.89.xx)
2142FDM 1.xFree Download Manager (FDM) download accelerator
2143FDM 2.xFree Download Manager (FDM) download accelerator
2144Feed Seeker Bot (RSS Feed Seeker Feed Seeker bot (
2145Feed24.comFeed24 news feed and blog search
2146Feed::Find/0.0xFeed::Find - Syndication feed (RSS/Atom) auto-discovery
2147Feedable/0.1 (compatible; MSIE 6.0; Windows NT 5.1)Feedable beta web based RSS service
2148FeedChecker/0.01Unknown robot from the University of Tokyo (157.82.157.xx)
2149FeedDemon/2.7 (; Microsoft Windows XP)FeedDemon RSS reader
2150Feedfetcher-Google-iGoogleGadgets; (+ news feed feetcher for iGoogle gadgets
2151Feedfetcher-Google; (+ Feedfetcher - RSS and Atom feed crawler
2152FeedForAll rss2html.php v2FeedForAll RSS feed robot
2153FeedHub FeedDiscovery/1.0 ( news feed personalization engine powered by mSpoke
2154FeedHub MetaDataFetcher/1.0 ( news feed personalization engine powered by mSpoke
2155Feedjit Favicon Crawler 1.0Feedjit news feed service favicon crawler
2156Feedreader 3.xx (Powered by Newsbrain)Newsbrain Feedreader3
2157Feedshow/x.0 (; 1 subscriber)FeedShow online RSS feed reader
2158FeedshowOnline ( online RSS feed reader
2159FeedZcollector v1.x (Platinum) - Feed (RSS, ATOM and RDF) capturing software
2160Felix - Mixcat Crawler (+ robot
2161fetch libfetch/2.0FreeBSD download tool
2162FFC Trap Door SpiderFrequent Finders spider via
2163Filangy/0.01-beta (Filangy;; search and bookmark service
2164Filangy/1.0x (Filangy;; search and bookmark service
2165Filangy/1.0x (Filangy;; search and bookmark service (+ Boost Network link checking
2167FileHound x.xFileHound download manager
2168Filtrbox/1.0filtrbox media content (news) monitoring
2169FindAnISP.com_ISP_Finder_v99aFind An ISP robot
2170Findexa Crawler ( business search (Norway) via Findexa
2171findlinks/ (+ NextLinks - German vocabulary and hyperlink search
2172FineBotFinesearch robot
2173Finjan-prefetchFinjan Vital Security Web Appliance security solution
2174Firefly/ robot
2175Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5) robot
2176Firefox ( robot from Czech Technical University Prague (147.32.141.xx)
2177Firefox_1.0.6 ( robot from Czech Technical University Prague (147.32.141.xx) Search - POC:firstgov.webmasters@gsa.govAT&T/Fast Search robot for FirstGov (U.S.Government) portal
2179firstsbotFirstsfind Germany robot / link checking
2180Flapbot/0.7.2 (Flaptor Crawler;; crawler at flaptor period com)Flaptor information retrieval solutions robot
2181FlashGetJetCar/FlashGet download manager
2182FLATARTS_FAVICOFlatArts Favorites Icon Tool
2183Flexum search service
2184Flexum/ search service
2185FlickBot 2.0 RPT-HTTPClient/ Movie Find robot
2186flunkyMetacarta / Cogent robot
2187fly/6.01 libwww/4.0Dunknown 1.0/http://www.flyindex.netFLY Index Metasearch link checking
2189FnooleBot/2.5.2 (+ news crawler
2190FocusedSampler/1.0IBM's Almaden Research robot Spider/0.1 beta 1 ( social search robot
2192FollowSite Bot ( )FollowSite robot - website monitoring ( )FollowSite robot - website monitoring; search Scorpionbots robot (
2195Francis/1.0 ( Search (Germany) robot (
2196Franklin Locator 1.8Some spam bot download-link validator /0.1Free Downloads shareware directory link checking (; robot
2199Frelicbot/1.0 + backlink checking bot (beta)
2200FreshDownload/x.xxFresh Download download manager
2201FreshNotes crawler< report problems to crawler-at-freshnotes-dot-comFreshNotes - music related artist search (72.3.225.xx)
2202FSurf15a 01Some site scanning tool via diff. IPs
2203FTB-Bot the Best search robot
2204Full Web Bot 0416BSome site scanning tool from diff. IPs i.e.: - 66.28.240.xx ( - 68.5.174.xx (
2205Full Web Bot 0516BSome site scanning tool i.e. from - 68.154.96.xx (
2206Full Web Bot 2816BSome site scanning tool from (
2207FuseBulb.ComFuseBulb search
2208FyberSpider (+ FyberSpider robot
2209GagglebotSome user from ( using an Innerprise robot tool
2210GAIS Robot/1.0B2Seed Search robot
2211Gaisbot/3.0 (; Taiwan robot
2212Gaisbot/3.0+(;+ Taiwan robot
2213GalaxyBot/1.0 ( robot (
2214Gallent Search Spider v1.4 Robot 2 ( Search directory (UK)
2215gamekitbot/1.0 (+ game search engine - Germany
2217GammaSpider/1.0GammaWare GammaSpider
2218gazz/x.x ( / robot
2219geckobotGeckobot user robot
2220Generic Mobile Phone (compatible; Googlebot-Mobile/2.1; + Mobile Search crawler
2221generic_crawler/01.0217/Unknown robot from Carnegie Mellon University (
2222GenesisBrowser (HTTP 1.1; 0.9; XP SP2; .NET CLR 2.0.50727)Lunascape Genesis browser
2223genieBot ( search
2224geniebot search
2225GeoBot/1.0Unknown robot from (
2226GeonaBot 1.x; Search robot / link checking
2227geourl/2.0b2GeoURL ICBM Address Server - a location-to-URL reverse directory
2228GeoURLBot 1.0 ( ICBM Address Server - a location-to-URL reverse directory
2229GetBotGetbot web downloading tool / site grabber
2230GetRight/3.x.xGetRight download manager
2231GetRight/4.5xxGetRight download manager
2232GetRight/4.xGetRight download manager
2233GetRight/4.x[a-e]GetRight download manager
2234GetRight/6.1 (Pro)GetRight download manager
2235GetRightPro/6.0beta2GetRight download manager
2236GetWeb/0.1 libwww-perl/5.16GetWeb - web page to email service
2237GhostRouteHunter/20021130 (; Ghost Route Hunter
2238gigabaz/3.1x (; Brainbot (Germany) robot
2239Gigabot/2.0 ( robot (64.62.168.xx)
2240Gigabot/2.0/ robot
2241Gigabot/2.0; robot
2242Gigabot/2.0attGigablast robot
2243Gigabot/3.0 ( robot
2244Gigabot/x.0Gigablast robot (64.62.168.xx)
2245GigabotSiteSearch/2.0 ( robot (64.62.168.xx)
2246GNODSPIDER ( spider
2247Go!Zilla 3.x (!Zilla download manager
2248Go!Zilla/4.x.x.xxGo!Zilla download manager
2249Go-Ahead-Got-It/1.1GotIt web accelerator (discontinued)
2250Goblin/0.9 ( (195.226.137.xx) robot
2251Goblin/0.9.x ( (195.226.137.xx) robot
2252GoForIt.comGoForIt Search robot
2253GOFORITBOT ( )GoForIt Search robot
2254GoGuides.Org Link directory & search link checking
2255GoldenFeed Spider 1.0 ( - RSS search engine
2256Goldfire ServerInvention Machines Goldfire Server
2257gonzo1[P] + German local search robot
2258gonzo2[P] + German local search robot
2259Goofer/0.2Some private robot ( client)
2260Google TalkGoogle instant messenger
2261googlebot (larbin2.6.0@unspecified.mail)Packard Bell Net user robot (*not* Google)
2262Googlebot-Image/1.0Google image crawler (
2263Googlebot-Image/1.0 ( image crawler (
2264Googlebot/2.1 ( robot 66.249.64.XXX
2265Googlebot/2.1 ( robot 66.249.64.XXX
2266Googlebot/Test ( robot 66.249.64.XXX
2267Gordon's Spider/Nutch-0.9 (; social networking service via Amazon Web Services
2268GrapeFX/0.3 libwww/5.4.0Grapeshot web search system API
2269great-plains-web-spider/flatlandbot (Flatland Industries Web Spider;; Industries vertical search solution
2270GreatNews/1.0GreatNews 1.0 Beta RSS reader
2271GreenBrowserGreenBrowser - IE based browser (China)
2272gridwell ( gridwell favicon display
2273GrigorBot 0.8 ( Search bot
2274Gromit/1.0Australasian Legal Information Institute (AustLII) robot
2275grub crawler( open source crawler
2276grub-clientGrub open source crawler
2277gsa-crawler (Enterprise; GID-01422; Search Appliance robot (216.239.xx.xx)
2278gsa-crawler (Enterprise; GID-01742; Search Appliance robot (216.239.xx.xx)
2279gsa-crawler (Enterprise; GIX-02057; Enterprise Search Appliance used by Enhesa (
2280gsa-crawler (Enterprise; GIX-03519; Enterprise Search Appliance used by IBM (
2281gsa-crawler (Enterprise; GIX-0xxxx; Search Appliance robot (216.239.xx.xx)
2282GSiteCrawler/v1.xx rev. xxx ( - Google sitemap generator for Windows
2283Guestbook Auto SubmitterGuestbook spamming tool
2284Gulliver/1.3Northernlight robot
2285Gulper Web Bot 0.2.4 ( Collaborative Web Resource Categorization and Ranking Project robot
2286Gungho/0.08004 ( - Extensible web crawler written in Perl by Google Code
2287GurujiBot/1.0 (+ : the Indian search engine robot
2288GurujiImageBot/1.0 (+ : the Indian search engine picture crawler
2289Haier-T10C/1.0 iPanel/2.0 WAP2.0 (compatible; UP.Browser/; UPG1; UP/4.0; Embedded)Openwave Mobile Browser on Haier T10C mobile
2290HappyFunBot/1.1Happy Fun Search robot
2291Harvest-NG/1.0.2Harvest-NG web crawler used by
2292Haste/0.12 (HOME: - web mapping and monitoring system
2293Hatena Antenna/0.4 ( Japan robot
2294Hatena Mobile Gateway/1.0Hatena Japan proxy for handheld/mobile clients
2295Hatena Pagetitle Agent/1.0Hatena Japan robot
2296Hatena RSS/0.3 ( Japan RSS feed robot
2297HatenaScreenshot/1.0 (checker)Hatena::Diary (Japan) web page screenshot robot
2298hbtronix.spider.2 -- - Domain name spider (Germany)
2299HeinrichderMiragoRobotMirago Germany robot
2300HeinrichderMiragoRobot ( Germany robot
2301Helix/1.x ( - The SiteSearch (Canada) web crawler
2302HenriLeRobotMirago ( France robot
2303HenrytheMiragoRobotMirago search (UK) robot
2304HenryTheMiragoRobot ( search (UK) robot
2305hgrepurl/1.0O'Reilly's Perl LWP example client program from Web Client Programming with Perl
2306Hi! I'm CsCrawler my homepage: RPT-HTTPClient/0.3-3University of Kassel Germany CsCrawler using the HTTPClient library
2307HiDownloadHiDownload download manager
2308Hippias/0.9 BetaHippias robot
2309HitListPilot Hitlist web analytics solution
2310Hitwise Spider v1.0 http://www.hitwise.comHitwise spider
2311HLoaderdiff. IPs / unknown services
2312holmes/3.11 ( / Centrum Search (Czech Republic) robot from
2313holmes/3.9 ( (Poland) search robot
2314holmes/3.xx (OnetSzukaj/5.0; + (Poland) search robot
2315holmes/x.xMorfeo / Centrum Search (Czech Republic) robot from
2316HolmesBot ( search robot (Georgia)
2317HomePageSearch( robot
2318Homerbot: www.homerweb.comHomerweb search robot
2319Honda-Search/0.7.2 (Nutch;; - Honda cars related search robot
2320HooWWWer/2.1.3 (debugging run) (+ | - Next Generation Information Retrieval robot
2321HooWWWer/2.1.x ( | - Next Generation Information Retrieval robot (128.214.112.xx)
2322HotJava/1.0.1/JRE1.1.xHotJava browser plus HTML Component 1.1.x
2323Hotzonu/x.0Hotzuno - Japanese BBS reader client
2324HPL/Nutch-0.9 -Unknown robot from HP Labs
2325htdig/3.1.6 ( robot ( using htdig
2326htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer)htdig used by the Academie de Toulouse
2327htdig/3.1.x (root@localhost)htdig search tool
2328Html Link Validator ( Software link validation tool
2329HTML2JPG Blackbox, http://www.html2jpg.comHTML2JPG webpage to image converter
2330HTML2JPG EnterpriseHTML2JPG webpage to image converter
2331HTMLParser/1.xHTML Parser Java library to parse HTML
2332HTTP RetrieverPHP HTTP client to access Web servers
2333 (Unix)Anonymous web proxy service
2334http://Ask.24x.Info/ ( 24x Info (Germany) DMOZ related robot
2335 ACONTBOTACONTBOT - Acont search Germany robot
2336 software directory link checking
2337's Almaden Research robot
2338 []IBM's Almaden Research robot
2339 [wf216]IBM's Almaden Research robot
2340 search robot
2341http://www.monogol.deMonogol - German open source search engine project (
2342 Search Engine (Denmark) robot
2343HTTP::Lite/2.x.xHTTP::Lite - Standalone Perl module for retreiving HTTP documents
2344HTTPEyesHTTPEyes - Web proxy cache
2345HTTPResume v. 1.xHTTPResume Amiga download manager
2346httpunit/1.5HttpUnit - Java test code for emulating browser behaviour
2347httpunit/1.xHttpUnit - Java browser behavior simulation tool
2348Hybrid/1.2 [en] (OS Independent)Hybrid Share mono C#/Gtk# application for file sharing
2349HyperEstraier/1.x.xxHyper Estraier full-text search system
2350i1searchbot/2.0 (i1search web crawler;; robot
2351IAArchiver-1.0Alexa / The Internet Archive (
2352iaskspiderUnknown robot (reads robots.txt) from chinatelecom (219.142.78.xx)
2353iaskspider2 ( search / Sina portal robot (China)
2354ia_archiverAlexa / The Internet Archive (
2355ia_archiver-web.archive.orgAlexa / The Internet Archive (
2356ia_archiver/1.6Alexa / The Internet Archive (
2357IBrowse/2.2 (AmigaOS 3.5)IOSpirit iBrowse Amiga Browser
2358IBrowse/2.2 (Windows 3.1)IOSpirit iBrowse Amiga Browser
2359iCab/2.5.2 (Macintosh; I; PPC)iCab MAC Web browser
2360ICC-Crawler(Mozilla-compatible;; icc-crawl(at)ml(dot)nict(dot)go(dot)jp)Knowledge Clustered Group ICC-Crawler (University of Tokyo - Japan)
2361ICC-Crawler(Mozilla-compatible;;icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp)Knowledge Clustered Group ICC-Crawler (University of Tokyo - Japan)
2362iCCrawler ( - Intelligence Competence Center (Germany) robot
2363ICCrawler - ICjobs ( - Intelligence Competence Center (Germany) robot
2364ICE Browser/5.05 (Java 1.4.0; Windows 2000 5.0 x86)ICE Java browser
2365ichiro/x.0 ( Japan / Inktomi robot (210.173.179.xx)
2366ichiro/x.0 ( Japan / Inktomi robot (210.173.179.xx)
2367IconSurf/2.0 favicon finder (see - Visual Surf Engine / favicon finder
2368IconSurf/2.0 favicon monitor (see - Visual Surf Engine / favicon finder
2369ICOO Loader v.x.x.xicooLoader download manager
2370ICRA_label_spider/x.0ICRA (Internet Content Rating Association) label spider
2371icsbot-0.1ICS Robot Search Engine (International Christian school of Seoul)
2372IDAInternet Download Accelerator
2373ideare - SignSite/1.xJanas ( / robot
2374iearthworm/1.0, UA from Yahoo China
2375IEFav172FreeSome bookmark manager (; 0 subscribers) - online rss aggregator (in development)
2377igdeSpyder (compatible;; + search (Russia) robot
2378iGetter/1.x (Macintosh;G;PPC)iGetter download manager
2379iGetter/2 (Macintosh; U; PPC Mac OS X; en)iGetter download manager
2380IIITBOT/1.1 (Indian Language Web Search Engine;; pvvpr at iiit dot ac dot in)Webkhoj - Indian language search engine
2381ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. For more information please visit;; Knowledge Search robot
2382ilial/Nutch-0.9-devUnknown robot from UCLA using Nutch
2383IlseBot/1.xIlse Netherlands robot (62.69.178.xx)
2384IlTrovatore-Setaccio ( Trovatore - Italian search engine robot
2385Iltrovatore-Setaccio/0.3-dev (Indexing;; Trovatore - Italian search engine robot
2386IlTrovatore-Setaccio/1.2 ( Trovatore - Italian search engine robot
2387Iltrovatore-Setaccio/1.2 (It-bot;; Trovatore - Italian search engine robot
2388iltrovatore-setaccio/1.2-dev (spidering; Trovatore - Italian search engine robot
2389IlTrovatore/1.2 (IlTrovatore;; Trovatore - Italian search engine robot
2390ImageVisu/v4.x.xImageVisu image and graphics viewer - display files from the Web (HTTP and ECWP)
2391ImageWalker/2.0 ( copyright infringement crawler
2392Incutio HttpClient v0.xHttpClient - a PHP Web Client Class
2393IncyWincy data gatherer(webmaster@loopimprovements.comIncyWincy search engine using DMOZ Open Directory database
2394IncyWincy page crawler(webmaster@loopimprovements.comIncyWincy search engine using DMOZ Open Directory database
2395IncyWincy( robot using IncyWincy search engine
2396IncyWincy( search engine using DMOZ Open Directory database
2397IncyWincy/2.1( search engine using DMOZ Open Directory database Crawler7Index the Web (69.57.134.xx) crawler
2399Industry Program 1.0.xSpam bot from diff. IPs
2400Inet libraryInet Library Resource Center robot
2401InetURL/1.0InetURL IVM (phone software) plugin for web server access ? ( il Sud dei Motori di search - Italia
2403Infoaxe./Nutch-0.9Infoaxe - search history and bookmark service
2404infoConveraCrawler/0.8 ( RetrievalWare Internet Spider (63.241.61.x)
2405InfoFly/1.0 ( Ingelin spider
2406InfoLink/1.xInfoLink link checking tool
2407INFOMINE/8.0 AddersINFOMINE Scholary Internet Resource Collection crawler
2408INFOMINE/8.0 RemoteServicesINFOMINE Scholary Internet Resource Collection crawler
2409INFOMINE/8.0 VLCrawler ( Scholary Internet Resource Collection crawler
2410InfoNaviRobot(F107) Robot
2411InfoSeek Sidewinder/0.9Infoseek robot
2412InfoSeek Sidewinder/1.0AInfoseek robot
2413InfoSeek Sidewinder/1.1AInfoseek robot
2414Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX)Infoseek robot
2415Infoseek SideWinder/2.0B (Linux 2.4 i686)Infoseek Japan robot
2416INGRID/3.0 MT (; Netherlands robot (62.69.178.xx)
2417Inktomi SearchInktomi (Hotbot-Lycos NBCi etc.) robot
2418InnerpriseBot/1.0 ( Search engine software ( search and find world wide!Insitor Search robot (
2420InsitornautInsitor Search robot (
2421InstallShield DigitalWizarddownload manager
2422integrity/1.6Integrity - website broken link checker for MAC OSx
2423Intelix/0.x (cs;; Intelix robot for Eurotran translation software ?
2424Interarchy/x.x.x (InterarchyCrawler)Interarchy file transfer software - SFTP/FTP client for Mac OS X
2425Internet Ninja x.0Dream Train (Japan) Internet search robot
2426InternetArchive/0.8-dev(Nutch;;nutch-agent@lucene.apacheHeritrix - The Internet Archive's open-source crawler based on Nutch (
2427InternetLinkAgent/3.1Internet Link Agent - link checking tool
2428InternetSeer.comInternetseer Web site monitoring / Claymont robot
2429intraVnews/1.xintraVNews - Feed reader & RSS aggregator for Outlook
2430IOI/2.0 (ISC Open Index crawler;; Open Index crawler using Nutch
2431IP*Works! V5 HTTP/S Component - by /n software - www.nsoftware.comIP*Works! HTTP Component
2432http://www.ip2location.comIP2Location - Reverse lookup geographical data and ISP by IP
2433IP2MapBot/1.1 http://www.ip2map.comIP2Map - geographical IP mapping
2434IPiumBot laurion(dot)comLaurions Ipium robot
2435IpselonBot/0.xx-beta (Ipselon;; Web Search robot
2436Iria/1.xxaIria download manager
2437IRLbot/1.0 ( - Texas A&M University research project crawler
2438IRLbot/3.0 (compatible; MSIE 6.0; - Texas A&M University research project crawler
2439IrssiUrlLog/0.2url_log - Irssi Perl url grabber
2440Irvine/1.x.xIrvine downloading tool
2441ISC Systems iRc Search 2.1Unknown spambot / harvester from diff. IPs
2442iSiloX/4.xx Windows/32iSiloX document converter for iSilo reader
2443isurf ( University of Alberta link-checking ?
2444iTunes/x.x.xiTunes UA name for access and decrypt the iTunes music store pages
2445IUPUI Research Bot v 1.9aSome spam bot from 66.139.78.xx(x)
2446iVia Page Fetcher ( robot - Open source Internet portal & virtual library system software
2447iVia/4.0 CanonizeUrl ( robot - Open source Internet portal & virtual library system software
2448IWAgent/ 1.0 - www.brandprotect.comBD BrandProtect - brand, company or trademarks online monitoring
2449J-PHONE/3.0/J-SH07Proxy message from
2450Jabot/6.x ( Directory Japan robot (163.138.95.xx)
2451Jabot/7.x.x ( Directory Japan robot (163.138.95.xx)
2452JackGerman Domanova (offline since Feb.02) robot
2453Jakarta Commons-HttpClient/2.0xxxJakarta Commons (Java based) HTTP client
2454Jakarta Commons-HttpClient/3.0-rcxJakarta Commons (Java based) HTTP client
2455Jambot/0.1.x (Jambot;; search robot (70.146.82.xx)
2456Jambot/0.2.1 (Jambot;; search robot
2457Java 1.1Java VM
2458Java/1.4.1_01Java VM
2459Java1.0.21.0Java VM
2460Java1.1.xx.xJava VM
2461Java1.3.0rc1Java VM
2462Java1.3.xJava VM
2463Java1.4.0Java VM
2464Jayde Crawler. http://www.jayde.comJayde B2B Search robot (66.28.139.xx)
2465JBH Agent 2.0some site downloading tool ? via
2466jBrowser/J2ME Profile/MIDP-1.0 Configuration/CLDC-1.0 (Google WAP Proxy/1.0)WAP 2.0 / jBrowser for handhelds
2467JCheckLinks/0.1 RPT-HTTPClient/0.3-1JCheckLinks Java hyperlink validator
2468JDK/1.1Java Development Kit
2469Jeode/1.x.xInsignias Jeode (PDA) Java platform
2470Jetbot/1.0JetEye Search robot ( /
2471JetBrains Omea Reader 1.0.x ( RSS - Atom - newsgroups web page reader
2472JetBrains Omea Reader 2.0 Release Candidate 1 ( RSS - Atom - newsgroups web page reader
2473JetCarJetCar / Flashget download manager
2474Jigsaw/2.2.x W3C_CSS_Validator_JFouffa/2.0Jigsaw - W3C's CSS Validator Server
2475JoBo/1.x ( website downloading program
2476JoBo/@JOBO_VERSION@( website downloading program
2477JobSpider_BA/ - German (213.61.218.xx) job search JobRoboter
2478JOC Web SpiderJocsoft Web Spider - website downloading tool
2479JordoMedia/1.0 RSS File Reader ( Media RSS / Atom feed directory link checking
2480Journster [alpha] ( RSS/Atom aggregator RSS/Atom aggregator 0.5 ( RSS/Atom aggregator
2482JRTS Check Favorites UtilityCheck Favorites bookmark checking
2483JRTwine Software Check Favorites UtilityCheck Favorites bookmark checking
2484Jyxobot/xJyxo search (Czech Republic) robot (212.71.128.xx)
2485K-Meleon/0.6 (Windows; U; Windows NT 5.1; en-US; rv:0.9.5) Gecko/20011011K-meleon browser - Windows 2000
2486k2spiderVerity K2 Spider ( Network search software)
2487KAIST AITrc CrawlerUnknown robot from AITrc (Advanced Information Technology Research Center) - Korea
2488KakleBot - (KakleBot -; http://; ranked metasearch robot
2489kalooga/kalooga-4.0-dev-datahouse (Kalooga;; image crawler
2490kalooga/KaloogaBot (Kalooga;; image crawler
2491Kapere ( site grapper / web downloader
2492Kazehakase/0.x.x.[x]Kazehakase - Gecko based browser (Japan)
2493KDDI-SN22 UP.Browser/6.0.7 (GUI) MMP/1.1 (Google WAP Proxy/1.0)Openwave UP.Browser for mobiles via Google WAP Proxy (216.239.33.x)
2494Kenjin SpiderKenjin Spider search agent
2495Kevin Kevin crawler (link checking ?) via 68.39.148.xx (
2496Kevin AlertsKevin crawler (website monitoring) via 68.39.148.xx (
2497KE_1.0/2.0 libwww/ robot
2498KFSW-Bot (Version: 1.01 powered by KFSW Perl search script from KFSW (Germany)
2499kinja-imagebot ( weblog search robot
2500kinjabot ( weblog search robot
2501KIT-Fireball/2.0Fireball search (Germany) robot
2502KIT-Fireball/2.0 (compatible; Mozilla 4.0; MSIE 5.5)Fireball search (Germany) robot
2503Klondike/1.50 (WSP Win32) (Google WAP Proxy/1.0)Klondike WAP Browser
2504KnowItAll( of Washington KnowItAll - web information extraction (ODP) directory robot
2506Kontiki Client x.xxKontiki Client download manager
2507Krugle/Krugle,Nutch/0.8+ (Krugle web crawler;; source code search engine for developers (
2508KSbot/1.0 (KnowledgeStorm crawler;; technology industry crawler for
2509kuloko-bot/0.xKuloko contextual search robot
2510kulokobot kuloko@backweave.comKuloko contextual search robot
2511kulturarw3/0.1National Library of Sweden Heritage Project robot
2512KummHttp/1.1 (compatible; KummClient; Linux rulez)Link or server checking from Sanoma Budapest (
2513KWC-KX9/1109 UP.Browser/ (GUI) MMP/2.0 UP.Link/ for mobiles on Kyocera KWC-KX9 cellphone
2514Labrador/0.2;; Blog Track - Blog and news feed crawler
2515LachesisIntels Lachesis web site response time monitoring tool
2516lanshanbot/1.0Unknown robot from Easten Network China (
2517lanshanbot/1.0 (+ robot from Easten Network China (
2518LapozzBot/1.4 ( search (Hungary) robot (82.131.195.xx)
2519LapozzBot/1.5 (+ search (Hungary) robot (82.131.195.xx)
2520larbin ( indexer used as ( robot
2521LARBIN-EXPERIMENTAL ( robot from 66.230.140.xx (
2522larbin_2.1.1 larbin2.1.1@somewhere.comLarbin indexer used by Mitsubishi Electric Research Labs
2523larbin_2.2.0 ( indexer used as crawler
2524larbin_2.2.1_de_Viennot ( indexer used as Inria robot
2525larbin_2.2.2 ( indexer used as Kyoto University robot
2526larbin_2.2.2_guillaume ( indexer used as Inria robot
2527larbin_2.6.0 (larbin2.6.0@unspecified.mail)Larbin indexer used by an unknown client
2528larbin_2.6.1 (larbin2.6.1@unspecified.mail)Larbin indexer used by diff. IPs / services
2529larbin_2.6.2 ( indexer used by National Institut of Informatics (NII/Japan)
2530larbin_2.6.2 (larbin2.6.2@unspecified.mail)Larbin indexer used by diff. IPs
2531larbin_2.6.2 (listonATccDOTgatechDOTedu)Larbin indexer used as robot by Georgia Institute of Technology
2532larbin_2.6.2 ( indexer used by Technical University of Crete
2533larbin_2.6.2 ( indexer used as Lemur Consulting robot
2534larbin_2.6.2 ( indexer used as robot via (
2535larbin_2.6.3 ( robot from EPFL Lausanne Switzerland (
2536larbin_2.6.3 ( indexer used by search
2537larbin_2.6.3_for_( tsilande@hiit.fiLarbin indexer used by Next Generation Information Retrieval (NGIR)
2538larbin_2.6_basileocaml ( indexer used by CEA / DCom Rechercher
2539larbin_devel ( indexer used as Inria robot
2540lawinfo-crawler/Nutch-0.9-dev (Crawler for pages;; - Lawyer and attorney directory
2541lc/$ROADS::Version libwww-perl/5.00ROADS - Perl web based subject based gateway tool
2542lcabotAccept: */*unknown robot via /
2543LeapTag/0.8.1.beta081.r3750 (compatible; Mozilla 4.0; MSIE 5.5; news reader and content discovery tool
2544LECodeChecker/3.0 libgetdoc/1.0Linkexchange crawler
2545LeechGet 200x ( download manager
2546LEIA/ (site is offline) robot
2547LEIA/3.01pr (LEIAcrawler; [SNIP]) (site is offline) robot + logfile spamming for Lets crawl! search (Germany)
2549LexiBot/1.00Lexibot (exMataHari) search software
2550LG-LX260 POLARIS-LX260/2.0 MMP/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.1Polaris mobile browser on LG LX 260 Sprint Rumor phone
2551LG/U8138/v1.0LG 8138 Mobile Phone browser
2552Libby_1.1/libwww-perl/ robot
2553libcurl-agent/1.0libcurl's (multiprotocol file transfer library) standard user-agent name
2554LibertyW (+ search for mobile (France)
2555libWeb/clsHTTP -- Telecom Search robot
2556libwww-perl/5.41CMP United Media robot
2557libwww-perl/5.45SplatSearch robot (207.44.142.xx)
2558libwww-perl/5.48Alexa robot
2559libwww-perl/5.50diff. IPs / services
2560libwww-perl/5.52 FP/2.1Fast Search robot
2561libwww-perl/5.52 FP/4.0Fast Search robot
2562libwww-perl/5.53diff. IPs / services
2563libwww-perl/5.63Profile for You internet profiling (?)
2564libwww-perl/5.64unknown link checking from (
2565libwww-perl/5.65Amidalla search engine robot (62.241.33.xx)
2566libwww-perl/5.800SplatSearch robot (72.36.210.xx)
2567libwww/5.3.2Mediater Rechercher robot
2568Liferea/0.x.x (Linux; en_US.UTF-8; - Linux feed reader
2569Liferea/1.x.x (Linux; es_ES.UTF-8; (Linux Feed Reader) news aggregator for Unix and Linux
2570LightningDownload/1.0beta2Lightning Download manager
2571LightningDownload/1.x.xLightning Download manager
2572LightningDownload/1.x.x [Accelerated x]Lightning Download manager
2573LijitSpider/Nutch-0.9 (Reports crawler;; info(a)lijit(d)com)Lijit blog search spider
2574Lincoln State Web BrowserSome spam bot
2575Link Valet Online 1.xLink Valet online link checking
2576LinkAlarm/2.xLinkalarm link validation
2577LinkbotLinkbot Pro link checking software
2578linkbotRpsoft 2000 Site-Crawler
2579Linkbot x.0Linkbot Pro link checking software
2580LinkCheck ( - linkchecking tool for Unix/Linux
2581LinkLint-checkonly/2.x.xLinklint - Perl html link checker
2582LinkLint-spider/2.x.xLinklint - Perl html link checker
2583linknzbotLinknz - The Kiwi Search Engine (New Zealand)
2584LinkPimpin v1.0Link-Pimp web directory link checking
2585LinkProver 2.1TafWeb link checking program
2586Links (0.9x; Linux 2.4.7-10 i686)Links text browser for Unix & OS/2
2587Links (0.9xpre12; Linux 2.2.14-5.0 i686; 80x24)Links text browser for Unix & OS/2
2588Links (2.xpre7; Linux 2.4.18 i586; x)Links text browser for Unix & OS/2
2589Links - SQL directory management program
2590Links 2.0 ( SQL directory management program
2591Links SQL ( SQL directory management program
2592Links4US-Crawler, (+ ODP based directory link checking
2593LinkScan/11.0beta2 UnixShareware robot from (used by Indiafocus/Indiainfo)Shareware robot from (used by Indiafocus/Indiainfo and others)
2594LinkScan/9.0g UnixShareware robot from (used by Indiafocus/Indiainfo and others)
2595LinkScan/x.x UnixShareware robot from ( bookmark checking
2597LinkSonar/1.35LinkSonar (Japan) link checking tool
2598LinkSweeper/1.xLeft Side Software's LinkSweeper (ceased) bookmark utility
2599LinkWalkerSeventwentyfour link checking robot
2600link_check3.plx libwww-perl/5.65link check 3 - Perl HTML link checker (from Perl for Web Site Management)
2601ListBidBot (freelance job spider / link checking ?
2602LiveTrans/Nutch-0.9 (maintainer: cobain at iis dot sinica dot edu dot tw; Lab: LiveTrans - Online query and terminology translation service
2603Llaut/1.0 ( robot - Universitat de les Illes Balears (Spain)
2604LMQueueBot/0.2E-Mail harvesting robot - same as ContactBot
2605lmspider ( from Scansoft (192.133.61.xx) - Web text collector
2606LNSpiderguyLexis-Nexis robot
2607LocalBot/1.0 ( company information collector
2608LocalcomBot/1.2.x ( local search robot (
2609Lockstep Spider/1.0Lockstep (website content protection tool) user agent
2610Look.comGlobalQueue spider (
2611Lotus-Notes/4.5 ( Windows-NT )Lotus Notes browser
2612LotusDiscovery/x.0 (compatible; Mozilla 4.0; MSIE 4.01; Windows NT)IBM Lotus Discovery Server
2613Lovel as 1.0 ( + song lyrics search
2614LTI/LemurProject Nutch Spider/Nutch-1.0-dev (lti crawler for CMU;; changkuk at cmu dot edu)LTI - The Lemur Toolkit for Language Modeling and Information Retrieval via Yahoo
2615LTI/LemurProject Nutch Spider/Nutch-1.0-dev (Research spider using Nutch;; - The Lemur Toolkit for Language Modeling and Information Retrieval via Yahoo URL (Linux Wiki) link checking
2617LunascapeLunascape IE based browser (Japan)
2618lwp-trivial/1.32Ultimate Search / Smartdesk (no website) robot
2619lwp-trivial/1.34Search4free robot
2620lwp-trivial/1.35Expert HTML online source viewer
2621LWP::Simple/5.22Perl LWP:Collective module
2622LWP::Simple/5.36Perl LWP:Collective module
2623LWP::Simple/5.48Perl LWP:Collective module - Linkomatic robot
2624LWP::Simple/5.50Perl LWP:Collective module - link checking (in conjunction w. libwww-perl/5.50)
2625LWP::Simple/5.51Perl LWP:Collective module - Inktomi (62.253.64.x) robot
2626LWP::Simple/5.53Perl LWP:Collective module -
2627LWP::Simple/5.63Perl LWP:Collective module
2628LWP::Simple/5.803ThePlanet/ Google Adsense refferer spam bot from 70.85.116.* / /
2629Lycos_Spider_(modspider)Lycos spider
2630Lycos_Spider_(T-Rex)Lycos spider
2631Lynx/2-4-2 (Bobcat/0.5 [DOS] Jp Beta04)Bobcat - Text and Lynx based DOS browser
2632Lynx/2.6 libwww-FM/2.14Lynx 2.x text mode browser
2633Lynx/2.8 (; online service - uses lynx browser for crawlers view of web pages
2634Lynx/2.8.3dev.9 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6Lynx 2.x text mode browser
2635Lynx/2.8.4rel.1 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6c ( 2.x text mode browser used as robot via
2636Mac Finder 1.0.xxSome spam bot
2637Mackster( )UKWizz search robot client user-agent ?
2639MagicWML/1.0 (forcewml)All Magic/Wap wml service for mobile devices
2640MagpieRSS/0.7x (+ RSS - PHP RSS Parser
2641Mahiti.Com/Mahiti Crawler-1.0 (Mahiti.Com; ; India search crawler
2642Mail.Ru/ search
2643mailto:webcraft@bea.comUnknown Bea robot
2644mammoth/1.0 ( Systems mammoth robot
2645MantraAgentLooksmart robot ( of the Internet visual search engine index robot
2647Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot) robot
2648Marketwave Hit ListPilot Hitlist web site analysis
2649MartiniLooksmart directory page analysis
2650MARTINILooksmart directory page analysis
2651Marvin v0.3Marvin Medhunt robot
2652MaSagool/1.0 (MaSagool;; search Japan robot
2653Mass Downloader 2.xMass Downloader download manager
2654MasterSeekMasterseek (Scandinavia) Beta business search ?
2655Mata Hari/2.00Lexibot (exMataHari) search software
2656Matrix S.p.A. - FAST Enterprise Crawler 6 (Unknown admin e-mail address)Virgilio Italy robot ( using Fast Enterprise Search
2657maxomobot/dev-20051201 (maxomo;; multimedia search robot
2658McBot/5.001 (windows; U; NT4.0; en-us)Unknown robot from McAfee Austria (80.123.144.xx)
2659MDbot/1.0 (+ files search robot
2660Media Player ClassicSubstitute for Windows Media Player
2661MediaCrawler-1.0 (Experimental)Media Find crawler
2662Mediapartners-Google/2.1 ( AdSense robot
2663MediaSearch/ Media Search
2664MegaSheep v1.0 ( internet sheep)Search UK robot
2665Megite2.0 ( web2.0 RSS and news service software
2666Mercator-1.xMercator crawler software (used by Altavista)
2667Mercator-2.0Mercator crawler software (used by Altavista)
2668Mercator-Scrub-1.1Mercator crawler software (used by Altavista)
2669Metaeuro Web Crawler/0.2 (MetaEuro Web Search Clustering Engine;; crawler at metaeuro dot com) Clustering Web Search Engine crawler
2670MetaGer-LinkCheckerMetaGer search robot (Germany)
2671MetagerBot/0.8-dev (MetagerBot;; )MetaGer search robot (Germany)
2672MetaGer_PreChecker0.1MetaGer search robot (Germany)
2673MetaProducts Download Express/1.xDownload Express download manager
2674Metaspinner/0.01 (Metaspinner;; search robot - Germany
2675metatagsdir/0.7 (+ directory index spider
2676MFC Foundation Class Library 4.0Microsoft Foundation Class Library - i.e. used for e-mail harvesting from 68.154.96.xx ( user agent
2678MFHttpScanAdvanced Site Crawler web site ripper and extractor
2679MicroBazGigaBaz Brainbot (Germany) robot (213.139.152.xx)
2680Microsoft Data Access Internet Publishing Provider Cache ManagerMS Office 2000 acting as WebDAV client
2681Microsoft Data Access Internet Publishing Provider DAVMS Office 2000 acting as WebDAV client
2682Microsoft Data Access Internet Publishing Provider Protocol DiscoveryServer probe for data access operations using MS Frontpage with OPTION header
2683Microsoft Data Access Internet Publishing Provider Protocol DiscoveryMS Office 2000 acting as WebDAV client
2684Microsoft Log Parser 2.2Microsoft Log Parser text query tool
2685Microsoft Small Business IndexerMS Small Business Server content indexer
2686Microsoft URL Control - 6.00.8xxxuser agent looks for form-mail components (spam-bot)
2687MicrosoftPrototypeCrawler (How's my crawling? robot from (131.107.163.xx)
2688Microsoft_Internet_Explorer_5.00.438 ( Computing SmartFilterWhere / Bess web filter (192.55.214.xx)
2689MIIxpc/ HTTP server message
2690Mindjet MindManagerThe Mindjet blog MindManager category
2691minibotunknown robot via Korea Telecom (
2692miniRank/1.6 (Website ranking;; robot)mini- Rank website popularity tool
2693MiracleAlphaTestunknown robot via Telecom (210.143.35.xx)
2694Missauga Locate 1.0.0Some spam bot
2695Missigua Locator 1.9Some spam bot
2696Missouri College BrowseSome spam bot
2697Mister Pix II 2.02aMister PiX picture finding software
2698Mister PiX version.dllMister PiX picture finding software
2699Misterbot-Nutch/0.7.1 (Misterbot-Nutch;; search France robot
2700Miva ( / ex search robot (
2701Mizzu Labs 2.2Some spam bot from Jasmine Internet - Bangkok (203.147.0.xx)
2702MJ12bot/vx.x.x ( DSearch MJ12bot (Experimental distributed crawler)
2703MJ12bot/vx.x.x ( DSearch MJ12bot (Experimental distributed crawler)
2704MJBot (SEO assessment)MJB SEO Club MJBot
2705MLBot ( - metadata labs web crawler for building a media index (beta)
2706MnogoSearch/3.2.xxmnoGoSearch (ex UdmSearch) software robot
2707Mo College 1.9Unknown bad bot - maybe guestbook spamming or email harvesting
2708moget/x.x ( Japan / Inktomi robot (210.173.179.xx)
2709mogimogi/1.0Goo Japan / Inktomi robot (210.173.179.xx)
2710moiNAG 0.02moiNag - generator
2711MojeekBot/0.x (archi; Search Preview robot (217.155.205.xx)
2712monkeyagentGreasemonkey Firefox extension
2713MoonBrowser (version 0.41 Beta4)Moonbrowser - IE based browser (Japan)
2714Moreoverbot/x.00 (+ / FeedDirect RSS feed robot
2715Morris - Mixcat Crawler ( robot s. also Felix
2716Motoricerca-Robots.txt-Checker/1.0 ( online checker
2717Motorola-V3m ObigoObigo WAP browser for mobiles on Motorola V3
2718Mouse-House/7.4 (spider_monkey spider info at Group Ltd. Spider Monkey robot
2719MovableType/x.xMovable Type web-based personal publishing system
2720mozDex/0.xx-dev (mozDex;; Open search engine spider (
2721Mozi!Bell Nexxia / Sympatico Canada user robot & spoofed referer from diff. IPs
2722MozillaUnknown robots from diff. IPs
2723Mozilla (libwhisker/2.4)libwhisker - HTTP client and utility - Perl library
2724Mozilla ( robot
2725Mozilla 4.0(compatible; BotSeer/1.0; + search engine for robots.txt
2726Mozilla/1.1 (compatible; MSPIE 2.0; Windows CE)PDA Pocket IE 2.x Windows CE
2727Mozilla/1.10 [en] (Compatible; RISC OS 3.70; Oregano 1.10)Oregano browser for RISC OS
2728Mozilla/1.22 (compatible; MSIE 2.0d; Windows NT)IE 2.x WinNT
2729Mozilla/1.22 (compatible; MSIE 5.01; PalmOS 3.0) EudoraWeb 2EudoraWeb 2.0 browser (Eudora Internet Suite) for PalmOS
2730Mozilla/2.0Dummy user agent - i.e. used by Namo Web Editor
2731Mozilla/2.0 (compatible; AOL 3.0; Mac_PowerPC)AOL Mac
2732Mozilla/2.0 (Compatible; AOL-IWENG 3.0; Win16)AOL Win 3.x
2733Mozilla/2.0 (compatible; Ask Jeeves)Ask Jeeves /Teoma robot
2734Mozilla/2.0 (compatible; Ask Jeeves/Teoma)Ask Jeeves /Teoma robot
2735Mozilla/2.0 (compatible; Ask Jeeves/Teoma; Jeeves /Teoma robot
2736Mozilla/2.0 (compatible; Ask Jeeves/Teoma; Jeeves /Teoma robot
2737Mozilla/2.0 (compatible; EZResult -- Internet Search Engine)Direct Hit Robot
2738Mozilla/2.0 (compatible; MS FrontPage x.0)MS Frontpage x.x web editor
2739Mozilla/2.0 (compatible; MSIE 2.1; Mac_PowerPC)IE 2.x Mac Power PC
2740Mozilla/2.0 (compatible; MSIE 3.02; Update a; AK; Windows NT)IE 3.x WinNT
2741Mozilla/2.0 (compatible; MSIE 3.02; Update a; AOL 3.0; Windows 95)IE 3.x AOL Win95
2742Mozilla/2.0 (compatible; MSIE 3.0; AK; Windows 95)IE 3.x Win95
2743Mozilla/2.0 (compatible; MSIE 3.0; Windows 3.1)IE 3.x Win 3.1
2744Mozilla/2.0 (compatible; MSIE 3.0B; Win32)IE 3.x WinXP
2745Mozilla/2.0 (compatible; NEWT ActiveX; Win32)Borland Delphi .OCX component used by WebCollector email harverster
2746Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E)Thunderstone's Webinator Web indexing program
2747Mozilla/2.0 compatible; Check&Get 1.1x (Windows 98)Check&Get bookmark and link checking tool
2748Mozilla/2.01 (Win16; I)Netscape 2.x Win3.x International
2749Mozilla/2.02Gold (Win95; I)Netscape 2.x Gold Win95
2750Mozilla/3.0 (compatible)Faked user agent for diff. purposes i.e.: - some download manager - E-mail harvesting
2751Mozilla/3.0 (compatible; AvantGo 3.2)AvantGo PDA browser
2752Mozilla/3.0 (compatible; Fluffy the spider;; robot
2753Mozilla/3.0 (compatible; HP Web PrintSmart 04b0 Web PrintSmart (discontinued) - web page printing software
2754Mozilla/3.0 (compatible; Indy Library)Internet Direct Library for Borland (often used as e-mail address collector and mass mailing tool)
2755Mozilla/3.0 (compatible; Linkman)Outertechs Linkman bookmark tool
2756Mozilla/3.0 (compatible; MuscatFerret/1.5.4; robot
2757Mozilla/3.0 (compatible; MuscatFerret/1.5; robot
2758Mozilla/3.0 (compatible; MuscatFerret/1.6.x; robot
2759Mozilla/3.0 (compatible; netart generator/1.0; libwww-perl/5.64)Netart Generator - script generated random websites
2760Mozilla/3.0 (compatible; NetPositive/2.2)NetPositive BEOS browser
2761Mozilla/3.0 (compatible; Opera/3.0; Windows 3.1) v3.1Opera 3.x Win3.x
2762Mozilla/3.0 (compatible; Opera/3.0; Windows 95/NT4) 3.2Opera 3.x Win95/NT
2763Mozilla/3.0 (compatible; PerMan Surfer 3.0; Win95)Perman Surfer bookmark tool
2764Mozilla/3.0 (compatible; REL Software Web Link Validator 2.x)Web Link Validator link validation software
2765Mozilla/3.0 (compatible; scan4mail (advanced version) online mail extraction service
2766Mozilla/3.0 (compatible; ScollSpider; UK search engine robot (
2767Mozilla/3.0 (compatible; Web Link Validator 2.x)Web Link Validator link validation softwareWeb Link Validator link validation software
2768Mozilla/3.0 (compatible; WebCapture x.x; Auto; Windows)Xelios Web Capture (now Wysigot) website downloading tool (Discontinued)
2769Mozilla/3.0 (compatible; search engine positioning using Thunderstone's Webinator
2770Mozilla/3.0 (compatible;'s Media Monitor using Thunderstone's Webinator
2771Mozilla/3.0 (Compatible;Viking/1.8)Viking server user/client
2772Mozilla/3.0 (DreamPassport/3.0)One of DC-Sakuras download manager user-agent names
2773Mozilla/3.0 (INGRID/3.0 MT;; Netherlands robot (62.69.178.xx)
2774Mozilla/3.0 (Liberate DTV 1.1)Liberate DTV server suite / TV-emulator
2775Mozilla/3.0 (Planetweb/2.100 JS SSL US; Dreamcast US)Planetweb 2.1 Browser (discontinued) for Dreamcast
2776Mozilla/3.0 (;; /Inktomi robot
2777Mozilla/3.0 (Slurp/cat;; (Hotbot-Lycos-NBCi) robot - 72.30.61.xx(x)
2778Mozilla/3.0 (Slurp/si;; (Hotbot-Lycos-NBCi) robot - 72.30.61.xx(x)
2779Mozilla/3.0 (Vagabondo/1.1 MT;; robot Netherland -
2780Mozilla/3.0 (Vagabondo/1.x MT;; robot Netherland -
2781Mozilla/3.0 (Vagabondo/2.0 MT;; Netherlands robot
2782Mozilla/3.0 (Vagabondo/2.0 MT;; robot Netherland -
2783Mozilla/3.0 (Win16; I)Netscape 3.x Win3.x
2784Mozilla/3.0 (Win95; I)Netscape 3.x Win95
2785Mozilla/3.0 (WinNT; I)Netscape 3.x WinNT
2786Mozilla/3.0 (WorldGate Gazelle 3.5.1 build 11; FreeBSD2.2.8-STABLE)Netscape 3.x FreeBSD
2787Mozilla/3.0 (X11; I; OSF1 V4.0 alpha)Netscape 3.x OSF1 V4.0 alpha
2788Mozilla/3.0 NAVIO_AOLTV (11; 13; Philips; PH200; 1; R2.0C36_AOL.0110OPTIK; R2.0.0139d_OPTIK)AOL Web TV
2789Mozilla/3.0 WebTV/1.2 (compatible; MSIE 2.0)WebTV
2790Mozilla/3.01 (compatible; AmigaVoyager/2.95; AmigaOS/MC680x0)Amiga Voyager Browser Amiga
2791Mozilla/3.01 (Compatible; Links2Go Similarity Engine)Links2Go robot
2792Mozilla/3.01 (compatible; Netbox/3.5 R92; Linux 2.2)Netgem Netbox cable modem TV Box Linux
2793Mozilla/3.01-C-MACOS8 (Macintosh; I; PPC)Netscape 3.x Mac
2794Mozilla/3.01Gold (X11; I; Linux 2.0.32 i486)Netscape 3.x Linux
2795Mozilla/3.01Gold (X11; I; SunOS 5.5.1 sun4m)Netscape 3.x SunOS
2796Mozilla/3.01SGoldC-SGI (X11; I; IRIX 6.3 IP32)Netscape 3.x Irix
2797Mozilla/3.04 (compatible; ANTFresco/2.13; RISC OS 4.02)ANT Fresco Browser Risc OS
2798Mozilla/3.04 (compatible; NCBrowser/2.35; ANTFresco/2.17; RISC OS-NC 5.13 Laz1UK1309)NCBrowser ANT Fresco Browser Risc OS
2799Mozilla/3.04 (compatible;QNX Voyager 2.03B ;Photon)QNX OS Voyager embedded browser
2800Mozilla/3.x (I-Opener 1.1; Netpliance)I-Opener (was web PC
2801Mozilla/4.0Yahoo Mindset: Intent-driven Search (
2802Mozilla/4.0 (agadine3.0) www.agada.deAgada search (Germany) robot
2803Mozilla/4.0 (Compatible); URLBase 6URLBase 6 bookmark manager
2804Mozilla/4.0 (compatible: AstraSpider V.2.1 :! adult search robot (66.98.252.xx)
2805Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; robot Netherland -
2806Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; robot Netherland -
2807Mozilla/4.0 (compatible; ReGet Deluxe 5.1; Windows NT 5.1)ReGet Deluxe! download manager
2808Mozilla/4.0 (compatible; Advanced Email Extractor v2.xx)Advanced Email Extractor e-mail collector (spam bot)
2809Mozilla/4.0 (compatible; Arachmo)Arachmo Spider - web site file extraction tool
2810Mozilla/4.0 (compatible; BorderManager 3.0)Novell Border Manager security suite
2811Mozilla/4.0 (compatible; BOTW Spider; + of the Web directory link checking
2812Mozilla/4.0 (compatible; B_L_I_T_Z_B_O_T)Blitzsuche Germany robot
2813Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)Content Control from Blue Coat
2814Mozilla/4.0 (compatible; Check&Get 3.0; Windows NT)Check&Get bookmark manager, web change monitor and archiver
2815Mozilla/4.0 (compatible; Christcrawler (was
2816Mozilla/4.0 (compatible; crawlx, Search Marketing crawler (
2817Mozilla/4.0 (compatible; DAUMOA-video; + - Daum search Korea robot (
2818Mozilla/4.0 (compatible; DepSpid/5.0x; + distributed web crawler for link dependencies
2819Mozilla/4.0 (compatible; DnloadMage 1.0)Download Mage download manager
2820Mozilla/4.0 (compatible; FastCrawler3 crawler
2821Mozilla/4.0 (compatible; FDSE robot)Fluid Dynamics Search Engine (FDSE) robot used by
2822Mozilla/4.0 (compatible; GPU p2p crawler Distributed Search Engine crawler
2823Mozilla/4.0 (compatible; grub-client-0.2.x; Crawl your stuff with open source crawler
2824Mozilla/4.0 (compatible; grub-client-0.3.x; Crawl your own stuff with open source crawler
2825Mozilla/4.0 (compatible; grub-client-2.x)Grub open source crawler used by Looksmart ( 64.241.242.xx)
2826Mozilla/4.0 (compatible; ibisBrowser)ibisBrowser Japanese mobile browser
2827Mozilla/4.0 (compatible; ICS iChain Cool Solutions caching
2828Mozilla/4.0 (compatible; IE-Favorites-Check-0.5)IE Favorites Check - Bookmark manager
2829Mozilla/4.0 (compatible; Iplexx Spider/1.0 Austria (webhosting company) logfile spamming bot
2830Mozilla/4.0 (compatible; KeepNI web site monitor)KeepNi Monitors - Web site monitoring / link checking tool
2831Mozilla/4.0 (compatible; Link Utility; Link Utility link checking tool
2832Mozilla/4.0 (compatible; Lotus-Notes/5.0; Windows-NT)Lotus Notes 5.0 browser
2833Mozilla/4.0 (compatible; MSIE 4.01; AOL 4.0; Windows 98)IE 4.x AOL Win98
2834Mozilla/4.0 (compatible; MSIE 4.01; Mac_PowerPC)IE 4.x Mac Power PC
2835Mozilla/4.0 (compatible; MSIE 4.01; MSIECrawler; Windows 95)Internet Explorer 4.0 URL check
2836Mozilla/4.0 (compatible; MSIE 4.01; b o t)Vonna search robot
2837Mozilla/4.0 (compatible; MSIE 4.01; Windows 95)IE 4.x Win95
2838Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; MSN Companion 2.0; 800x600; Compaq)IE PDA Browser Windows CE
2839Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer OrangeBot-Mobile 2008.0 ( France robot for mobiles
2840Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPS; 240x320)IE for Windows CE on a PocketPC (HP iPAQ)
2841Mozilla/4.0 (compatible; MSIE 4.01; Windows NT Windows CE)IE PDA Browser Windows CE
2842Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)IE 4.x WinNT
2843Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot) Microsoftdiff. IPs / services i.e.: - Microsoft server information robot (see link) - Okanagan Internet Junction web filter (robot)
2844Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) ACRUnknown robot from American College of Radiology (ACR) running MS Site Server
2845Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Indonesia InteractiveIndonesia Interactive Web-portal robot on MS Site Server
2846Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) WebQuest DesignsWebquestdesigns hosting
2847Mozilla/4.0 (compatible; MSIE 5.01; Windows 95) via Avirt Gateway Server v4.0Avirt Gateway proxy server
2848Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) ( ( robot
2849Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; NetCaptor 6.5.0RC1)NetCaptor IE browser addon
2850Mozilla/4.0 (compatible; MSIE 5.0; AOL 5.0; Windows 95; DigExt; Gateway2000; 5.x AOL Win95 Sureseeker search plugin
2851Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC; AtHome021)IE 5.x Mac PowerPC AtHome user
2852Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: crawler
2853Mozilla/4.0 (compatible; MSIE 5.0; Win32) via proxy gateway CERN-HTTPD/3.0 libwww/2.17WinXP via CERN httpd proxy server
2854Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; robot
2855Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 ( robot
2856Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; robot
2857Mozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en]WinME Opera 5.x
2858Mozilla/4.0 (compatible; MSIE 5.0; Windows ME; Link Checker 2.x.xx's Link Checker
2859Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS AgentBeijing Express Email Address Extractor via DHCP Data Transport Services (DTS)
2860Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; (browser plug-in) robot
2861Mozilla/4.0 (compatible; MSIE 5.0;; robot (
2862Mozilla/4.0 (compatible; MSIE 5.0;;; robot (
2863Mozilla/4.0 (compatible; MSIE 5.0; YANDEX)Yandex Search Russia link checking (
2864Mozilla/4.0 (compatible; MSIE 5.5; AOL 4.0; Windows 98; GoBeez ( starting page plugin
2865Mozilla/4.0 (compatible; MSIE 5.5; Windows 95; Transmission Segment; Hotbar 2.0)IE 5.5 Win95 Hotbar plug-in
2866Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Crazy Browser 1.x.x)Crazy Browser - IE based tabbed Browser
2867Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; KITV4.7 Wanadoo)Wanadoo Internet services
2868Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; SAFEXPLORER TL)Safexplorer ( - site is offline) kids browser
2869Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; SYMPA; Katiesoft 7; SimulBrowse 3.0)Katiesoft Scroll (ex now discarded) & SimulBrowse (ex now dead) IE browser plugins
2870Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; BTinternet V8.1)Windows ME BTOpenworld Internet services
2871Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; MSIECrawler)Windows ME Internet Explorer URL check
2872Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)Cobion Germany Brand Protection Services robot
2873Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; QXW03018)Cobion Germany Brand Protection Services robot
2874Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Active Cache RequestIE 5.5 Win2000 / user agent
2875Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API RequestMaybe: - MS Internet Security & Acceleration Server (ISA) cache refreshing request (see link) or - IE 5.5 Win2000 probably with some (website) API request component (see 2nd link) - suspected as email-harvester / site scanning tool (see
2876Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; .NET CLR 1.0.3705)IE 5.5 Win2000 with MS.NET SDK
2877Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; AIRF)IE 5.5 Win2000 / user agent w. AI RoboForm (AIRF) password manager
2878Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; AspTear 1.5)AspTear URL fetching program component / spider
2879Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; N_o_k_i_a) network
2880Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461) RPT-HTTPClient/0.3-3EUnknown Object Sciences Corp. robot using the HTTPClient
2881Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +;; SpiderThread Revision: 3.10Singingfish media spider ( via AOL search
2882Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows 98; .NET CLR 1.1.4322; MEGAUPLOAD 2.0)Megaupload Mega Manager - Download manager toolbar for IE
2883Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows NT 5.1; SV1; HbTools 4.7.2)Hotbar IE graphical skin
2884Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Skampy/0.9.x [en] directory link checker
2885Mozilla/4.0 (compatible; MSIE 6.0; TargetSeek/1.0; + Crawler concerning electronics industry product announcements
2886Mozilla/4.0 (compatible; MSIE 6.0; Win32) WebWasher 3.0IE 6.0 WebWasher ad filter
2887Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) REL Software Web Link Validator 2.x)Web Link Validator link validation software
2888Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) Web Link Validator 2.x)Web Link Validator link validation software
2889Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Net M@nager V3.02 - 6.0 Netmanager IE add-on
2890Mozilla/4.0 (compatible; MSIE 6.0; Windows 98;; web analyzer & link checker
2891Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; Multibrowser - IE based browser
2892Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; Lunascape 2.1.3)Lunascape IE based browser (Japan)
2893Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Google Wireless Transcoder;)Google wireless transcoder (GWT) proxy for rewriting websites for mobiles
2894Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP entries t_st;üzilla (Germany) - ODP link checking using Robozilla
2895Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP links test;üzilla (Germany) - ODP link checking using Robozilla
2896Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; bot; .NET CLR 1.1.4322)ZoomSpider.Net indexing robot for several directorys
2897Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) ( robot from 64.246.44.xx
2898Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Covac UPPS Cathan 1.2.5;)Covac Software UPPS (Universal PHP Proxy Server) - free public proxy server
2899Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Crayon Crawler; snprtz|T04056566514940; (R1 1.5))GetNetWise Crayon Crawler web filter
2900Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Deepnet Explorer)Deepnet Explorer - IE based browser
2901Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; heritrix/1.3.0 Internet Archive's open-source web project used by Analysis Projects at UW
2902Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 3.0)IE 6x WinXP Hotbar plug-in
2903Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; iOpus-I-M)IE 6x WinXP iOpus Internet Macros - Internet-based macro recorder
2904Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; iRider 2.21.1108; FDM)iRider - IE based browser / Free Download Manager (FDM)
2905Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; KKman3.0)KKman - Japanese IE based browser
2906Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MathPlayer2.0)IE 6x WinXP MathPlayer mathematical notation plugin
2907Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Maxthon)Maxton (ex MyIE2) - IE based browser
2908Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; PeoplePal 3.0; MSIECrawler)IE 6x WinXP peoplepc online PeoplePal IE toolbar
2909Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461; IOpener Release 1.1.04)IE 6x WinXP / I-Opener (was web PC
2910Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 search (China) robot
2911Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SIMBAR Enabled; InfoPath.1)SimBar IE toolbar for accessing The Sims sites / Infopath IE form & spreadsheet plugin
2912Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; 1.760; .NET CLR 1.1.4322)IE 6x WinXP Stumble Upon IE toolbar
2913Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Embedded Web Browser from:; MSIECrawler)Balsa Productions embedded web browser package for Borland Delphi
2914Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; )ChangeDetection robot for web page monitoring
2915Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)IE 6x WinXP also used by WebSite Pro HTML editor
2916Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; DX-Browser - German IE based browser
2917Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; ezPeer+ v1.0 Beta (; ezPeer+ v1.0 (; .NET CLR 1.1.4322; MSIECrawler)ezPeer+ P2P IE addon
2918Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; IBP; .NET CLR 1.1.4322)Axandra IBP website promotion software ?
2919Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MRA 4.3 (build 01218))MRA = Agent - Instant Messenger / VoIP
2920Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)MSN Search robot (207.46.89.xx)
2921Mozilla/4.0 (compatible; MSIE 7.0; Win32) Link Commander 4.0Link Commander bookmark manager
2922Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; bgft)IE 7.0 - WinXP
2923Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB5; User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; ; .NET CLR 2.0.50727)GTB = Google Toolbar Internet Explorer add-on
2924Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0)MSIE 7.0 *and* Trident token used by Internet Explorer 8 in compatibility view mode
2925Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)Internet Explorer 8
2926Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)IE 8.0 (beta) on Win Vista
2927Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; Orange 8.0; GTB6.3; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from:; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30618; OfficeLiveConnector.1.3; OfficeLivePatch.1.3)Bsalsa embedded browser
2928Mozilla/4.0 (compatible; MSIE enviable; DAUMOA 2.0; DAUM Web Robot; Daum Communications Corp., Korea; + - Daum search Korea robot (
2929Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.1; DAUM Web Robot; Daum Communications Corp., Korea)DAUMOA - Daum search Korea robot (
2930Mozilla/4.0 (compatible; NaverBot/1.0; Search Korea Naverbot
2931Mozilla/4.0 (compatible; Netcraft Web Server Survey)Netcraft webserver info
2932Mozilla/4.0 (compatible; NetPromoter Spider; Link Utility link checking tool
2933Mozilla/4.0 (compatible; Opera/3.0; Windows 4.10) 3.51 [en]Opera 3.x WinNT
2934Mozilla/4.0 (compatible; Powermarks/3.5; Windows 95/98/2000/NT)Powermarks bookmark manager
2935Mozilla/4.0 (compatible; RSS Popper)RSS Popper - MS Outlook RSS reader plugin
2936Mozilla/4.0 (compatible; SiteKiosk 4.0; MSIE 5.0; Windows 98; SiteCoach 1.0)SiteKiosk public terminal browser
2937Mozilla/4.0 (compatible; SpeedySpider; Search Speedyspider (
2938Mozilla/4.0 (compatible; SPENG)SiteProbe - website status checking
2939Mozilla/4.0 (compatible; SuperCleaner 2.xx; Windows 98)Super Cleaner privacy tool (bookmark checking)
2940Mozilla/4.0 (compatible; Synapse)Synapse - Apache web service for processing XML documents
2941Mozilla/4.0 (compatible; WebCapture 3.0; Windows)Web2PDF - Adobe Acrobat plugin for site traversal and other services for the Web Capture feature
2942Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)Windows HTTP Services (WinHTTP / XML-parser)
2943Mozilla/4.0 (compatible; WSN Links)WSN Links PHP directory software
2944Mozilla/4.0 (compatible;; urlchecker1.0)Euro Directory (German / Austrian) directory link checking
2945Mozilla/4.0 (compatible; robot
2946Mozilla/4.0 (compatible; Linkguard Online 1.0; Windows NT) link validation (service is offline)
2947Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid)Yahoo Search Japan robot (203.141.52.)
2948Mozilla/4.0 (compatible; Yahoo Japan; for robot study; kasugiya)Yahoo Japan robot (202.93.76.xx)
2949Mozilla/4.0 (compatible;MSIE 6.0; Windows NT 5.0; H010818)Faked IE id string used by DeepTrawl link checking tool
2950Mozilla/4.0 (fantomBrowser)spoofed referer by Fantomaster (Multiblocker) anonymity products
2951Mozilla/4.0 (fantomCrew Browser)spoofed referer by Fantomaster (Multiblocker) anonymity products
2952Mozilla/4.0 ( robot from - - (
2953Mozilla/4.0 (JemmaTheTourist; Jemma spider
2954Mozilla/4.0 (MobilePhone PM-8200/US/1.0) NetFront/3.x MMP/2.0NetFront (v3.x) for Pocket PC (here on Sanyo PM-8200 cell phone)
2955Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1; robot from
2956Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible; Googlebot/2.1; robot from
2957Mozilla/4.0 (Mozilla;; robot from
2958Mozilla/4.0 (Sleek Spider/1.2)ASI - Any Search Info robot
2959Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot;; (Looksmart) online bookmark tool robot
2960Mozilla/4.0 compatible ZyBorg/1.0 (; robot
2961Mozilla/4.0 compatible ZyBorg/1.0 (; robot
2962Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (; robot
2963Mozilla/4.0 compatible ZyBorg/1.0 for Homepage (; robot
2964Mozilla/4.0 efp@gmx.netUnknown robot from 66.230.140.xx ( maybe an e-mail collector
2965Mozilla/4.0 WebTV/2.6 (compatible; MSIE 4.0)WebTV
2966Mozilla/4.0 [en] (Ask Jeeves Corporate Spider)Ask / Ask Jeeves robot
2967Mozilla/4.0(compatible; Zealbot 1.0)LookSmart spider
2968Mozilla/4.01 (compatible; NORAD National Defence Network)HideMe - Web based anonymous proxy server service
2969Mozilla/4.01 [en](Win95;I)Some download manager spoofing Netscape 4.01
2970Mozilla/4.02 [en] (X11; I; SunOS 5.6 sun4u)Netscape 4.x SunOS 5.6
2971Mozilla/4.04 (compatible; Dulance bot; + Bot - Dulance automated price comparison engine
2972Mozilla/4.04 [en] (X11; I; HP-UX B.10.20 9000/712)Netscape 4.x HP-Unix
2973Mozilla/4.04 [en] (X11; I; IRIX 5.3 IP22)Netscape 4.x IRIX
2974Mozilla/4.05 (Macintosh; I; 68K Nav)Netscape 4.x Macintosh 68k
2975Mozilla/4.05 (Macintosh; I; PPC Nav)Netscape 4.x Macintosh PowerPC
2976Mozilla/4.05 [en] (X11; I; SunOS 4.1.4 sun4m)Netscape 4.x SunOS 4.1.4
2977Mozilla/4.08 [en] (Win98; U ;Nav)Version 4.08 [en]-98306
2978Mozilla/4.08 [en] (WinNT; U)Netscape 4.x WinNT
2979Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_TrueRobot/1.4 libwww/ robot
2980Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_VoilaBot/1.6 libwww/ robot
2981Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)HTTrack Offline Browser
2982Mozilla/4.5 (compatible; iCab 2.5.3; Macintosh; I; PPC)iCab MAC Web browser MAC Power PC
2983Mozilla/4.5 (compatible; OmniWeb/4.0.5; Mac_PowerPC)OmniWeb 4.x.x Mac browser
2984Mozilla/4.5 (compatible; OmniWeb/4.1-beta-1; Mac_PowerPC)OmniWeb 4.x.x Mac browser
2985Mozilla/4.5 RPT-HTTPClient/0.3-2different IPs using the HTTPClient library (mostly link checking)
2986Mozilla/4.5 [en]C-CCK-MCD {RuralNet} (Win98; I)RuralNet Internet Services
2987Mozilla/4.5b1 [en] (X11; I; Linux 2.0.35 i586)Netscape 4.x Linux
2988Mozilla/4.6 [en] ( robot for (216.239.114.xx)
2989Mozilla/4.61 [de] (OS/2; I)Netscape 4.x OS/2
2990Mozilla/4.61 [en] (X11; U; ) - BrowseX (2.0.0 Windows)BrowseX cross-platform browser
2991Mozilla/4.7Nameprotect ( - snoopbot
2992Mozilla/4.7 (compatible; earch and text mining spider
2993Mozilla/4.7 (compatible; Intelliseek; (64.158.138.xx) robot
2994Mozilla/4.7 (compatible; OffByOne; Windows 98) Webster Pro V3.2OffByOne Browser
2995Mozilla/4.7 (compatible; Whizbang)WhizBang! Labs information extraction robot
2996Mozilla/4.7 (compatible; WhizBang;! Labs information extraction robot
2997Mozilla/4.7 [en]( - Becomecom shopping search (64.124.85.xx(x))
2998Mozilla/4.7 [en]( - exava shopping search (64.124.85.xx(x))
2999Mozilla/4.7 [en]C-CCK-MCD {Yahoo;YIP052400} (Win95; I)unknown
3000Mozilla/4.72 [en] (BACS robot
3001Mozilla/4.72C-CCK-MCD Caldera Systems OpenLinux [en] (X11; U; Linux 2.2.14 i686)Netscpape 4.7x Caldera Open Linux Pentium III
3002Mozilla/4.75C-ja [ja] (X11; U; OSF1 V5.1 alpha)Netscape 4.7x Japan OSF1 alpha
3003Mozilla/4.76 (Windows 98; U) Opera 5.12 [en]Opera 5.x Win 98
3004Mozilla/4.76 [en] (X11; U; FreeBSD 4.4-STABLE i386)Netscape 4.7x FreeBSD
3005Mozilla/4.76 [en] (X11; U; SunOS 5.7 sun4u)Netscape 4.7x SunOS
3006Mozilla/4.77C-SGI [en] (X11; U; IRIX 6.5 IP32)IRIX 6.5
3007Mozilla/5.0GigaMedia / NTT DoCoMo robot
3008Mozilla/5.0 (+ Mammoth/0.1Eurekster Swicki community search using SLI-Systems site search engine Mammoth
3009Mozilla/5.0 (+ Mammoth/0.1SLI Systems mammoth robot
3010Mozilla/5.0 (Clustered-Search-Bot/1.0;; search robot
3011Mozilla/5.0 (compatible) GM RSS Panel XGreasemonkey RSS panel Firefox plugin
3012Mozilla/5.0 (compatible; + search robot
3013Mozilla/5.0 (compatible; 008/0.83;;) Gecko/2008032620008 distributed crawler for 80legs
3014Mozilla/5.0 (compatible; Abonti/0.8 - WebSearch beta robot
3015Mozilla/5.0 (compatible; aiHitBot/1.0; + Aihit crawler
3016Mozilla/5.0 (compatible; AnsearchBot/1.x; + Australian search robot
3017Mozilla/5.0 (compatible; archive.org_bot/1.10.0 + Library of Congress Minerva crawler
3018Mozilla/5.0 (compatible; archive.org_bot/1.13.1x - The Internet Archive's open-source crawler (
3019Mozilla/5.0 (compatible; archive.org_bot/1.5.0-200506132127 Hurricane KatrinaHeritrix - The Internet Archive's open-source crawler
3020Mozilla/5.0 (compatible; Ask Jeeves/Teoma; Jeeves /Teoma robot
3021Mozilla/5.0 (compatible; BanBots/2.0b; Fetch; + BanBots Perl script robot
3022Mozilla/5.0 (compatible; BecomeBot/1.23; - shopping search (64.124.85.xx(x))
3023Mozilla/5.0 (compatible; BecomeBot/1.xx; MSIE 6.0 compatible; - shopping search (64.124.85.xx(x))
3024Mozilla/5.0 (compatible; BecomeBot/2.0beta; - shopping search (64.124.85.xx(x))
3025Mozilla/5.0 (compatible; BecomeBot/2.x; MSIE 6.0 compatible; - shopping search (64.124.85.xx(x))
3026Mozilla/5.0 (compatible; BecomeJPBot/2.3; MSIE 6.0 compatible; + - shopping search (64.124.85.xx(x))
3027Mozilla/5.0 (compatible; BlogRefsBot/0.1; blog robot
3028Mozilla/5.0 (compatible; Bot; + Webservice RSS / news crawler (Germany)
3029Mozilla/5.0 (compatible; BuzzRankingBot/1.0; + internet content analysis
3030Mozilla/5.0 (compatible; Charlotte/1.0b; indexing spider for Searchme / Wikiseek
3031Mozilla/5.0 (compatible; Charlotte/1.0b; indexing spider for Searchme / Wikiseek
3032Mozilla/5.0 (compatible; Crawling jpeg; graphics crawler or downloading agent from Yamana Laboratory - Waseda Univerity Japan (133.9.238.xx)
3033Mozilla/5.0 (compatible; Custo 3 (; Windows NT 5.1)Custo web site spidering tool (link checking)
3034Mozilla/5.0 (compatible; de/1.13.2 + German travel related search via Amazon Web Services
3035Mozilla/5.0 (compatible; Diffbot/0.1; + beta - RSS and news feed crawler
3036Mozilla/5.0 (compatible; DNS-Digger-Explorer/1.0; + - DNS server neighbourhood search
3037Mozilla/5.0 (compatible; DNS-Digger/1.0; + - DNS server neighbourhood search
3038Mozilla/5.0 (compatible;; (Czech Republic) search robot (194.108.39.xx)
3039Mozilla/5.0 (compatible; EARTHCOM/2.2; + / search (Czech Republic)
3040Mozilla/5.0 (compatible; egothor/8.0g; + Faculty of Mathematics and Physics using Egothor open source crawler
3041Mozilla/5.0 (compatible; Exabot Test/3.0; + (France) search robot (193.47.80.xx)
3042Mozilla/5.0 (compatible; FatBot 2.0; - Shopping search robot
3043Mozilla/5.0 (compatible; Galbot/1.0; + tagging robot (beta) - Denmark
3044mozilla/5.0 (compatible; genevabot Single-Site Search Engine used by Healthdash health search
3045Mozilla/5.0 (compatible; Google Desktop) Paros/3.2.12Paros - a Java based HTTP/HTTPS proxy
3046Mozilla/5.0 (compatible; Googlebot/2.1; robot
3047mozilla/5.0 (compatible; heritrix/1.0.4 robot using Heritrix
3048Mozilla/5.0 (compatible; heritrix/1.10.2 + Stanford University InfoLab robot using Heritrix
3049Mozilla/5.0 (compatible; heritrix/1.12.1 + news feed search using Heritrix
3050Mozilla/5.0 (compatible; heritrix/1.12.1 + vertical search via Amazon Web Services
3051Mozilla/5.0 (compatible; heritrix/1.12.1 + [] vertical search via Amazon Web Services
3052mozilla/5.0 (compatible; heritrix/1.3.0 Internet Archive's open-source web project
3053Mozilla/5.0 (compatible; heritrix/1.4.0 + Beta search Spain ( using Heritrix
3054Mozilla/5.0 (compatible; heritrix/1.4t data mining robot using Heritrix
3055Mozilla/5.0 (compatible; heritrix/1.5.0 WebCrawling Project (Germany) using Heritrix
3056Mozilla/5.0 (compatible; heritrix/1.5.0-200506231921 Internet Archive crawler (Australia) using Heritrix
3057Mozilla/5.0 (compatible; heritrix/1.6.0 (beta) search for computer scientists and programmers using Heritrix open-source crawler
3058Mozilla/5.0 (compatible; heritrix/1.7.0 + website collection project using Heritrix
3059Mozilla/5.0 (compatible; Heritrix/1.8.0 social web archiving service
3060Mozilla/5.0 (compatible; heritrix/1.x.x + - Accelovation Market Discovery software robot
3061Mozilla/5.0 (compatible; heritrix/2.0.0-RC1 + AOL robot using Heritrix
3062Mozilla/5.0 (compatible; Hermit Search. Com; + - Products and service search robot
3063Mozilla/5.0 (compatible; http://www.IsMySiteUp.Net/bot/ )IsMySiteUp? - Online website monitoring service
3064Mozilla/5.0 (compatible; )UptimeAuditor - real time web monitoring
3065Mozilla/5.0 (compatible; HyperixScoop/1.3; + vertical search crawler
3066Mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0)Unknown robot (reads robots.txt) from chinatelecom (219.142.78.xx)
3067Mozilla/5.0 (compatible; IDBot/1.0; + - Russian search project
3068Mozilla/5.0 (compatible; InterseekWeb/3.x) (Slovenia) search using Interseek/Web Interseek/API Search Engine
3069Mozilla/5.0 (compatible; Jim + social web archiving service
3070Mozilla/5.0 (compatible; Konqueror/2.0.1; X11); Supports MD5-Digest; Supports gzip encodingKonqueror 2.0.x X11
3071Mozilla/5.0 (compatible; Konqueror/2.1.1; X11)Konqueror 2.1.x X11
3072Mozilla/5.0 (compatible; Konqueror/2.2.2)Konqueror 2.2.x
3073Mozilla/5.0 (compatible; Konqueror/2.2.2; Linux 2.4.14-xfs; X11; i686)Konqueror 2.2.x Linux
3074Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)Exalead (France) search robot (193.47.80.xx)
3075Mozilla/5.0 (compatible; LemSpider 0.1)Lemur Consulting LemIR spider
3076Mozilla/5.0 (compatible; LinksManager.com_bot online link checking service
3077Mozilla/5.0 (compatible; LinkStash Bookmark Manager; Bookmark Manager
3078Mozilla/5.0 (compatible; MojeekBot/2.0; Search Preview robot (217.155.205.xx)
3079Mozilla/5.0 (compatible; MOSBookmarks/v2.6-Plus; Link Checker)Joomla!/Mambo component - MosBookmarks (bot) link checking
3080Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; entertainment and video network crawler
3081Mozilla/5.0 (compatible; OnetSzukaj/5.0; Szukaj (Search) robot (
3082Mozilla/5.0 (compatible; PagestackerBot; online bookmark service
3083Mozilla/5.0 (compatible; PalmeraBot; Version 0.001PalmeraBot - search engine robot
3084Mozilla/5.0 (compatible; PEAR HTTP_Request class; feed search (Japan) using Pear HTTP
3085Mozilla/5.0 (compatible; Phonifier; + mobile access to web content
3086Mozilla/5.0 (compatible; ODP link checker; + ODP link checking bot
3087Mozilla/5.0 (compatible; search (Slovenia) robot via Interseek
3088Mozilla/5.0 (compatible; search (Slovenia) robot via Interseek
3089Mozilla/5.0 (compatible; Proximic crawler; + Publisher Widget - RSS and news content generator
3090Mozilla/5.0 (compatible; PWeBot/3.1; PWeBot crawler (Argentina)
3091Mozilla/5.0 (compatible; Quantcastbot/1.0; - Open Internet Ratings Service
3092Mozilla/5.0 (compatible; robtexbot/1.0; )robtex - Multi-RBL check and AS-numbercheck
3093Mozilla/5.0 (compatible; ScoutJet; + (Blekko) search web crawler
3094Mozilla/5.0 (compatible; Scrubby/2.2; the web robot (66.93.156.xx)
3095Mozilla/5.0 (compatible; ShunixBot/1.x.x + (France) beta / test semantic web indexing robot
3096Mozilla/5.0 (compatible; ShunixBot/1.x; (France) beta / test semantic web indexing robot
3097Mozilla/5.0 (compatible; SkreemRBot + - Audio search engine
3098Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv: Gecko/20061206 Firefox/ Firefox Search Plugin
3099Mozilla/5.0 (compatible; SpurlBot/0.2) bookmark service & search engine (
3100Mozilla/5.0 (compatible; SummizeBot + - Opinion and review search robot
3101Mozilla/5.0 (compatible; SYCLIKControl/LinkChecker;)Syclik Control web content management system
3102Mozilla/5.0 (compatible; Synoobot/0.9; web directory robot
3103Mozilla/5.0 (compatible; Theophrastus/x.x; Internet Spider for a basic search engine project
3104Mozilla/5.0 (compatible; TridentSpider/3.1)Interseek - Java search engine technology used for Pogodak search
3105Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; robot
3106Mozilla/5.0 (compatible; Webduniabot/1.0; + search (India) robot
3107Mozilla/5.0 (compatible; Windows NT 5.0; phpwebbrainBot/0.1 - online bookmark service (Germany)
3108Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 + (beta) search for computer scientists and programmers using Heritrix open-source crawler
3109Mozilla/5.0 (compatible; WoW Lemmings Kathune/2.0; spider for World of Warcraft guild data. Used to power WoW Lemmings
3110Mozilla/5.0 (compatible; XTbot/1.0v; + - Server and online services monitoring
3111Mozilla/5.0 (compatible; Yahoo! DE Slurp; / Inktomi search robot
3112Mozilla/5.0 (compatible; Yahoo! Slurp China; robot ( for Yahoo China
3113Mozilla/5.0 (compatible; Yahoo! Slurp; robot for Yahoo (via
3114Mozilla/5.0 (compatible; YesupBot/1.0; + Seo - Toronto SEO Service
3115Mozilla/5.0 (compatible; Yoono; - community based search ( / 194.0.179.[x]xx)
3116Mozilla/5.0 (compatible; YoudaoBot/1.0;; )Youdao search (China) robot
3117Mozilla/5.0 (compatible; Zenbot/1.3; + robot for the Southern African Zen search service
3118Mozilla/5.0 (compatible; zermelo + [,]Powerset Natural Language Search crawler (under development) using Heritrix via Amazon Web Services
3119Mozilla/5.0 (compatible;archive.org_bot/1.7.1; collectionId=316; Archive-It; + - The Internet Archive's open-source crawler (
3120Mozilla/5.0 (compatible;archive.org_bot/heritrix-1.9.0-200608171144 + Machine Internet Archive crawler
3121Mozilla/5.0 (compatible;FindITAnswersbot/1.0;+ - Search engine for software developers
3122Mozilla/5.0 (compatible;MAINSEEK_BOT)Mainseek search (Poland) robot
3123Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; - Technical demo to take screenshot of any URL
3124Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b5) Gecko/2008032619 Firefox/3.0b5Mozilla Firefox 3.0 beta (Gran Paradiso) for MacOS
3125Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.0.1) Gecko/20021219 Chimera/0.6Chimera browser (Mozilla/Gecko engine) - now Camino Mac PowerPC
3126Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.0.1) Gecko/20030306 Camino/0.7Camino browser (Mozilla/Gecko engine) - ex Chimera Mac PowerPC
3127Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US) AppleWebKit/xx (KHTML like Gecko) OmniWeb/v5xx.xxOmniWeb 5.x.x Mac OS X browser
3128Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) AppleWebKit/xxx.x (KHTML like Gecko) Safari/12x.xSafari 1.2x browser (Mozilla/Gecko engine) MAC OS X
3129Mozilla/5.0 (Macintosh; U; PPC; en-US; rv:0.9.2) Gecko/20010726 Netscape6/6.1Netscape 6.x Mac PowerPC
3130Mozilla/5.0 ( robot from
3131Mozilla/5.0 (Sage)Sage - RSS and Atom feed reader extension for Mozilla Firefox
3132Mozilla/5.0 (Slurp/cat;; (Hotbot-Lycos - NBCi etc.) robot
3133Mozilla/5.0 (Slurp/si;; (Hotbot-Lycos - NBCi etc.) robot
3134Mozilla/5.0 (SunOS 5.8 sun4u; U) Opera 5.0 [en]Opera 5.x SunOS
3135Mozilla/5.0 (Twiceler-0.9 experimental web crawler
3136Mozilla/5.0 (Version: xxxx Type:xx)Some spambot from Romania ( & 86.123.65.xx) - Maybe email harvesting
3137Mozilla/5.0 ( search robot (64.5.245.xx /
3138Mozilla/5.0 (Windows; U; Win98; en-US; rv:0.9.2) Gecko/20010726 Netscape6/6.1Netscape 6.x Win98
3139Mozilla/5.0 (Windows; U; Win98; en-US; rv:x.xx) Gecko/20030423 Firebird Browser/0.6Firebird browser (Mozilla/Gecko engine) - ex Phoenix Win98
3140Mozilla/5.0 (Windows; U; Win9x; en; Stable) Gecko/20020911 Beonex/0.8.1-stableBeonex Communicator browser (Mozilla/Gecko engine)
3141Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.7) NimbleCrawler 1.11 obeys UserAgent NimbleCrawler For problems contact: crawler_at_dataalchemy.comHealthline health related search robot (72.5.115.xx)
3142Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/ Safari/525.19Google Chrome browser based on WebKit (Safari)
3143Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/20060731 Firefox/ Flock/ web browser built on Mozilla technologies
3144Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/20071127 Firefox/ x.0Megaupload Mega Manager - Download manager plugin for Firefox
3145Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/2008092215 Firefox/3.0.1 Orca/1.1 beta 3Orca browser - based on Gecko
3146Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.x.x) Gecko/20041107 Firefox/x.xFirefox browser (Mozilla/Gecko engine) - ex Firebird WinXP
3147Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.xx) Gecko/20030504 Mozilla Firebird/0.6Firebird browser (Mozilla/Gecko engine) - ex Phoenix WinXP
3148Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; Gecko/20041027 Mnenhy/ - enhanced mail & news Mozilla based browser
3149Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( robot
3150Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5Mozilla Firefox 3.0 beta (Gran Paradiso) for Win
3151Mozilla/5.0 (Windows; U;XMPP Tiscali Communicator v.10.0.1; Windows NT 5.1; it; rv: Gecko/20070309 Firefox/ Communicator - Online services suite
3152Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For problems contact: crawler@healthHealthline health related search robot (72.5.115.xx)
3153Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.comHealthline health related search robot (72.5.115.xx)
3154Mozilla/5.0 (X11; Linux i686; U;rv: 1.7.13) Gecko/20070322 Kazehakase/ - Gecko based browser (Japan)
3155Mozilla/5.0 (X11; U; Linux 2.4.2-2 i586; en-US; m18) Gecko/20010131 Netscape6/6.01Netscape 6.x Linux
3156Mozilla/5.0 (X11; U; Linux i686; de-AT; rv: Gecko/20060309 SeaMonkey/1.0SeaMonkey browser suite (ex Mozilla) on Linux
3157Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.7.6) Gecko/20050405 Epiphany/1.6.1 (Ubuntu) (Ubuntu package 1.0.2)Epiphany (Mozilla/Gecko engine) browser Linux
3158Mozilla/5.0 (X11; U; Linux i686; en-US; Nautilus/1.0Final) Gecko/20020408Nautilus (developed by 1.x Browser Linux
3159Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.3) Gecko/20010801Mozilla (Gecko) 0.9x browser Linux
3160Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); Gecko/20021130Spinn3r social network crawler
3161Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2b) Gecko/20021007 Phoenix/0.3Phoenix 0.3 browser (Mozilla/Gecko engine) - now Firebird Linux
3162Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040413 Epiphany/1.2.1Epiphany (Mozilla/Gecko engine) browser Linux
3163Mozilla/5.0 (X11; U; Linux i686; en-US; rv: Gecko/20060909 Firefox/ SnapPreviewBotSnap Firefox Search Plugin
3164Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1) Gecko/20061129 BonEcho/2.0Bon Echo Alpha - developer preview of future Firefox browser
3165Mozilla/5.0 (X11; U; Linux i686; en-US; rv: Gecko/20061205 Iceweasel/ (Debian- - the GNU version of the Firefox browser
3166Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9a8) Gecko/2007100619 GranParadiso/3.0a8Mozilla Firefox 3.0 beta (Gran Paradiso) for Linux
3167Mozilla/5.0 Galeon/1.0.2 (X11; Linux i686; U;) Gecko/20011224Galeon 1.x Browser Linux
3168Mozilla/5.0 gURLChecker/0.x.x (Linux)gURLChecker - GNOME link checking tool
3169Mozilla/5.0 URL-SpiderURL Spider - used by
3170Mozilla/5.0 spider
3171Mozilla/5.0 search robot (64.5.245.xx /
3172Mozilla/5.0 whoiam [] German business directory
3173Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 : Collaborative Web Resource Categorization and Ranking Project robot
3174MQbot project's MetaQuerier robot
3175MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler;; project's MetaQuerier robot
3176MSFrontPage/4.0MS Frontpage 4.x
3177MSIE 4.0 (Win95)Some faked UA - maybe for a download manager
3178MSIE-5.13 (larbin@unspecified.mail)unknown robot from (209.126.176.x)
3179msnbot-media/1.0 (+ media search robot
3180msnbot-Products/1.0 (+ Live product search (Beta) robot
3181MSNBOT/0.xx ( Search robot - -
3182msnbot/x.xx ( Search robot - -
3183MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default)Microsoft search for mobiles
3184MSNPTC/1.0MSN Search robot - -
3185MSProxy/2.0Microsoft proxy server
3186MSRBOTMacEdition CodeBitch link checking
3188Mulder, VCR-1.0StreamBox VCR user agent
3189multiBlocker browserMultiblocker (Fantomaster) anonymity software user
3190multicrawler ( for DERI Galway's Semantic Web Search Engine cluster
3191MultiText/0.1Virginia Tech Digital Library Research Laboratory robot
3192MusicWalker2.0 (! musical directory link checking
3193MVAClientUnknown bad bot from diff. Taiwanese IPs
3194My WinHTTP ConnectionWindows HTTP Services (WinHTTP)
3195myDaemonunknown user robot (
3196MyGetRight/1.0.0GetRight download manager
3197MyGetRight/1.0bGetRight download manager Crawler 2.0Mylinea France web catalogue crawler
3199mylinkcheck/1.02VDOG - SEO webdirecory (Germany) link checking
3200Naamah 1.0.1/Blogbot ( (Germany) robot
3201Naamah 1.0a/Blogbot ( (Germany) robot
3202NABOT/5.0Naver Japan / Korea robot
3203nabot_1.0Naver Japan / Korea robot
3204NameOfAgent (CMS Spider)Badbot searching for Wordpress wp-login.php
3206NASA Search 1.0Unknown spambot / harvester from diff. IPs
3207NationalDirectory-WebSpider/1.3Nationaldirectory spider
3208NationalDirectoryAddURL/1.0Nationaldirectory spider
3209NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / Japan / Korea robot
3210NaverBot_dloader/1.5Naver Japan / Korea robot
3211NavissoBotNavisso closed beta robot (
3212NavissoBot/1.7 (+ closed beta robot (
3213NCSA Beta 1 ( Information Archival robot
3214Nebullabot/2.2 ( distributed crawler (Germany)
3215NEC Research Agent -- compuman at Researchindex robot - now CiteSeer.IST scientific document index
3216NEC-Hayek/ user agent ? NEC Researchindex robot ?
3217Net-Seekr Bot/Net-Seekr Bot V1 ( Seekr search robot
3218NetAnts/1.2xNetAnts download manager
3219NETCOMplete/x.xxNetComplete IE browser package
3220NetinfoBot/1.0 ( search (Bulgaria) robot
3221NetLookout/2.24Netlookout internet notifier
3222Netluchs/0.8-dev ( ;; ___don' (Germany) search (193.164.8.xx)
3223NetMechanic Vx.0NetMechanic link checker
3224NetNewsWire/2.x (Mac OS X; NetNewsWire - Mac RSS feed reader
3225NetNoseCrawler/v1.0unknown InCom (216.0.107.xx) robot
3226Netprospector JavaCrawlerNetprospector metasearch software
3227NetPumper/x.xxNetpumper download manager
3228NetResearchServer( robot (209.87.232.x)
3229NetResearchServer/x.x( search engine using DMOZ database
3230NetSeer/Nutch-0.9 (NetSeer Crawler;; search (beta) crawler via Amazon Web Services - see also Teemer
3231NetSprint -- 2.0Wirtualna Polska / Netsprint search (Poland) robot
3232NetWhatCrawler/0.06-dev (NetWhatCrawler from;; Search crawler (
3233NetZippyNetzippy robot
3235newsearchengine (ThisUser@unspecified.mail)Unknown ( client robot
3236NewsGator FetchLinks extension/0.2.0 ( plugin for NewsGator RSS reader
3237NewsGatorOnline/2.0 (; 1 subscribers)NewsGator online RSS reader
3238NextGenSearchBot 1 (for information visit Crawler for Business People Search
3239NextopiaBOT (+ distributed crawler client beta v0.xNextopia crawler
3240NG-Search/0.90 (NG-SearchBot;; )find your keywords - semantic search (Germany) robot
3241NG/1.0Exalead (France) search robot (193.47.80.xx)
3242NG/4.0.1229Exalead Websearch image crawler (193.47.80.xx)
3243nicebotUnknown UA from PlanetLab distributed network
3244NICO/1.0NicoZone childsafe search robot
3245Nikita the Spider ( the Spider - Online HTML validation , link checking
3246NITLE Blog Spider/0.01Experimental LSI (?) robot from 140.233.69.xx (
3247Nitro Downloader 1.x ( manager
3248Noago SpiderNoago spider
3249Nocilla/ user robot
3250Nokia-WAPToolkit/1.2 googlebot(at)googlebot.comGoogle WAP robot
3251Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi crawler for mobile search
3252Nokia6610/1.0 (3.09) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible;YahooSeeker/M1A1-R2D2; - Yahoo mobile web crawling robot
3253Nokia7110/1.0 (05.01) (Google WAP Proxy/1.0)Google WAP proxy
3254NokodoBot/1.x (+ public beta search robot (67.18.222.xx)
3255Norbert the Spider( UK Search Engine robot
3256noxtrumbot/1.0 ( search robot (Spain)
3257noyona_0_1Noyona job search (preview)
3258NP/0.1 (NP;; copyright search robot (24.177.134.x)
3259NPBot ( copyright search robot (24.177.134.x)
3260NPBot-1/2.0Nameprotect copyright search robot (24.177.134.x)
3261Nsauditor/1.xNsauditor Network Security Auditor
3262NSPlayer/10.0.0.xxxx WMFSDK/10.0NetShow Media Player = Windows Media Player 10 (; social search application (; social search application
3265nttdirectory_robot/0.9 ( Directory robot
3266Nucleus SiteList LinkChecker/1.1Nucleus CMS SiteList link managing plugin
3267nuSearch Spider (compatible; MSIE 4.01)nuSearch spider (
3268NuSearch Spider (compatible; MSIE 6.0)nuSearch spider (
3269NuSearch Spider www.nusearch.comnuSearch spider (
3270NutchNutch open source robot
3271Nutch crawler/Nutch-0.9 (; search for handheld devices using Nutch
3272Nutch/Nutch-0.9 (Eurobot; )Ayell Euronet business directory robot using Nutch
3273NutchCVS/0.06-dev (Nutch;; content filtering engine ( powered by Nutch
3274NutchCVS/0.0x-dev (Nutch;; open source robot
3275NutchCVS/0.7.1 (Nutch running at UW;; from University of Washington Computer Science & Engineering (
3276NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.;; ec2test at Elastic Compute Cloud (Amazon EC2) robot
3277NutchOrg/0.0x-dev (Nutch;; open source robot
3278nutchsearch/Nutch-0.9 (Nutch Search 1.0; herceg_novi at yahoo dot com)Unknown robot using Nutch (maybe private crawling) via Cox network (
3279NutchVinegarCrawl/Nutch-0.8.1 (Vinegar;; eytanadar at gmail dot com)Unknown crawler from University of Washington - Computer science
3280obidos-bot (just looking for books.)Weblog bookwatch robot
3281ObjectsSearch/0.01-dev (ObjectsSearch;; Search robot
3282ObjectsSearch/0.0x (ObjectsSearch;; Search robot
3283oBot ((compatible;Win32))Cobion Germany Brand Protection Services robot
3284Ocelli/1.x ( Engineering Search robot (
3285OctopusOctopus download manager
3286Octora Beta - www.octora.comOctora blog or RSS information crawler - beta (66.228.114.xx)
3287Octora Beta Bot - www.octora.comOctora RSS feed search
3288Offline Explorer 1.*Meta Products Offlinebrowser
3289OliverPerryClaymont robot / Internetseer Web Site Monitoring
3290OmniExplorer_Bot/1.0x (+ Internet CategorizerOmniExplorer car & shopping search ( car & shopping search (
3291OmniExplorer_Bot/1.0x (+ Job CrawlerOmniExplorer car & shopping search (
3292OmniExplorer_Bot/1.1x (+ Torrent CrawlerOmniExplorer car & shopping search (
3293OmniExplorer_Bot/x.xx (+ WorldIndexerOmniExplorer car & shopping search (
3294onCHECK Linkchecker von fuer German web directory link checking
3295onCHECK-Robot, German web directory link checking SA- Szukaj (Search) robot (
3297online link validator ( link validation spider
3298Online24-Bot (Version: 1.0x, powered by shopping portal (Germany) link checking
3299OntoSpider/1.0 libwww-perl/5.65OntoSpider - Dutch robot for a research project. (195.11.244.xx)
3300OOZBOT/0.20 ( ; agentname at setooz dot_com )SeetooZ search crawler
3301OpenAcoon v4.0.x ( open source search engine (used by Acoon search)
3302Openbot/3.0+(;+ robot
3303Openfind data gatherer- Openbot/3.0+(;+ robot
3304Openfind Robot/ robot
3305OpenISearch/1.x ( i search robot - search engine in development
3306OpenTaggerBot ( social bookmarking system
3307OpenTextSiteCrawler/2.9.2OpenText crawler
3308OpenWebSpider/0.x.x ( - Open Source web search engine
3309OpenWebSpider/xOpenWebSpider - Open Source web search engine
3310Opera/5.0 (Linux 2.0.38 i386; U) [en]Opera 5.0 Linux
3311Opera/5.11 (Windows ME; U) [ru]Opera 5.11 faked WinME referer
3312Opera/5.12 (Windows 98; U) [en]Opera 5.12 Win98
3313Opera/6.01 (larbin@unspecified.mail)unknown robot from (209.126.176.x)
3314Opera/6.x (Linux 2.4.8-26mdk i686; U) [en]Opera 6.x- Mandrake Linux
3315Opera/6.x (Windows NT 4.0; U) [de]Opera 6.x WinNT
3316Opera/7.x (Windows NT 5.1; U) [en]Opera 7.x WinXP
3317Opera/8.xx (Windows NT 5.1; U; en)Opera 8.x (Beta) WinXP
3318Opera/9.0 (Windows NT 5.1; U; en)Opera 9 (Beta) Browser
3319Opera/9.00 (Windows NT 5.1; U; de)Opera 9 (final)
3320Opera/9.60 (Windows NT 5.1; U; de) Presto/2.1.1Opera browser 9.6x on WinXP (Presto = Operas rendering engine)
3321OpidooBOT (larbin2.6.3@unspecified.mail)Opidoo Search Belgium robot (
3322OPWV-SDK UP.Browser/ (GUI) MMP/2.0 Push/POOpen Wave Phone Simulator SDK
3323Oracle Application Server Web Cache 10gOracle Application Server cache
3324Oracle iMTCrawlerOracle interMedia Text - Text and web documents indexing
3325Oracle Ultra SearchOracle Search
3326OrangeSpiderOrangeslicer semantic search (Beta) Germany (
3327Orbiter/T-2.0 (+ - DailyOrbit search spider
3328Orca Browser ( browser - based on Gecko
3329OSSProxy 1.3.305.321 (Build 305.321 Win32 en-us)(Dec 21 2005 16:30:54)Marketscore (was Netsetter) internet accelerator
3330OutfoxBot/0.x (For internet experiments; http://; robot from Chinanet (
3331OutfoxMelonBot/0.5 (for internet experiments; http://; robot from Chinanet (
3332Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; crawler (
3333OWR_Crawler 0.1Unknown robot from 198.169.127.xx (
3334ozelot/2.7.3 (Search engine indexer;; - Flying Cat's search engine robot (Germany)
3335PADLibrary - PAD file software robot for
3336PageBitesHyperBot/600 ( job search crawler
3337Pagebull visual search engine
3338Pagestacker BotPagestacker online bookmark service
3339page_verifier ( Computing SmartFilter Tools - malware crawler
3340PagmIEDownloadDownloadmanager ?
3341parallelContextFocusCrawler1.1parallelContextFocusCrawler1.1CFC crawler used by Italian academic and research network (GARR)
3342ParaSite/1.0b ( robot
3343Patwebbot ( (Germany) robot
3344pavuk/0.9pl29b i686-pc-linux-gnuPavuk web downloading program for Unix
3345PBrowse 1.4bSome site scanning tool via diff. IPs- i.e.: - (
3346pd02_1.0.0 / robot
3347PEAR HTTP_Request class ( )Pear HTTP_Request PHP extension package
3348PEERbot www.peerbot.comPeerbot - favicon search robot
3349PeopleChat/Search_EngineUnknown robot from (
3350PEval 1.4bSome site scanning tool via diff. IPs
3351PHP/3.x.xxdiff. IPs / services
3352PHP/4.0.4pl1diff. IPs / services
3353PHP/4.0.6diff. IPs / services- i.e.: -NTT/Verio Inc. link checker
3354PHP/4.1.1diff. IPs / services- i.e.: - link checking
3355PHP/4.1.2diff. IPs / services- i.e.: - 209.114.200.xx = MyNetCrawler link checking - = Mixcat crawler
3356PicoSearch/1.0Pico Search robot
3357Piffany_Web_Scraper_v0.xPiffany targeted search web spider
3358Piffany_Web_Spider_v0.xPiffany targeted search web spider
3359PigeonBot1.0 BETAWhois Source domain name information robot (66.249.26.xx)- s.also: - SurveyBot
3360PingALink Monitoring Services 1.0PingALink website monitoring
3361PingALink Monitoring Services 1.0 ( website monitoring
3362Pingdom GIGRIB ( web site monitoring
3363pipeLiner/0.3a (PipeLine Spider;; webmaster'at' search (DMOZ based) search robot (24.106.39. xxx)
3364pipeLiner/0.xx (PipeLine Spider; search (DMOZ based) search robot (24.106.39. xxx)
3365PitaPita crawler
3366Pizilla++ ver 2.45Private user-agent via Hurricane Electric Internet Services
3367PJspider/3.0 (; spider
3368Plagger/0.x.xx ( - pluggable RSS/Atom feed aggregator written in Perl
3369PlagiarBot/1.0unknown robot
3370PlantyNet_WebRobot_V1.9 dhkang@plantynet.comPlantynet web filtering services - Blacklist DB robot
3371plinki/0.1 (you got plinked! (thats a good thing..);; UA from
3372PluckFeedCrawler/2.0 (compatible; Mozilla 4.0; MSIE 5.5;; 1 subscribers)Pluck RSS feed crawler
3373Pluggd/Nutch-0.9 (automated crawler;support at pluggd dot com)Pluggd Podcast search engine
3374Pockey-GetHTML/4.12.0 (Win32; GUI; ix86)Yutaka Endo's Pockey / GetHTML / GetHTMLW - some downloading tool from Japan
3375Pockey-GetHTML/x.xxYutaka Endo's Pockey / GetHTML / GetHTMLW - some downloading tool from Japan
3376Pockey/x.x.xYutaka Endo's Pockey / GetHTML / GetHTMLW - some downloading tool from Japan
3377Pockey7.x.x(WIN32GUI)Yutaka Endo's Pockey / GetHTML / GetHTMLW - some downloading tool from Japan
3378POE-Component-Client-HTTP/0.64 (perl; N; POE; en; rv:0.640000)HTTP user-agent for POE (portable networking framework for Perl )
3379PoirotThePlanet/ Google Adsense refferer spam bot from 70.85.116.* / /
3380polybot 1.0 ( webcrawler
3381Pompos/1.x / Iliad French recherche robot
3382Pompos/1.x pompos@iliad.frIliad / Free French recherche robot
3383Popdexter/1.0Popdex - web site popularity crawler
3384Port Huron LabsUnknown spam bot / harvester (63.223.10.***)
3385PortalBSpider/2.0 ( (now Alacra search) spider
3386portalmmm/2.0 S500i(c20;TB)portalmmm IMode mobile browser
3387PostFavoritesYahoo ( / 216.109.121.xx) favorites tracking robot
3388potbot 1.0Potbot : A simple IRC bot written in Perl
3389PRCrawler/Nutch-0.9 (data mining development project; Rialto - data mining development project
3390PrivacyFinder Cache Bot v1.0CUPS robot for AT&T Privacy Bird Privacy Preferences (P3P) enhancements
3391PrivacyFinder/1.1CUPS robot for AT&T Privacy Bird Privacy Preferences (P3P) enhancements
3392Privoxy/3.0 (Anonymous)Privoxy web proxy
3393Production Bot 0116BSome site scanning tool from diff. IPs- i.e.: - 67.99.33.x (
3394Production Bot 2016BSome site scanning tool from diff. IPs- i.e.: - 216.232.64.xx (
3395Production Bot DOT 3016BSome site scanning tool from diff. IPs- i.e.: - (
3396Program Shareware 1.0.2Some spam bot
3397Progressive Downloadunknown
3398Progressive Download HTTP checkunknown
3399Project XP5 [2.03.07-111203]XP5 robot
3400PROve AnswerBot 4.0Answerchase PROve Answerbot
3401ProWebGuide Link Checker ( robot
3402psbot/0.1 (+ robot (
3403PSurf15a 11Some site scanning tool via diff. IPs- i.e.: QWest Net
3404PSurf15a 51Some site scanning tool via diff. IPs- i.e.: Optonline net (
3405PSurf15a VASome site scanning tool via diff. IPs- i.e.: - ( - ( - (
3406psychecloneUnknown website grabbing / ripping for unknown purposes from 208.66.195.x - Russia
3407PubCrawl ( robot from Stanford University ( = PubCrawl.Stanford.EDU)
3408puf/0.91beta6a (Linux 2.2.18; i686)Parallel URL Fetcher downloading tool
3409puf/0.93.2a (Linux 2.4.18; i686)Parallel URL Fetcher downloading tool
3410pulseBot (pulse Web Miner)WebarooBot - Webaroo web site search / theme based downloading tool (
3411PureSightPureSight Internet content filter
3412PuxaRapido v1.0Puxa Rapido download manager
3413PWeBot/1.2 Inspector ( PWeBot link checking (Argentina)
3414PycURLFast Search robot (using PycURL Python component- s. below)
3415PycURL/7.xx.xPycURL - Python interface to libcurl
3416Python-urllib/1.1xPython URL fetcher - robot used by Naver Japan/Korea
3417Python-urllib/2.0a1Python URL fetcher - robot used by Google Web Directory ( Web Directory robot
3419QEAVis Agent/Nutch-0.9 (Quantitative Evaluation of Academic Websites Visibility; Quantitative Evaluation of Academic Websites Visibility using Nutch
3420QPCreep Test Rig ( We are not indexing- just testing )Quepasa!com (Latin American search) robot
3421QuepasaCreep ( )Quepasa!com (Latin American search) robot
3422QuepasaCreep v0.9.1xQuepasa!com (Latin American search) robot
3423QueryN MetasearchQueryN Metasearch robot
3424Quicksilver (Blacktree,MacOSX)Blacktrees Quicksilver helper application for Mac
3425QuickTime\xaa.7.0.4 (qtver=7.0.4;cpu=PPC;os=Mac 10.3.9)Quicktime for Macintosh
3426QweeryBot/3.01 ( for the Qweery search engine (in development) - Netherland
3427Qweery_robot.txt_CheckBot/3.01 ( for the Qweery search engine (in development) - Netherland
3428R6_CommentReader_( RSS feed comment crawler
3429R6_FeedFetcher_( Rss feed crawler
3430rabaz (rabaz at gigabaz dot com)gigaBaz - the brainbot (Germany) robot
3431RaBot/1.0 Agent-admin/phortse@hanmail.netDAUMOA - Daum search Korea robot (
3432Rainbot1.1Bot Provider for the All Womans Bot Service?
3433ramBot xtreme (was robot (Germany)
3434RAMPyBot - (RAMPyBot -;; Search Engine robot (
3435RAMPyBot/0.8-dev (Nutch;; Search Engine robot (
3436Rank Exec ( Reciprocal Link Manager 1.x/botRank Exec reciprocal link checking
3437Rankivabot/3.2 (; 3.2; vzmxikn)Rankiva website popularity robot
3438Rational SiteCheck (Windows NT)Innova/IBM Rational SiteCheck - Rational robot
3439RAYSPIDER/Nutch-0.9Unknown spider from Raytheon Company - maybe Raytheon High Speed Guard proxy
3440ReadABlog Spider (compatible; 1.1; feed update; A Blog - RSS feed and blog search engine
3441RealDownload/ download manager
3442REAP-crawler Nutch/Nutch-1.0-dev (Reap Project;; Reap Project)The REAP Web Crawler for the REAP project
3443Reaper [2.03.10-031204] ( robot for SiteSearch
3444Reaper/2.0x (+ robot for SiteSearch
3445REBOL Core 2.x.x.x.xREBOL messaging language for distributed Internet apps
3446REBOL View 1.x.x.x.xREBOL/View - machine independent internet client application
3447RebusnetBot (+ software site - link / submission checking
3448RebusnetPADBot/1.5x (+ software site - link / submission checking
3449reciprocal links checker ( reciprocal link checker
3450RedBot/redbot-1.0 ( Crawler; redbot at rediff dot com) search link checking
3451RedCarpet/1.2 ( crawler for Pronto price comparison search(
3452RedCell/0.1 (InfoSec Search Bot (Coming Soon);; Bot for's security related search (
3453RedCell/0.1 (RedCell;; Bot for's security related search (
3454RedKernel WWW-Spider 2/0 (+ Softwares robot
3455REL Link Checker Lite x.xREL Link Checker Lite free version of Web Link Validator
3456RepoMonkey Bait & Tackle/v1.01unknown
3457Rewebber/1.2 libwww-perl/5.41Rewebber proxy service
3458rico/0.1Applied Semantics Auto-Categorizer for QWestDex Direct
3459RixBot ( Rebol Indexer for the RIX - Rebol related search (195.204.121.xx)
3460RMA/1.0 (compatible; RealMedia)StreamBox VCR user agent
3461RMA/1.0 (compatible; RealMedia)Real Media server acting as client
3462RoboCrawl ( Content search crawler
3463RoboCrawl ( Content Search (207.44.220.xx) robot
3464RoboPal ( Australia metasearch robot (61.68.139.xx)
3465Robot/www.pj-search.comPopJapanSearch robot
3466Robot: NutchCrawler- Owner: wdavies@acm.orgExperimental robot using Wget via
3467Robot@SuperSnooper.ComSupersnooper robot
3468Robozilla/1.0Netscape Directory / DMOZ Open Directory link crawler
3469Rome Client ( Ver: 0.9ROME - Open source Java tools for RSS and Atom feeds
3470Rotondo/3.1 libwww/ robot
3471RPT-HTTPClient/0.3-xdifferent IPs using the HTTPClient library (mostly link checking)
3472RRC ( ( robot
3473RssBandit/ (.NET CLR 1.1.4322.2407; WinNT 5.1.2600.0; (.NET CLR 1.1.4322.2407; WinNT 5.1.2600.0; )RSS Bandit RSS/Atom reader for .NET framework RSS/Atom Feed RobotRSS Micro Search - RSS feed search engine
3475RSSOwl/1.2.3 2006-11-26 (Windows; U; zhtw)RSSOwl embedded RSS feed reader
3476RSSOwl/1.2.4 Preview Release 2007-04-15 (Windows; U; zhtw)RSSOwl embedded RSS feed reader
3477RssReader/1.0.xx.x ( Microsoft Windows NT 5.1.2600.0Ykoon RssReader news feed reader
3478RSurf15a 41Some site scanning tool via diff. IPs- i.e.: - ( -
3479RSurf15a 51Some site scanning tool via diff. IPs- i.e.: - ( -
3480RSurf15a 81Some site scanning tool via diff. IPs- i.e.: - ( -
3481Rubbot/1.0 (+ blog spider
3482RufusBot (Rufus Web Miner; - Webaroo web site search / theme based downloading tool (
3483RufusBot (Rufus Web Miner; - Webaroo web site search / theme based downloading tool (
3484Rumours-Agentunknown robot from (
3485RX BarRX (Reflexive Search) Bar for IE
3486S&L Spider ( & Links directory spider
3487S.T.A.L.K.E.R. ( link checking ?
3488SafariBookmarkChecker (+ for Mac OS X
3489sait/Nutch-0.9 (SAIT Research; robot - unknown robot from Samsung International Korea
3490SandCrawler - Compatibility TestingSandcrawler robot from Microsoft (131.107.0.xx)
3491SapphireWebCrawler/1.0 (Sapphire Web Crawler using Nutch;; Web Crawler from Carnegie Mellon University's Language Technologies Institute
3492SapphireWebCrawler/Nutch-1.0-dev (Sapphire Web Crawler using Nutch;; Web Crawler from Carnegie Mellon University's Language Technologies Institute
3493savvybot/0.2WebSavvy Directory robot
3494SBIder/0.7 (SBIder;; SBIder Nutch based crawler
3495SBIder/0.8-dev (SBIder;; SBIder Nutch based crawler
3496SBL-BOT ( Labs Black Widow web site ripper
3497ScanWebScanWeb - regular expression based web page searching tool
3498ScholarUniverse/0.8 (Nutch;+; - Scholarly experts search robot Spider Pro (USP) used by German Schwarzmann GmbH
3500Science Traveller International 1X/1.01X Web Browser
3501ScollSpider/2.0 (+ UK search engine robot (
3502Scooter-3.0.EUAltavista robot
3503Scooter-3.0.FSAltavista robot
3504Scooter-3.0.HDAltavista robot
3505Scooter-3.0.VNSAltavista robot
3506Scooter-3.0QIAltavista robot
3507Scooter-3.2Altavista robot
3508Scooter-3.2.BTAltavista robot
3509Scooter-3.2.DILAltavista robot
3510Scooter-3.2.EXAltavista robot
3511Scooter-3.2.JTAltavista robot
3512Scooter-3.2.NIVAltavista robot
3513Scooter-3.2.SF0Altavista robot
3514Scooter-3.2.snippetAltavista robot
3515Scooter-3.3devAltavista robot
3516Scooter-ARS-1.1Altavista robot
3517Scooter-ARS-1.1-ihAltavista robot
3518scooter-venus-3.0.vnsAltavista robot
3519Scooter-W3-1.0Altavista robot
3520Scooter-W3.1.2Altavista robot
3521Scooter/1.0Altavista robot
3522Scooter/1.0 scooter@pa.dec.comAltavista robot
3523Scooter/1.1 (custom)Altavista robot
3524Scooter/2.0 G.R.A.B. V1.1.0Altavista robot
3525Scooter/2.0 G.R.A.B. X2.0Altavista robot
3526Scooter/3.3Altavista robot
3527Scooter/3.3.QA.pczukorAltavista robot
3528Scooter/3.3.vscooterAltavista robot
3529Scooter/3.3_SFAltavista robot
3530Scooter2_Mercator_x-x.0Altavista using Mercator robot
3531Scooter_bh0-3.0.3Altavista robot
3532Scooter_trk3-3.0.3Altavista robot
3533Scope (Mars+)Scope Navigator mobile browser (Japan)
3534ScoutAboutSome robot using Research Republic ScoutAbout Research Tool
3535ScoutAnt/0.1; + search robot
3536scoutmasterScoutMaster information retrieval software
3537Scrubby/2.x ( the web robot (66.93.156.xx)
3538Scrubby/3.0 (+ the web robot (66.93.156.xx)
3539Search+URL Search+ search software
3540Search-Engine-StudioXtreem Search Engine Studio - SE software robot V1.4.2 (; robot
3543Search/1.0 ( Search web indexing / site searching tool
3544searchbot admin@google.comUnknown robot / website grabber from Chinatelecom (
3545SearchByUsa/2 (SearchByUsa;; robot (
3546SearchdayBotSearchday (Germany) search robot
3547SearchExpress Spider0.99Searchexpress spider
3548SearchGuild/DMOZ/Experiment ( forum & directory robot (
3549SearchGuild_DMOZ_Experiment ( forum & directory robot (
3550Searchit-Now Robot/2.2 (+ robot (69.93.107.xx)
3551Searchmee! Spider v0.98aSearchmee! Search Engine (prototype) robot by
3552SearchSight/2.0 ( search robot robot
3554Searchspider/1.2 (SearchSpider;; robot
3555SearchTone2.0 - IDEAREJanas ( / robot
3556Seekbot/1.0 ( HTTPFetcher/0.3seekport. beta search (Germany) robot
3557Seekbot/1.0 ( RobotsTxtFetcher/1.0 (XDF)seekport. beta search (Germany) robot
3558Seekbot/1.0 ( RobotsTxtFetcher/1.2seekport. beta search (Germany) robot
3559Seeker.lookseek.comLookseek search robot / link checking web filtering software
3561Semager/1.1 ( (was NG-Search) semantic search - Germany
3562Semager/1.x ( (was NG-Search) semantic search - Germany
3563semanticdiscovery/0.xSemantic Discovery domain checking tool
3564Sensis Web Crawler (search_comments\at\sensis\dot\com\dot\au)Sensis Australia search robot Web Crawler (search_comments\at\sensis\dot\com\dot\au)Sensis Australia search robot
3566SeznamBot/1.0Seznam Search (Czech Republic) robot
3567SeznamBot/1.0 (+ Search (Czech Republic) robot
3568SeznamBot/2.0-test (+ Search (Czech Republic) robot
3569ShablastBot 1.0Unknown robot from - Website has no content - Ignores robots.txt
3570Shareaza v1.x.x.xxShareaza P2P peer-to-peer download client
3571SharewarePlazaFileCheckBot/1.0+(+ File Check Bot - link checking
3572Shim CrawlerChikayama-Taura Lab Shim-Crawler used for The Kototoi Project (Japan) - (133.11.36.xx)
3573Shim-Crawler(Mozilla-compatible;; Lab Shim-Crawler used for The Kototoi Project (Japan) - (133.11.36.xx)
3574ShopWiki/1.0 ( + shopping search based on LittleWiki search
3575ShopWiki/1.0 ( + shopping search based on LittleWiki search Crawler 2.0Shoula Search Engine crawler
3577SietsCrawler/1.1 (+ Crawler - Web based site crawling application
3578Sigram/Nutch-1.0-dev (Test agent for Nutch development;; bot at sigram dot com)Sigram's Nutch robot - crawler testing
3579Siigle Orumcex v.001 Turkey ( search (Turkey) robot
3580silk/1.0Slider Search directory robot (
3581silk/1.0 (+ Search directory robot (
3582SimpleFavPanel/1.2SimpleFavPanel - IE newsfeed panel plugin
3583Simpy 1.x; bookmarking and personal search engine
3584Simpy/1.x (Simpy;; feedback at simpy dot com)Simpy bookmarking and personal search engine
3585Sirketcebot/v.01 (çe search - Turkey
3586SiteBar/3.x.x (Bookmark Server; online bookmark manager
3587SiteBar/x.xSiteBar bookmark server
3588SiteBar/x.x.x (Bookmark Server; bookmark server
3589sitecheck.internetseer.comInternetseer Web Site Monitoring / Claymont robot (For more info see: Web Site Monitoring
3591SiteRecon+(xx)SiteRecon website monitoring spider at xx minute intervals
3592SiteSnaggerPC Magazin web site downloadmanager
3593SiteSpider +( Spider robot (66.249.17.xx)
3594SiteSucker/1.x.xSiteSucker Mac website downloading tool
3595SiteTaggerBot ( bookmark organizer site rating systemSiteTruth - Automatic site legitimacy rating system (217.146.159.xx) internet filter
3598SiteXpertXtreeme SiteXpert sitemap & search engine builder
3599Skampy/0.9.x ( directory link checker
3600Skimpy/0.x ( directory link checker
3601Skywalker/0.1 (Skywalker; anonymous; anonymous)Visvo distributed website crawler based on Nutch
3602Slarp/ robot
3603SleipnirSleipnir - Japanese Explorer based browser & search bar
3604Sleipnir Version 1.xxSleipnir - Japanese Explorer based browser & search bar
3605Sleipnir Version2.xSleipnir - Japanese Explorer based browser & search bar
3606Sleipnir/2.xxSleipnir - Japanese Explorer based browser & search bar
3607Slider_Search_v1-deSlider Search directory robot (
3608SlimBrowserSlim Browser (IE based browser) - uses this user agent for favicon.ico only
3609Slurp/2.0 (; (Hotbot-Lycos-NBCi) robot
3610Slurp/2.0-KiteWeekly (; (Hotbot-Lycos-NBCi) robot
3611Slurp/si (; (Hotbot-Lycos-NBCi) robot
3612Slurpy Verifier/1.0Inktomi (Hotbot-Lycos-NBCi) robot - 72.30.61.xx(x)
3613SlySearch ( robot (now Turnitin robot)
3614SlySearch/1.0 robot (now Turnitin robot)
3615SlySearch/1.x http://www.slysearch.comSlysearch robot (now Turnitin robot)
3616SmartDownload/1.2.67 (Win32; Jan 12 1999)Netzip/Smartdownload download manager
3617SmartDownload/1.2.77 (Win32; Feb 1 2000)Netzip/Smartdownload download manager
3618SmartDownload/1.2.77 (Win32; Jun 19 2001)Netzip/Smartdownload download manager
3619smartwit.comLoop Improvements NRS Enterprise search (69.44.155.xx[x])
3620SmiffyDCMetaSpider/1.0SmiffyDCMetaSpider - Robot to check the retro-adding of Dublin Core metadata
3621sna-0.0.1 ( PHP-client
3622sna-0.0.1 mikeelliott@hotmail.comSnoopy PHP-client beta crawler v0Unknown bot from ( also as Snapbot/1.0
3624Snapbot/1.0Unknown bot from ( - also as
3625Snapbot/1.0 (Snap Shots, + bot from Psinet / Cogentco - not from
3626Snappy/1.1 ( )My UrlTrends online web ranking service
3627Snarfer/0.x.x ( RSS reader
3628SnoopRob/x.xUnknown robot from 217.229.156.xx (T-Online Germany)
3629Snoopy v1.xxSnoopy PHP-client
3630Snoopy v1.xx- : User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MyIE2)Snoopy PHP-client
3631Snoopy_v0.xxSnoopy PHP-client
3632SnykeBot/0.6 ( France robot
3633SocSciBot ()Link crawler for the social sciences
3634SoftBank/1.0/812SH/SHJ001 Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1NetFront browser on Softbank mobile phone
3635SoftHypermarketFileCheckBot/1.0+(+ Hypermarket link checking
3636Softizerbot ( software directory link checking
3637sogou develop spiderUnknown UA from Chinanet ( faking Sogou search robot
3638Sogou Orion spider/3.0(+ UA from Chinanet (220.181.18.xx) faking Sogou search robot
3639sogou spiderUnknown UA from Chinanet ( faking Sogou search robot
3640Sogou web spider/3.0(+ UA from Chinanet ( faking Sogou search robot
3641sohu agentUnknown UA from Chinanet ( faking Sogou search robot
3642sohu-searchSohu (Search Fox) search robot China (
3643Sosospider+(+ search (China) spider
3644Space Bison/0.02 [fu] (Win67; X; SK)Default Proxomitron (discontinued) filtering proxy user agent identifier
3645SpeedDownload/1.xSpeed Download (Mac) download manager
3646speedfind ramBot xtreme robot
3647Speedy Spider (Beta/x.x; search robot
3648Speedy Spider (Entireweb; Beta/1.0; search spider
3649Speedy_Spider ( search robot
3650Sphere Scout&v4.0 - scout at sphere dot comSphere blog and news search robot
3651SphiderSphider - a lightweight search engine in PHP
3652Spida/ robot
3653Spider-Sleek/2.0 (+ ODP/DMOZ spider
3654spider.batsch.comBatsch robot - v:1.04TerraNautic spider for Schnellsuchen touristic search (Germany) - www.yellopet.comYellopet spider
3657Spider/ admin@maxbot.comMaxbot .gov .mil .edu indexing robot
3658SpiderKU/0.xUnknown robot from CPE at Kasetsart University (
3659SpiderManYahoo Search user agent or spider (
3660SpiderMonkey/7.0x ( info at Canada robot
3661Spinne/ robot
3662Spinne/2.0 medMedkatalog (medical catalogue) Austria robot
3663Spinne/2.0 med_AHMedkatalog (medical catalogue) Austria robot
3664Spock Crawler ( - people search application - via Amazon web services (Version: 1.02- powered by (German sports related search) link checking / robot
3666sproose/0.1-alpha (sproose crawler;; personalized search (38.100.225.xx)
3667SQ WebscannerSQ Webscanner Mac download manager
3668Squid-PrefetchSimple page-prefetch for Squid web proxy
3669squidclamSquidclam is a replacement for SquidClamAV-Redirector
3670SquidClamAV_Redirector 1.x.xSCAVR - Squid helper script for scanning download URLs for viruses
3671Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu)AOL Search / Pacific Internet Exchange robot
3672Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)diff. IPs / services i.e.: - robot - Websense (Internet filtering) robot
3673Sqworm/2.9.89-BETA (beta_release; 20020130-839; i686-pc-linux-gnu)Time Warner Telecom user robot ?
3674SSurf15a 11Some site scanning tool via diff. IPs i.e.: - ( - (216.108.198.xx)
3675StackRambler/x.x Rambler search (Russia) robot (81.19.6x.xx)
3676Stamina/1.4Stamina download manager
3677Star DownloaderStar Downloader download manager
3678StarDownloader/1.xxStar Downloader download manager
3679stat statcrawler@gmail.comExperimental search engine spider from
3680Steeler/1.x ( crawler
3681Steeler/3.3 ( - University of Tokyo web crawler
3682Strategic Board Bot (+ Board blog & news search robot
3683Strategic Board Bot (+ Board blogs and news aggregator robot
3684Submission Spider at surfsafely.comSurfsafely submission verifier (Germany) search robot (CrawlerAgent v0.103) (Germany) search robot
3687suchbotSuchbot Germany robot Austria robot
3689suchpadbot/1.0 (+ search Germany robot (213.239.194.xx)
3690Sunrise XP/2.xSunrise XP handheld news / website reader and converter
3691Sunrise/0.42g (Windows XP)Sunrise XP web sites and newsfeeds converter and handheld reader
3692SuperBot/x.x (Win32)SuperBot website copier
3693SuperBot/x.x.x.xx (Windows XP)SuperBot website copier
3694Superdownloads SpidermanUbbi Superdownloads (Brazil) link checking
3695SURF SurfControl Web Filtering
3696SurferF3 1/0Wanadoo Rechereche robot
3697SurfMasterMaskbit Surfmaster bookmark tool
3698SurveyBot/2.2 Whois SourceWhois Source domain name information robot (66.249.26.xx)
3699SurveyBot/2.3 (Whois Source)Whois Source domain name information robot (66.249.26.xx)
3700suzuranYokogao Search Engine robot (Kanazawa University)
3701SWB/V1.4 (HP)HP Secure Web Browser for OpenVMS
3702swbot/0.9c libwww/5.3.1unknown
3703Swooglebot/2.0. (+ Swoogle's semantic web crawler
3704SWSBot-Images/1.2 - SmartWareSoft (85.186.255.xx) software search engine created for
3705SygolBot http://www.sygol.netSygol Search (Italy) robot
3706Sylera/1.2.xSylera browser (Japan) user robot
3708SyncIT/x.xSyncIT link validation
3709Syndirella/0.91preSyndirella desktop information aggregator (beta)
3710SynoBotSynomia (France) robot
3711Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawlerSyntryx Solution Suite - domain / keyword crawler (216.7.179.xx)
3712Szukacz/ (Polish search) robot
3713Szukacz/1.x (robot;; (Polish search) robot
3714T-Online BrowserGerman T-Online browser & internet suite (+ directory index
3716Tagword ( DMOZ survey - ODP link checking robot
3717Tagyu Agent/1.0Tagyu - bookmark collection online tag generator
3718Talkro Web-Shot/1.0 (E-mail: Home: Talkro IR robot
3719TAMU_CS_IRL_CRAWLER/1.0Texas A&M University - Dept. of Computer Science crawler (server or link checking ?) user link validation ? botTarget Your News - user submitted links
3722TCDBOT/Nutch-0.8 (PhD student research;; mcgettrs at t c d dot IE)Trinity College Dublin (Ireland) TCDBOT
3723TEHTTP header for transfer encoding used as user agent name ?
3724TeamSoft WinInet ComponentWinInet Internet client app.
3725TECOMAC-Crawler/0.xTecomac Gmbh (Germany) crawler software - now Arexera Information Technologies
3726Tecomi Bot ( (Germany) beta / test robot (
3727Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.;; crawler for NetSeer search (beta) via Amazon Web Services - see also NetSeer/Nutch
3728Teleport Pro/1.2x(.1xxx)Teleport (website) downloading tool
3729Teoma MPTeoma crawler (65.214.36.xx[x])
3730teomaagent crawler-admin@teoma.comTeoma crawler (65.214.36.xx[x])
3731teomaagent1 []Teoma crawler (65.214.36.xx[x])
3732teoma_agent1Teoma crawler (65.214.36.xx[x])
3733Teradex Mapper;; http://www.teradex.comTeradex Directory robot
3734TeragramCrawlerTeragram multilingual text & data processing software
3735terraminds-bot/1.0 ( blog search (Germany)
3736TerrawizBot/1.0 (+ Indian Search Engine robot
3737Test spiderNoceans Information Portfolio Manager (66.35.69.x)
3738TestCrawler/Nutch-0.9 (Testing Crawler for Research ;; tgautier at balihoo dot com)Balihoo - Search Engine for Advertising Media
3739The Expert HTML Source Viewer ( HTML online source viewer
3740TheRarestParser/0.2a ( Rarest Words - Linguistic experiment crawler via Amazon Web Services
3741TheSuBot/0.1 ( robot (Germany) for an unknown theme based search engine
3742thumbshots-de-Bot (Version: 1.02, powered by website thumbnail service (Germany) robot
3743thumbshots-de-Bot (Version: 1.02- powered by (Germany) robot
3744timboBot/0.9 Blogs timbo bot blog robot
3745 web page monitoring tool
3746TinEye/1.1 ( crawler for an open image search project
3747tivraSpider/1.0 ( spider from AT&T Labs Research
3748TJG/SpiderTjgroup spider
3749TJvMultiHttpGrabber ComponentTJvHttpGrabber (JEDI Visual Component Library)
3750Tkensaku/x.x( Search (Japan) robot from (
3751toCrawl/UrlDispatcherUnknown robot from 195.68.98.xx (
3752Topodia/1.2-dev (Topodia - Crawler for HTTP content indexing;; search engine and personal information assistant (in development)
3753TOPOS robot/1.1 ( search (Russia) robot
3754traazibot/testengine (+! search (Germany) robot
3755Trailfire-bot/0.7.1 (Nutch;; web collection and annotating system
3756Trailfire-bot/0.7.1 (Trailfire page content analyzer;; web collection and annotating system
3757Trailfire/0.7.1 (Nutch;; web collection and annotating system
3758Trampelpfad-SpiderTrampelpfad Webkatalog spider
3759Trampelpfad-Spider-v0.1Trampelpfad Webkatalog spider
3760tricosMetaCheck 1.2216-08-1999 ( meta tag validation
3761TSurf15a 11some bad user agent
3762TulipChain/5.x ( Java/1.x.1_0x ( Linux/2.4.17Tulip Chain browser / link checker for directory
3763TulipChain/5.xx ( Java/1.x.1_0x ( Mac_OS_X/10.2.8Tulip Chain browser / link checker for directory
3764Tumblr/1.0 RSS syndication (+ ( Tumblelogs RSS and news syndication crawler
3765TurnitinBot/x.x ( (ex SlySearch) robot for helping educational institutions prevent plagiarism
3766Turnpike Emporium LinkChecker/0.1TurnPike Emporium Directory (207.67.198.x) link checking
3767TutorGig/1.5 (+ tutorial search robot
3768Tutorial Crawler 1.4 ( tutorial search robot
3769Twiceler experimental web crawler
3770Twiceler-0.9 experimental web crawler
3771Twisted PageGetterFile downloading component from Twisted Python
3772Twitturly / v0.xTwitt(url)y URL tracking service for Twitter via Amazon Web Services
3773Twotrees Reactive Filter V2.0Twotrees content filter
3774Tycoon Agent/Nutch-1.0-devTycoon - Hewlett-Packards distributed cluster solution robot
3775TygoBotTygo Search robot
3776TygoProwlerTygo Search robot
3777UCmoreUCMore - IE navigation and search plugin
3778UCMore Crawler AppUCMore - IE navigation and search plugin
3779UCWEB5.1Ucweb mobile browser
3780UDMuser agent - maybe UdmSearch (see UdmSearch) ?
3781UdmSearch/3.1.xUdmSearch / MySearch (now mnoGoSeach) offline browser/search client
3782UIowaCrawler/1.0University of Iowa Crawler- possibly MySpiders
3783UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; search robot
3784UltraseekInfoseek robot
3785Under the Rainbow 2.2Unknown mail harvester/spambot from ( Web Server IIS vs Apache Survey. See Results at www.DNSRight.comDNS Right - Online DNS tools
3787UofTDB_experiment ( robot from University of Toronto (
3788UP.Browser/3.01-IG01 UP.Link/ phone browser
3789updated/0.1-alpha (updated crawler;;! search robot
3790updated/0.1beta (;;! search robot
3791UPG1 UP/4.0 (compatible; Blazer 1.0)Handspring (PalmOS powered cellphone) Treo Blazer browser online link popularity check
3793UptimeBot( online link popularity check
3794URI::Fetch/0.06URI::Fetch - client for fetching HTTP pages and syndication feeds (RSS Atom)
3795URL Spider Pro/x.xx ( URL Spider Pro (now ES.NET) web indexing / site searching tool
3796URLBase/6.xURLBase - Internet shortcut manager
3797URLBlazeURLBlaze file sharing link toolkit
3798urlfan-bot/1.0; + news crawler
3799URLGetFileURLGetFile downloading tool
3800URL_Spider_Pro/x.xInnerprise URL Spider Pro (now ES.NET) web indexing / site searching tool
3801URL_Spider_Pro/x.x+( URL Spider Pro (now ES.NET) web indexing / site searching tool
3802User-Agent: BoardReader Favicon Fetcher /1.0 info@boardreader.comBoardReader search favicon fetcher
3803User-Agent: BoardReader Image Fetcher /1.0 info@boardreader.comBoardReader search image fetcher
3804User-Agent: LjSEEK Picture-Bot /1.0 - LiveJournal picture feed search
3805User-Agent: FileHeap! file downloader ( download manager
3806User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)Malformed UA header from some guestbook/forum spammer
3807User-Agent: Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - search robot
3808user-agent=Mozilla/3.01Goldunknown robot (reads robots.txt) or sitegrabber. From different IPs- ie.: 62.98.8.xx (
3809USyd-NLP-Spider ( of Sydney NLP Spider for research in Natural Language Processing
3810UtilMind HTTPGetWeb Thief Site Grabber
3811Utopia WebWasher 3.0WebWasher ad filter
3812uTorrent/1500uTorrent BitTorrent client
3813VadixBotUnknown bad behaving bot via Road Runner - see link
3814Vagabondo-WAP/2.0 (webcrawler at wise-guys dot nl; ProfileWiseGuys WAP pages robot
3815Vagabondo/1.x MT ( robot Netherland -
3816Vagabondo/2.0 MTWiseGuys robot Netherland -
3817Vagabondo/2.0 MT (webagent at wise-guys dot nl)WiseGuys robot Netherland -
3818Vagabondo/2.0 MT ( robot Netherland -
3819Vagabondo/3.0 (webagent at wise-guys dot nl)WiseGuys robot Netherland -
3820Vakes/0.01 (Vakes;; Directory link checking from Vakes
3821VayalaCreep-v0.0.1 ( ( robot
3822Vayala|Creep-v0.0.1 ( ( robot
3823vb wininetiNet Grabber - Internet content grabber
3824versus 0.2 (+ Project robot - Comparing methods for near-uniform URL sampling
3825versus crawler eda.baykan@epfl.chUnknown robot from EPFL University Switzerland (
3826VerticrawlbotVerticrawl - Semantic search engine solution (French) link submission checking - Networking4all Bot/x.xVerzamelgids NL link checking robot
3829Verzamelgids/2.2 ( NL link checking robot
3830Vespa CrawlerUnknown robot from Yahoo Norway
3831virus_detector ( G2 anti-virus and anti-spyware protection
3832VisBot/2.0 ( Crawler;; crawler for a search software under development
3833Visicom ToolbarSome IE toolbar made with Visicom Media Dynamic Toolbar software
3834Vision Research Lab image spider at vision.ece.ucsb.eduVision research lab's Cortina - content based image retrieval (128.111.60.xx)
3835VLC media player - version 0.8.5 Janus - (c) 1996-2006 the VideoLAN teamVLC - Cross-platform media player and streaming server
3836VMBot/0.x.x (VMBot;; - Vertical Search Engine (China)
3837Vortex/2.2 (+ Web Indexing Robot for a study on internet link distribution
3838voyager-hc/1.0Kosmix health, auto and travel search crawler (204.14.48.x /
3839voyager/1.0Kosmix health, auto and travel search crawler (204.14.48.x /
3840voyager/2.0 ( health, auto and travel search crawler (204.14.48.x /
3841VSE/1.0 ( search crawler (
3842VSE/1.0 ( search crawler (
3843vspiderVerity vspider indexing software
3844vspider/3.xVerity vspider indexing software
3845VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler;;+vwbot@cs.uiuc.eduVWBot - MetaQuerier Crawler for the MetaQuerier project at the University of Illinois
3846W3C-checklink/3.x.x.x libwww-perl/5.xxW3C Link Checker
3847W3C-checklink/4.x [4.xx] libwww-perl/5.xxxW3C Link Checker
3848W3C-WebCon/5.x.x libwww/5.x.xWebCon - the Libwww command line tool
3849W3CLineMode/5.4.0 libwww/5.x.xW3C Line Mode (character based Web browser)
3850W3CRobot/5.4.0 libwww/5.4.0Unknown link checking using Libwww via Korea Telecom (
3851W3C_Validator/ libwww-perl/5.xxW3C HTML-Code Validator
3852w3m/0.x.xxw3m Linux pager / text-based browser
3853W3SiteSearch Crawler_v1.1 http://www.w3sitesearch.deW3 Site Search (Germany) search engine solution 0.2 ( (Japan) Blog / RSS search crawler
3855WannaBe (Macintosh; PPC)Wanna-Be text mode browser
3856WapOnWindows 1.0WapOnWindows WAP browser for PCs
3857Watchfire WebXM 1.0Watchfire WebXM intranet solution
3858WAVcheck 1.0.x ( - Simple Vendor Discovery Tool for detecting client-side tags from web analytics vendors
3859Wavefire/0.8-dev (Wavefire;; local search community engine (
3860Waypath development crawler - info at waypath dot comWaypath blog discovery engine robot
3861Waypath Scout v2.x - info at waypath dot comWaypath blog discovery engine robot
3862WDG_Validator/1.1WDG HTML-code validator
3863Web Image's Web Image Collector (graphics downloading tool)
3864Web Link Validator 1.5Relsoft link checking software
3865Web SnooperRankMeter ranking software
3866web-bekannt (Version: 1.02, powered by German web directory link checking
3867web-bekannt (Version: 1.02, powered by German web directory link checking
3868Web-Bot V1.03Unkown link or server checking from Würzburg University Germany (132.187.10.xx)
3869Web-Robot/5.0 (en-US; Web-Robot Crawler/2.0.3Unknown robot from 69.50.233.x ( (leveled playing field;; info at / Web2x - Open data searching tool
3871WebAlta Crawler/1.2.1 ( search Russia crawler (85.21.201.xx)
3872WebarooBot (Webaroo Bot; / RufusBot from webaroo offline search service
3873WebarooBot (Webaroo Bot; / RufusBot from webaroo offline search service
3874WebAuto/3.4xxx (WinNT; I)Yanasoft WebAuto website copier / downloading tool
3875webbandit/4.xx.0Web Bandit personal search software
3876WebBug/5.xAmansoft WebBug web server protocol test - online news monitoring service
3878webcollage/1.xxWebCollage Syndicator graphics crawler/collector
3879WebCompass 2.0Quarterdecks WebCompass search tool
3880WebCopier vx.xWebCopier offline browser
3881WebCopier vx.xaWebCopier offline browser
3882WebCorp/1.0WebCorp linguistic search engine (UK)
3883webcrawl.netWebcrawl Search robot (
3884WebDownloader for X x.xxUnix/Linux Web Downloader
3885Webdup/0.9Unknown robot from
3886WebFetchWingFlyer WebFetch website downloading tool
3887webfetch/5.x.xwebfetch - command line tool to fetch files via HTTP
3888WebFilter Robot 1.0Verso NetSpective WebFilter
3889WebFilter Robot NetSpective WebFilter
3890WebFindBot( search robot
3891Webglimpse 2.xx.x ( search engine software
3892webGobbler/1.x.xwebGobbler - Online random image generator
3893webhackfake ?
3894WebImages 0.3 ( ) Web Images collage generator
3895WebLight/4.x.x (; web analyzer & link checker
3896Weblink's checker/WebLink's link management system for HTTP- FTP and Mail hyperlinks
3897Weblog Attitude Diffusion 1.0Los Alamos National Laboratoy weblog research project
3898webmeasurement-bot, http://rvs.informatik.uni-leipzig.deUnknown robot from Leipzig University (Germany) faculty for computer science
3899WebMiner/x.x [en] (Win98; I)WebMiner bulk file downloader
3900WeBoX/0.xxWeBoX (Japan) - Browser and web collector
3901WebPix 1.0 ( - picture downloading tool
3902WebQLCaesius WebQL - Custom robot/agent generator / web extraction software
3903WebRACE/1.1 (University of Cyprus- Distributed Crawler)WebRACE - HTTP retrieval- annotation and caching engine
3904WebRankSpider/1.37 (+ experimental web crawler
3905WebReaper vx.x - www.webreaper.netWebreaper download manager
3906WebReaper []Webreaper download manager
3907WebReaper []Webreaper download manager
3908WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; Search@WebSearch.COM.AU)Websearch Australia robot
3909WebSearchBench WebCrawler v0.1(Experimental)Dortmund University WebSearchBench - Open source search software
3910WebSearchBench WebCrawler V1.0 (Beta)- Prof. Dr.-Ing. Christoph Lindemann- Universität Dortmund- University WebSearchBench - Open source search software
3911Website Explorer/0.9.x.xWeb site downloading tool and offline browser (Japan)
3912Website eXtractorWebsite eXtractor web site downloading tool
3913WebsiteWorth v1.0Sootle web directory Website Worth ranking tool
3914Webspinne/1.0 robot (Add url robot) Search Engine robot / link checking
3916WebStat/1.0 (Unix; beta; 20040314)WebStat - Java statistical computing environment for the web
3917Webster v0.3 ( )Webster - Rev. Healeys web crawler pad browserWebsters Webmaster Archive (Germany) submission / pad checking
3919WebStripper/2.xxWebStripper download manager
3920WebTrafficExpress/x.0WebTrafficExpress IBM server software
3921WebTrends/3.0 (WinNT)Web Trends link analyzer
3922WebVac ( Stanford WebBase Project crawler
3923WebVal/1.0webval - Python link checking tool - Telefon: 01908 / (Germany) directory robot
3925WebVulnCrawl.unknown/1.0 libwww-perl/5.803Web Vulnerability Crawler
3926WebWatcherMonitor/2.01Studio Net.Idea's Web Watcher Monitor robot
3927WebZIP/x.x ( offline browser
3928Wells Search IIUnknown spam bot / harvester (62.163.**.** / 62.194.**.*)
3929WEP Search 00Some spam bot- see link
3930West Wind Internet Protocols 4.xxwwIPStuff - Internet client tools for Visual FoxPro
3931WFARCIBM's Almaden Research robot (Clever search project)
3932Wget/1.x(.x)GNU wget - file downloaderGNU wget - file downloader
3933Wget/1.x+cvs-stable (Red Hat modified)GNU wget - file downloader
3934Wget/1.x.x+cvsGNU wget - file downloader
3935Whatsup/x.xWhatsup Gold network monitor
3936whatUseek_winona/3.0WhatUSeek / Chubba robot
3937WhizBang! LabWhizBang! Labs (closed since May 2002) information extraction robot
3938Wildsoft Surfersome download agent
3939Willow Internet Crawler by Twotrees V2.1Twotrees crawler
3940WinampMPEG/2.00 (larbin@unspecified.mail)unknown robot from (209.126.176.x)
3941WincerSong Agent v1.0Super Affiliate Tracker agent by Wincer Song
3942Windows-Media-Player/10.00.00.xxxxWindows Media Player 10
3943WinGet 1.1Nicksoft WinGet download manager
3944WinHTTP Example/1.0Example code for a WinHTTP C++ library crawler
3945WinkBot/0.06 ( search engine web crawler;; beta search robot (64.13.136.x)
3946WinPodder ( - Podcast player and RSS reader
3947WinWAP/3.x (3.x.x.xx; Win32) (Google WAP Proxy/1.0)WinWap - Windows PC WAP browser
3948Wir sind die Borg (Version: 1.03, Sie wurden Assimiliert + web directory (Germany) link checking
3949WIRE/0.11 (Linux; i686; Bot,Robot,Spider,Crawler, crawler used by the University of Pisa - Italy
3950WIRE/0.x (Linux; i686; Bot,Robot,Spider,Crawler)WIRE - Web information retrieval environment crawler
3951WISEbot/1.0 (; Wisenut robot
3952WiseWire-Spider2Wisewire domain checker (Discontinued)
3953wish-project ( academic research project for link checking
3954WordChampBotWordchamp web page vocabulary / translation robot
3955WordPress/x.x.x.x PHP/4.x.xxWordPress personal Blog publishing platform
3956worio heritrix bot (+ (beta) search for computer scientists and programmers using Heritrix open-source crawler
3957woriobot ( (beta) search for computer scientists and programmers via Amazon Web Services
3958WorldLightEntireweb Search robot (
3959WorQmada/1.0unknown link checking (from ?
3960Wotbox/alpha0.6 (; spider
3961Wotbox/alpha0.x.x (; Java/1.4.1_02Wotbox spider
3962WSB WebCrawler V1.0 (Beta)- cl@cs.uni-dortmund.deWebSearchBench crawler from Dortmund University- Germany
3963WSB, http://websearchbench.cs.uni-dortmund.deWebSearchBench crawler from Dortmund University- Germany
3964wume_crawler/1.1 ( Lab's web crawler (
3965Wusage/x.0@boutell.comWusage log-file analysis
3966Wwlib/LinuxWWLib - Wolverhampton Univerity Web Library for classifying web documents
3967WWSBOT 1.x [--- ---]WWSBOT web server version checker
3968WWW-Mechanize/1.1xPerl web page fetching module
3969www.arianna.itArianna robot registry verify/ link checking ? crawlerThe DoWeb UK Business directory link checking
3972www4mail/2.x libwww-FM/2.14 (Unix; I)www4mail - web navigation & database search by e-mail
3973WWWC/1.0xWWWC Updating check of Web pages. (Japanese only)
3974WWWeasel Robot v1.00 ( Wide Weasel Germany robot
3975WWWOFFLE/2.xWWWoffle download manager
3976wwwster/1.x (Beta- robot from CIS at Munich University
3977wxDownload FastwxDownload Fast (wxDFast) open source download manager
3978X-Crawler Arexera (Germany) crawler software
3979Xaldon WebSpiderXaldon WebSpider offline browser
3980Xenu Link Sleuth 1.xxXenu link checker
3981Xenu's Link Sleuth 1.x[a-z]Xenu link checker
3982Xerka WebBot v1.0.0 [UPVOpenDir]XerKa text mining and information retrieval software
3983xine/1.0xine - free Linux / OS/2 multimedia player
3984xirq/0.1-beta (xirq;; search (beta) robot (
3985XMLSlurp/0.1 libwww-perl/5.805GPath / XMLSlurp - Expression language for tree structured data
3986XRL/2.00b1 (Linux; i686; en-us) (+ URL Shorten Service
3987XylixXylix Retrieval System software
3988xyro_( Crawler
3989Y!J-BSC/1.0 ( Search Japan robot (
3990Y!J-SRD/1.0Yahoo Search Japan robot (
3991Y!J/1.0 ( Search Japan robot (
3992Y!OASIS/TEST no-ad Mozilla/4.08 [en] (X11; I; FreeBSD 2.2.8-STABLE i386)Yahoo picture service for mobiles
3993Y!TunnelProY!TunnelPro - Yahoo! Messenger companion user agent
3994yacy (; v20040602; i386 Linux 2.4.26-gentoo-r13; java 1.4.2_06; MET/en)Yacy distributed P2P web search engine robot
3995yacybot (x86 Windows XP 5.1; java 1.5.0_06; Europe/de) yacy.netYacy distributed P2P web search engine robot
3996Yahoo Pipes 1.0(Yahoo) Pipes interactive data aggregator robot
3997Yahoo! MindsetYahoo Mindset: Intent-driven Search (
3998Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; )Yahoo blog indexing robot (
3999Yahoo-MMAudVid/1.0 (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com)Yahoo multimedia crawler (206.190.43.xx)
4000Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash ;Mozilla 4.0 compatible; MSIE 7.0;Windows NT 5.0; .NET CLR 2.0)Yahoo multimedia crawler
4001Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com)Yahoo multimedia crawler via (66.77.73.xx)
4002Yahoo-Test/4.0Yahoo Search robot
4003Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; crawler via Overture (
4004YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; Publisher Network RSS crawler
4005YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; Product Search crawler ( 68.142.195..x)
4006YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; Product Search crawler ( 66.196.93.x)
4007YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; Product Search crawler ( 66.196.93.x)
4008YahooSeeker/1.1 (compatible; Mozilla 4.0; MSIE 5.5; Product Search crawler ( 66.196.93.x)
4009YahooSeeker/bsv3.9 (compatible; Mozilla 4.0; MSIE 5.5; )Yahoo Product Search crawler ( 68.142.195..x)
4010YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ; )Yahoo robot
4011Yandex/1.01.001 (compatible; Win16; I)Yandex Search Russia link checking (
4012Yanga WorldSearch Bot v1.1/beta ( search robot by Gigabase (Russian Federation) search (Belgium) robot search Korea robot (
4015Yeti/0.01 (nhn/1noon,, check robots.txt daily and follows it) search Korea robot (
4016Yeti/1.0 (NHN Corp.; search (Korea) robot
4017yggdrasil/Nutch-0.9 (yggdrasil biorelated search engine; www dot biotec dot tu minus dresden do de slash schroeder; heiko dot dietze at biotec dot tu minus dresden dot de)yggdrasil spider for GoPubMed biorelated search engine
4018YodaoBot/1.0 (; )Yodao search (China)
4019yoofind/yoofind-0.1-dev (yoono webcrawler; ; MyEmail)Yoono - community based search ( / 194.0.179.[x]xx)
4020yoogliFetchAgent/0.1Yoogli search (under development) agent
4021yoono/1.0 web-crawler/1.0Yoono - community based search ( / 194.0.179.[x]xx)
4022YottaCars_Bot/4.12 (+ Car Search EngineYottaCars bot - YottaCar car search engine (
4023YottaShopping_Bot/4.12 (+ Shopping Search EngineYottaShopping bot - YottaShopping search engine (
4024YTunnelProY!TunnelPro - Yahoo! Messenger companion user agent
4025Z-Add Link Checker ( online link checker
4026Zao-CrawlerZao crawler for Kototoi Project
4027Zao-Crawler 0.2bZao crawler for Kototoi Project
4028Zao/0.1 ( crawler for Kototoi Project
4029ZBot/1.00 ( Internet Marketing Robot based on Webster Pro component
4030ZearchitZearchit German search / directory ( Moteur de Recherche robot ( Moteur de Recherche robot
4033zedzo.digest/0.1 ( Search (beta) robot
4034Zend_Http_ClientZend PHP frameworks Zend_Http_Client component
4035zermelo Mozilla/5.0 compatible; heritrix/1.12.1 (+ [,]Powerset Natural Language Search crawler (under development) using Heritrix via Amazon Web Services
4036zerxbot/Version 0.6 libwww-perl/5.79Zerx search robot ?
4037Zeus ThemeSite Viewer Webster Pro V2.9 Win32Zeus Internet Marketing Robot (based on Webster Pro)
4038Zeus xxxxx Webster Pro V2.9 Win32Zeus Internet Marketing Robot (based on Webster Pro)
4039Zeusbot/0.07 (Ulysseek's web-crawling robot;; robot for building the index
4040Ziggy -- The Clown From Hell!!Unknown agent (server- or link checking ?) from 198.173.158.xx
4041ZipppBot/0.xx (ZipppBot;; web search robot
4042ZIPPPCVS/0.xx (ZipppBot/.xx;; web search robot
4043Zippy v2.0 - Zippyfinder.comZippyfinder robot
4044Zoo Tycoon 2 Client -- http://www.zootycoon.comMicrosoft Zoo Tycoon 2 game client
4045ZoomSpider - wrensoft.comZoom Search Engine software spider Crawler/ ( Crawler collecting for machine learning; )
4047zspider/0.9-dev robot for a new search engine
4048ZyBorg/1.0 (; robot
4049Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv: Gecko/20100401 Firefox/3.6.3 (FM Scene 4.6.1)FM Scene soccer fan pages plugin
4050Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv: Gecko/20100401 Firefox/3.6.3 (.NET CLR 3.5.30729) (Prevx 3.0.5)PrevX security plugin
Sabalico Logo
Sabali Mail Logo
Sabali Analytics Logo
SEO Guide Logo
Keyword Tool Logo
Domain Search Logo
What is my IP Logo
My Device Logo
My Browser Logo
My Location Logo
My Weather Logo
Test Speed Logo
Website On Logo
Page Load Logo
Code Editor Logo
Format Code Logo
Encode File Logo
Generator Password Logo
QR Code Generator Logo
Barcode Generator Logo
Online Sign Logo
Counter Word Logo
Lorem Ipsum Generator Logo
Sprite Sheet Logo
Resize Image Logo
Image Compress Logo
Image Color Logo
Color Picker Logo
Color Convert Logo
rStatistics Logo
Free Data Logo
Unit Convert Logo
Byte Convert Logo
Coordinate Converter Logo
Temperature Convert Logo
Text Convert Logo
Currency Convert Logo
Phone Country Logo
To-Do List Logo
Finance Calculator Logo

Warning: file_get_contents( failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in /home/sabalico/public_html/addons/domains/ on line 164

Warning: file_get_contents( failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in /home/sabalico/public_html/addons/domains/ on line 165

Warning: file_get_contents( failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in /home/sabalico/public_html/addons/domains/ on line 166

Warning: file_get_contents( failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in /home/sabalico/public_html/addons/domains/ on line 167

Warning: file_get_contents( failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in /home/sabalico/public_html/addons/domains/ on line 168

Warning: file_get_contents( failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in /home/sabalico/public_html/addons/domains/ on line 169

Warning: file_get_contents( failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in /home/sabalico/public_html/addons/domains/ on line 170