Robots

Results: 4049

IDRobotUser Agent
2(^| )sentry\/sentry/8.22.0 (https://sentry.io)
3[wW]getWget/1.14 (linux-gnu)
4[wW]getWGETbot/1.0 (+http://wget.alanreed.org)
5[wW]getWget/1.20.3 (linux-gnu)
6^Apache-HttpClientApache-HttpClient/4.2.3 (java 1.5)
7^Apache-HttpClientApache-HttpClient/4.3.3 (java 1.5)
8^Apache-HttpClientApache-HttpClient/4.5.3 (Java/1.8.0_121)
9^Apache-HttpClientApache-HttpClient/4.5.2 (Java/1.8.0_181)
10^Apache-HttpClientApache-HttpClient/4.3.1 (java 1.5)
11^Apache-HttpClientApache-HttpClient/4.3.5 (java 1.5)
12^Apache-HttpClientApache-HttpClient/4.4.1 (Java/1.8.0_65)
13^Apache-HttpClientApache-HttpClient/4.5.2 (Java/1.8.0_65)
14^Apache-HttpClientApache-HttpClient/4.5.2 (Java/1.8.0_151)
15^Apache-HttpClientApache-HttpClient/4.5.2 (Java/1.8.0_161)
16^Apache-HttpClientApache-HttpClient/4.2.5 (java 1.5)
17^Apache-HttpClientApache-HttpClient/4.5.10 (Java/1.8.0_201)
18^Apache-HttpClientApache-HttpClient/4.5.7 (Java/11.0.3)
19^Apache-HttpClientApache-HttpClient/4.5.3-SNAPSHOT (Java/1.8.0_152)
20^curlcurl/7.29.0
21^curlcurl/7.55.1
22^curlcurl/7.54.0
23^curlcurl/7.47.0
24^curlcurl
25^curlcurl/7.65.3
26^curlcurl/7.64.1
27^curlcurl/7.64.0
28^LCCLCC (+http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
29007ac9 CrawlerMozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)
30123spider.de (Germany) web directory link checking123spider-Bot (Version: 1.02, powered by www.123spider.de
31192.com - UK web directory192.comAgent
321st ZipCommander Net - IE based browser1st ZipCommander (Net) - http://www.zipcommander.com/
3321seek.com (China) robot (218.17.90.xxx)AIBOT/2.1 By +(www.21seek.com A Real artificial intelligence search engine China)
342Bone online link checker2Bone_LinkChecker/1.0 libwww-perl/5.64
352ip.ru2ip.ru CMS Detector (https://2ip.ru/cms/)
36360SpiderMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider
37361SpiderMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
38362SpiderMozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 QIHU 360SE; 360Spider
39363SpiderMozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; )Firefox/1.5.0.11; 360Spider
40364SpiderMozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11)Firefox/1.5.0.11; 360Spider
41365SpiderMozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider;
42366SpiderMozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
43367SpiderMozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider
44368SpiderMozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
45369SpiderMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 360Spider
464Anything robot4anything.com LinkChecker v2.0
4780LegsMozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620
48A Passion for Jazz music related search robotBebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html )
49A-Online.at robot - now Jet2Web SearchA-Online Search
50A1 Keyword Research - search engine and keyword optimization softwareA1 Keyword Research/1.0.2 (+http://www.micro-sys.dk/products/keyword-research/) miggibot/2007.03.27
51A6-IndexerA6-Indexer
52Aardvark web crawler for Sun's Blog recommendationsaardvark-crawler
53Abacho / Crawler.de robotAbachoBOT
54Abacho / Crawler.de robotAbachoBOT (Mozilla compatible)
55ABCdatos - Castilian program & tutorial directoryABCdatos BotLink/5.xx.xxx#BBL
56Aberja Hybridsuchmaschine (Germany) link checkingAberja Checkomat
57AboundexAboundex/0.3 (http://www.aboundex.com/crawler/)
58AboundexAboundex/0.2 (http://www.aboundex.com/crawler/)
59About robotAbout/0.1libwww-perl/5.47
60acapbotMozilla/5.0 (compatible;acapbot/0.1.;treat like Googlebot)
61acapbotMozilla/5.0 (compatible;acapbot/0.1;treat like Googlebot)
62Accela Technology RSS feed crawlerAccelatech RSSCrawler/0.4
63Accoona Search robotaccoona
64Accoona Search robotAccoona-AI-Agent/1.1.1 (crawler at accoona dot com)
65Accoona Search robotAccoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)
66Ace Explorer - IE based browserAce Explorer
67Ackerm search robotAck (http://www.ackerm.com/)
68Acoi picture finder robotAcoiRobot
69Acoon.de search (Germany) robotAcoon Robot v1.50.001
70Acoon.de search (Germany) robotAcoon Robot v1.52 (http://www.acoon.de)
71Acoon.de search (Germany) robotAcoon-Robot 4.0.x.[xx] (http://www.acoon.de)
72Acoon.de search (Germany) robotAcoon-Robot v3.xx (http://www.acoon.de and http://www.acoon.com)
73Acorn Search ProjectAcorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot org)
74Activeworlds 3D homepage browserActiveworlds
75Activeworlds 3D homepage browserActiveWorlds/3.xx (xxx)
76Ad Muncher - banner killerAd Muncher v4.xx.x
77Ad Muncher - banner killerAd Muncher v4x Build xxxxx
78adbeat_botMozilla/5.0 (compatible; adbeat_bot; +support@adbeat.com; support@adbeat.com)
79AddSearchBotMozilla/5.0 (compatible; AddSearchBot/0.9; +http://www.addsearch.com/bot; info@addsearch.com)
80AddThisAddThis.com robot tech.support@clearspring.com
81ADmantXADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - support@admantx.com
82AdsBot GoogleMozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
83AdsBot-Google-MobileMozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
84AdsBot-Google-MobileMozilla/5.0 (Linux; Android 5.0; SM-G920A) AppleWebKit (KHTML, like Gecko) Chrome Mobile Safari (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
85AdsBot-Google-MobileAdsBot-Google-Mobile-Apps
86AdsBot-Google([^-]|$)AdsBot-Google (+http://www.google.com/adsbot.html)
87adscannerMozilla/5.0 (compatible; adscanner/)
88AdsTxtCrawlerAdsTxtCrawler/1.0
89AdvBotMozilla/5.0 (compatible; AdvBot/2.0; +http://advbot.net/bot.html)
90Aesop robotAESOP_com_SpiderMan
91Agada search (Germany) robotagadine/1.x.x (+http://www.agada.de)
92AHC\/AHC/2.0
93Ahrefs(Bot|SiteAudit)Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/)
94Ahrefs(Bot|SiteAudit)Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
95Ahrefs(Bot|SiteAudit)Mozilla/5.0 (compatible; AhrefsBot/6.1; News; +http://ahrefs.com/robot/)
96Ahrefs(Bot|SiteAudit)Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)
97Ahrefs(Bot|SiteAudit)Mozilla/5.0 (compatible; AhrefsBot/5.2; News; +http://ahrefs.com/robot/)
98Ahrefs(Bot|SiteAudit)Mozilla/5.0 (compatible; AhrefsSiteAudit/5.2; +http://ahrefs.com/robot/)
99AideRss - Postrank RSS and Blog filteringAideRSS/1.0 (aiderss.com)
100aiHitBotMozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about)
101Akregator news feed reader for KDEAkregator/1.2.9; librss/remnants
102Aladin robotAladin/3.324
103Aleksika Danmark - Search engine optimization spiderAleksika Spider/1.0 (+http://www.aleksika.com/)
104Alertinfo - French version of Feedreader 3.xxAlertInfo 2.0 (Powered by Newsbrain)
105Alexaia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
106Alexa crawlercrawler@alexa.com
107Allesklar.de robotAllesklar/0.1 libwww-perl/5.46
108Alligator download managerAlligator 1.31 (www.nearsoftware.com)
109AlphaBotMozilla/5.0 (compatible; AlphaBot/3.2; +http://alphaseobot.com/bot.html)
110Altavista robotAltaVista Intranet V2.0 AVS EVAL search@freeit.com
111Altavista robotAltaVista Intranet V2.0 Compaq Altavista Eval sveand@altavista.net
112Altavista robotAltaVista Intranet V2.0 evreka.com crawler@evreka.com
113Altavista robotAltaVista V2.0B crawler@evreka.com
114Altavista robotAVSearch-3.0(AltaVista/AVC)
115Altavista robot ??AV Fetch 1.0
116Amaya - W3C's Editor/Browseramaya/x.xx libwww/x.x.x
117Amazon CloudFrontAmazon CloudFront
118Amazon.com robot for checking their affiliate sitesamzn_assoc
119Amazon.com robot for checking their affiliate sitesaranhabot
120Amfibi Search robotAmfibiBOT
121Amfibi Search robotAmfibibot/0.06 (Amfibi Web Search; http://www.amfibi.com; agent@amfibi.com)
122Amfibi Search robotAmfibibot/0.07 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)
123Amfibi Search robotCabot/Nutch-0.9 (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; agent@amfibi.com)
124Amfibi Search robotCabot/Nutch-1.0-dev (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; agent@amfibi.com)
125amibot - Amidalla search engine robot (62.241.33.xx)amibot
126Amiga Miami TCP StackAmiTCP Miami (AmigaOS 2.04)
127AndersPinkBotMozilla/5.0 (compatible; AndersPinkBot/1.0; +http://anderspink.com/bot.html)
128Annomille Italian historical oriented robotAnnoMille spider 0.1 alpha - http://www.annomille.it
129annotate Google - Firefox extension for annotating Google search resultsannotate_google; http://ponderer.org/download/annotate_google.user.js
130AnswerBus natural language search using COLLATE technologyAnswerBus (http://www.answerbus.com/)
131AnswerChase search toolAnswerChase PROve x.0
132AnswerChase search toolAnswerChase x.0
133ANT Fresco BrowserANTFresco/x.xx
134Antibot (discontinued) robotantibot-V1.1.5/i586-linux-2.2
135AnyApexBot 1.0Mozilla/5.0 (compatible; AnyApexBot/1.0; +http://www.anyapex.com/bot.html)
136AnyEventMozilla/5.0 (compatible; U; AnyEvent-HTTP/2.24; +http://software.schmorp.de/pkg/AnyEvent)
137Anzwers (Yahoo) Australia robotAnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au;Engine)
138AOLMozilla/5.0 (compatible; MSIE 9.0; AOL 9.7; AOLBuild 4343.19; Windows NT 6.1; WOW64; Trident/5.0; FunWebProducts)
139AOLMozilla/4.0 (compatible; MSIE 8.0; AOL 9.7; AOLBuild 4343.19; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E)
140AOLMozilla/4.0 (compatible; MSIE 7.0; AOL 9.7; AOLBuild 4343.19; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E)
141AOLMozilla/4.0 (compatible; MSIE 8.0; AOL 9.7; AOLBuild 4343.19; Windows NT 5.1; Trident/4.0; GTB7.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
142AOLMozilla/4.0 (compatible; MSIE 8.0; AOL 9.7; AOLBuild 4343.21; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E)
143AOLMozilla/4.0 (compatible; MSIE 8.0; AOL 9.7; AOLBuild 4343.27; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
144AperciteMozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)
145Apexoo Search spiderApexoo Spider 1.x
146APIs-GoogleAPIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)
147Aport robotAport
148AppEngine-GoogleAppEngine-Google; (+http://code.google.com/appengine; appid: example)
149AppInsightsMozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; AppInsights)
150ApplebotMozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; +http://www.apple.com/go/applebot)
151ApplebotMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
152ApplebotMozilla/5.0 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
153ApplebotMozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
154ApplebotMozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
155Araby search - ArabiaArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby.com;)
156archive.org_botMozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
157archive.org_botMozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)
158archive.org_botMozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
159archive.org_botMozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 +http://archive.org/details/archive.org_bot)
160archive.org_botMozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.archive.org)
161archive.org_botMozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot)
162ArchiveBotArchiveTeam ArchiveBot/20170106.02 (wpull 2.0.2)
163Arikus inContext search engine softwareArikus_Spider
164Art of Computing blog search projectblogsearchbot-pumpkin-3
165Asaha search robot (Turkey)ASAHA Search Engine Turkey V.001 (http://www.asaha.com/)
166ASAHINA Antenna information detecting agentAsahina-Antenna/1.x
167ASAHINA Antenna information detecting agentAsahina-Antenna/1.x (libhina.pl/x.x ; libtime.pl/x.x)
168Ask 24x Info robotask.24x.info
169Ask About Oil - Petroleum related search (24.227.212.xxx) using NutchAskAboutOil/0.06-rcp (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@askaboutoil.com)
170askEd! / Inferret search (Japan) robot using Nutchasked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com)
171ASPSeek search engine softwareASPSeek/1.2.x
172ASPSeek search engine softwareASPSeek/1.2.xa
173ASPSeek search engine softwareASPseek/1.2.xx
174ASPSeek search engine softwareASPSeek/1.2.xxpre
175ASPSeek search engine software -Yahoo-Inc. / Telecom Canada robotASPSeek/1.2.5
176Associative Sort robotASSORT/0.10
177AT&T/Fast Search robot for FirstGov (U.S.Government) portalFirstGov.gov Search - POC:firstgov.webmasters@gsa.gov
178Atlocal local business search robotAtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html)
179Atomic Email Hunter email extracing and harvestingAtomic_Email_Hunter/4.0
180Atomz robotAtomz/1.0
181atSpider (ceased) email harvester / spambotatSpider/1.0
182Attentio social media monitoring and analysingAttentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; info@attentio.com)
183Audiovox SMT5600 (AT&T) Smartphone mobile phone browserAUDIOVOX-SMT5600
184Augurnet Swiss (was www.augurnet.ch) search robotaugurfind
185Augurnet Swiss (was www.augurnet.ch) search robotaugurnfind V-1.x
186Auto Email Pro Email harvesterautoemailspider
187Autowebdir - The Automatically Generated Web Directoryautowebdir 1.1 (www.autowebdir.com)
188Avant Browser - IE based browserAdvanced Browser (http://www.avantbrowser.com)
189Avant Browser - IE based browserAvant Browser (http://www.avantbrowser.com)
190AWeb Amiga browserAmiga-AWeb/3.4.167SE
191AWeb Amiga browserAWeb
192awesomecrawlerMozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.5 Safari/537.22 +awesomecrawler
193Axada search Germany robotaxadine/ (Axadine Crawler; http://www.axada.de/; )
194axiosaxios/0.18.0
195Axmo search robotAxmoRobot - Crawling your site for better indexing on www.axmo.com search engine.
196Ay-Up geo sync search robotFastBug http://www.ay-up.com
197Ayell Euronet business directory robotEurobot/1.0 (http://www.ayell.eu)
198Azureus Java BitTorrent ClientAzureus 2.x.x.x
199B2B BotB2B Bot
200Babaloo search robot (Slovenia)BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)
201BaBoom Web Portal (ODP) robot (66.98.254.xx)BaboomBot/1.x.x (+http://www.baboom.us)
202BackStreet Browser - Offline browser / website downloaderBackStreet Browser 3.x
203Baidu search (Japan) crawlerBaiduspider+(+http://www.baidu.com/search/spider_jp.html)
204Baidu search (Japan) image crawlerBaiduImagespider+(+http://www.baidu.jp/search/s308.html)
205Baidu spidering engine - used by diff. IPsBaiDuSpider
206Baidu spidering engine - used by diff. IPsBaiduspider+(+http://help.baidu.jp/system/05.html)
207Baidu spidering engine - used by diff. IPsBaiduspider+(+http://www.baidu.com/search/spider.htm)
208Baidu-YunGuanCeBaidu-YunGuanCe-VSBot(ce.baidu.com)
209Baidu-YunGuanCeBaidu-YunGuanCe-Bot(ce.baidu.com)
210Baidu-YunGuanCeBaidu-YunGuanCe-PerfBot(ce.baidu.com)
211Baidu-YunGuanCeBaidu-YunGuanCe-ScanBot(ce.baidu.com)
212Baidu-YunGuanCeBaidu-YunGuanCe-SLABot(ce.baidu.com)
213BaiduspiderBaiduspider+(+http://www.baidu.com/search/spider.htm)
214BaiduspiderMozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)
215BaiduspiderBaiduspider+(+http://www.baidu.com/search/spider_jp.html)
216BaiduspiderMozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
217Balihoo - Search Engine for Advertising MediaBloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com)
218Balihoo - Vertical search engine crawler (beta)Balihoo/Nutch-1.0-dev (Crawler for Balihoo.com search engine - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com)
219Barca Pro email & PIM softwareBarca/2.0.xxxx
220Barca Pro email & PIM softwareBarcaPro/1.4.xxxx
221Bark[rR]owlerBarkrowler/0.5.1 (experimenting / debugging - sorry for your logs ) http://www.exensa.com/crawl - admin@exensa.com -- based on BuBiNG
222Barrahome crawlerBarraHomeCrawler (albertof@barrahome.org)
223Basic Web Hacking 3 fake user-agent from Hellbound Hackers challengesbwh3_user_agent
224BazQuxMozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; 1 subscribers)
225BB4 network monitoringBigBrother/1.6e
226BD-Brandprotect copyright infringement crawlerBPImageWalker/2.0 (www.bdbrandprotect.com)
227BDCbotMozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
228Bdncentral Sitesearch robotBDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html) (X11; I; Linux 2.0.44 i686)
229beammachine web directory (Germany) link checkingBeamMachine/0.5 (dead link remover of www.beammachine.net)
230Beauty robot for Cosmoty - German beauty and wellness searchbeautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/)
231BehloolBotMozilla/5.0 (compatible; BehloolBot/beta; +http://www.webeaver.com/bot)
232Big Brother link checking toolBig Brother (http://pauillac.inria.fr/~fpottier/)
233BigClique Search robotBigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com; bot@bigclique.com)
234BIGLOTRONBIGLOTRON (Beta 2;GNU/Linux)
235Biglotron search (France) robotBIGLOTRON (Beta 2;GNU/Linux)
236Bigsearch.ca search robotBigsearch.ca/Nutch-x.x-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com)
237Bilbo - web frontend for the Nessus Security ScannerBilbo/2.3b-UNIX
238Bilgi.com (Beta) search robot - TurkeyBilgiBetaBot/0.8-dev (bilgi.com (Beta) ; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
239Bilgi.com (Beta) search robot - TurkeyBilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com)
240BingbotMozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
241BingbotMozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
242bingbotMozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)
243bingbotMozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
244bingbotMozilla/5.0 (seoanalyzer; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
245bingbotMozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
246bingbotMozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm
247bingbotMozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)
248BingbotMozilla/5.0 (compatible; bingbot/2.0 http://www.bing.com/bingbot.htm)
249bingbotMozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
250bingbotMozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
251BingbotMozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko BingPreview/1.0b
252bingbotMozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm)
253bingbotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Safari/537.36
254Bingbotmsnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
255bingbotMozilla/5.0 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm)
256bingbotMozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
257BingbotMozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
258BingbotMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
259BingbotMozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
260Bingbotmsnbot/2.0b (+http://search.msn.com/msnbot.htm)
261bingbotMozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) SitemapProbe
262BingbotMozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
263BingbotMozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
264BingbotMozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)
265BingPreview\/Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
266BingPreview\\/Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;WOW64;Trident/5.0;BingPreview/1.0b)
267BingPreview\\/Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko
268BingPreview\\/Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0;WOW64;Trident/6.0;BingPreview/1.0b)
269BingPreview\\/Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b
270BingPreview\\/Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
271binlarbinlar_2.6.3 binlar2.6.3@unspecified.mail
272binlarbinlar_2.6.3 binlar_2.6.3@unspecified.mail
273binlarbinlar_2.6.3 larbin2.6.3@unspecified.mail
274binlarbinlar_2.6.3 phanendra_kalapala@McAfee.com
275binlarbinlar_2.6.3 test@mgmt.mic
276Bisnisseek (was www.bisnisseek.com) robotCustom Spider www.bisnisseek.com /1.0
277Bitacle Blog Search Archive robotBitacle bot/1.1
278Bitacle Blog Search Archive robotBitacle Robot (V:1.0;) (http://www.bitacle.com)
279bitlybotbitlybot
280bitlybotbitlybot/2.0
281bitlybotbitlybot/3.0 (+http://bit.ly/)
282BitTorrent Search Engine btbot robotBTbot/0.x (+http://www.btbot.com/btbot.html)
283Biyubi Navigator - Mexican browser for Fenix OSBiyubi/x.x (Sistema Fenix; G11; Familia Toledo; es-mx)
284Blackberry Wireless Internet browser via Google WAP ProxyBlackBerry7520/4.0.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/5.0.3.3 UP.Link/5.1.2.12 (Google WAP Proxy/1.0)
285BlackboardBlackboard Safeassign
286BlackWidow web site scanner / downloading toolBlackWidow
287Blaiz Enterprises RawGrunt searchBlaiz-Bee/1.0 (+http://www.blaiz.net)
288Blaiz Enterprises RawGrunt searchBlaiz-Bee/2.00.8222 (BE Internet Search Engine http://www.rawgrunt.com)
289Blaiz Enterprises RawGrunt searchBlaiz-Bee/2.00.xxxx (+http://www.blaiz.net)
290BLEXBot\/Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
291BLEXBot\\/Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
292Blitzsuche Germany robotBlitzBOT@tricus.net
293Blitzsuche Germany robotBlitzBOT@tricus.net (Mozilla compatible)
294BlockNote web page editorBlockNote.Net
295BlogBridge RSS readerBlogBridge 2.13 (http://www.blogbridge.com/)
296blogdex robot from MIT.eduBlogBot/1.x
297Bloglines article searchBloglines Title Fetch/1.0 (http://www.bloglines.com)
298Bloglines graphics crawlerBloglines-Images/0.1 (http://www.bloglines.com)
299Bloglines news crawlerBloglines/3.1 (http://www.bloglines.com)
300blogmuraBotblogmuraBot (+http://www.blogmura.com)
301BlogsNow realtime link tracker robotBlogsNowBot, V 2.01 (+http://www.blogsnow.com/)
302BlogVipe news and Blog crawler (Netherlands)BlogVibeBot-v1.1 (spider@blogvibe.nl)
303blogWatcher robot from Okumura Group Tokyo (131.112.182.xxx)blogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWatcher/)
304Blooby search (beta) robotBloobyBot
305BLP_bbotBLP_bbot/0.1
306Bluefish HTML-editor for Linuxbluefish 0.6 HTML editor
307bnf.fr_botMozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/bnf.html)
308Bobby web accessibility desktop testing toolBobby/4.0.x RPT-HTTPClient/0.3-3E
309Boitho search (Norway) robot via 80.202.212.xx / 80.80.111.xxboitho.com-dc/0.xx (http://www.boitho.com/dcbot.html)
310Boitho search (Norway) robot via 80.202.212.xx / 80.80.111.xxboitho.com-robot/1.x
311Boitho search (Norway) robot via 80.202.212.xx / 80.80.111.xxboitho.com-robot/1.x (http://www.boitho.com/bot.html)
312BomboraBotMozilla/5.0 (compatible; BomboraBot/1.0; +http://www.bombora.com/bot)
313Bookdog - Mac bookmark managerBookdog/x.x
314Bookmark Base bookmark managerBookmarkBase(2/;http://bookmarkbase.com)
315Bookmark Buddy - favorite bookmark managerBookmark Buddy bookmark checker (http://www.bookmarkbuddy.net/)
316Bookmark Express bookmark managerBMLAUNCHER
317bot-pge.chlooe.combot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)
318Bot.AraTurka.comBot.AraTurka.com/0.0.1
319botifyMozilla/5.0 (compatible; botify; http://botify.com)
320BoxcarBotMozilla/5.0 (compatible; BoxcarBot/1.1; +awesome@boxcar.io)
321Brandimensions Brand Protection robotBDFetch
322BrandVerityMozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:41.0) Gecko/20100101 Firefox/55.0 BrandVerity/1.0 (http://www.brandverity.com/why-is-brandverity-visiting-me)
323BravoBrian bSTOP parental controlBravoBrian bstop.bravobrian.it
324BravoBrian bSTOP parental controlBStop.BravoBrian.it Agent Detector
325BrightCloud web filtering for classifying websitesBrightCrawler (http://www.brightcloud.com/brightcrawler.asp)
326BTWebClientBTWebClient/180B(9704)
327BUbiNGBUbiNG (+http://law.di.unimi.it/BUbiNG.html)
328Buck\/Buck/2.2; (+https://app.hypefactors.com/media-monitoring/about.html)
329BuildCMS crawler - market monitoring project of BuildCMSBuildCMS crawler (http://www.buildcms.com/crawler)
330Bulkfeeds: RSS directory link checkingBulkfeeds/r1752 (http://bulkfeeds.net/)
331BullsEye/Intelliseek robotBullsEye
332BurstFind search crawler (64.34.172.xx)BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawler@burstfind.com)
333Buscaplus (Spain) robotBuscaplus Robi/1.0 (http://www.buscaplus.com/robi/)
334Business.com robotbdcindexer_2.6.2 (research@bdc)
335BuzzbotBuzzbot/1.0 (Buzzbot; http://www.buzzstream.com; buzzbot@buzzstream.com)
336buzzbotBuzzbot/1.0 (Buzzbot; http://www.buzzstream.com; buzzbot@buzzstream.com)
337BytespiderMozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.4141.1682 Mobile Safari/537.36; Bytespider
338BytespiderMozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2988.1545 Mobile Safari/537.36; Bytespider
339BytespiderMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.5527.1507 Mobile Safari/537.36; Bytespider
340BytespiderMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.6023.1635 Mobile Safari/537.36; Bytespider
341BytespiderMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.4944.1981 Mobile Safari/537.36; Bytespider
342BytespiderMozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.4454.1745 Mobile Safari/537.36; Bytespider
343BytespiderMozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.7990.1979 Mobile Safari/537.36; Bytespider
344BytespiderMozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.3478.1649 Mobile Safari/537.36; Bytespider
345BytespiderMozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.5267.1259 Mobile Safari/537.36; Bytespider
346BytespiderMozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.7597.1164 Mobile Safari/537.36; Bytespider;bytespider@bytedance.com
347BytespiderMozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.2268.1523 Mobile Safari/537.36; Bytespider
348BytespiderMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2576.1836 Mobile Safari/537.36; Bytespider
349BytespiderMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.9681.1227 Mobile Safari/537.36; Bytespider
350BytespiderMozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.3754.1902 Mobile Safari/537.36; Bytespider
351BytespiderMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.3248.1547 Mobile Safari/537.36; Bytespider
352BytespiderMozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.9038.1080 Mobile Safari/537.36; Bytespider
353BytespiderMozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.5216.1326 Mobile Safari/537.36; Bytespider
354BytespiderMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.4022.1033 Mobile Safari/537.36; Bytespider
355BytespiderMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.3613.1739 Mobile Safari/537.36; Bytespider
356CaliperbotCaliperbot/1.0 (+http://www.conductor.com/caliperbot)
357Campulka.net Favorites checking toolFavorites Checking (http://campulka.net)
358CapsuleCheckerCapsuleChecker (http://www.capsulink.com/)
359Carnegie Mellon School robot/link checking ?billbot wjj@cs.cmu.edu
360Carnegie Mellon University WebBOTCarnegie_Mellon_University_Research_WebBOT-->PLEASE READ-->http://www.andrew.cmu.edu/~brgordon/webbot/index.html http://www.andrew.cmu.edu/~brgordon/webbot/index.html
361Carnegie Mellon University WebBOTCarnegie_Mellon_University_WebCrawler http://www.andrew.cmu.edu/~brgordon/webbot/index.html
362Catall.de search & web directory (Germany)Catall Spider
363CC Metadata ScaperCC Metadata Scaper http://wiki.creativecommons.org/Metadata_Scraper
364CCBotCCBot/2.0 (http://commoncrawl.org/faq/)
365CCBotCCBot/2.0 (https://commoncrawl.org/faq/)
366CERN Line Mode BrowserCERN-LineMode/2.15
367cg-eye CGI checkercg-eye interactive
368changedetectionMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )
369changedetectionMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;http://www.changedetection.com/bot.html )
370Charon Amiga download managerCharon/1.x (Amiga)
371Chat Catcher blog monitoring robotChat Catcher/1.0
372check_httpcheck_http/v2.2.1 (nagios-plugins 2.2.1)
373Checkbot link validationCheckbot/1.xx LWP/5.xx
374Checklinks - Perl link checkerCheckLinks/1.x.x
375CheckMarkNetwork\/CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)
376CheckWeb link validationCheckWeb
377Chilkat HTTP component user-agentChilkat/1.0.0 (+http://www.chilkatsoft.com/ChilkatHttpUA.asp)
378Chitika Inc. Blog advertisingChitika ContentHit 1.0
379Christcentral.com Christcrawler (was www.christcrawler.com)ChristCRAWLER 2.0
380Chrome-LighthouseMozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3694.0 Mobile Safari/537.36 Chrome-Lighthouse
381Chrome-LighthouseMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3694.0 Safari/537.36 Chrome-Lighthouse
382Chrome-LighthouseMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36(KHTML, like Gecko) Chrome/69.0.3464.0 Safari/537.36 Chrome-Lighthouse
383Chrome-LighthouseMozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36(KHTML, like Gecko) Chrome/69.0.3464.0 Mobile Safari/537.36 Chrome-Lighthouse
384CHttpClient - C++ class using WinInetCHttpClient by Open Text Corporation
385CipinetBot -Cipinet Search Engine Web CrawlerCipinetBot (http://www.cipinet.com/bot.html)
386Cisco Content EngineCE-Preload
387Cityreview regional search (Germany) link checkingCityreview Robot (+http://www.cityreview.org/crawler/)
388CJB Net anonymous socks proxy service (216.194.70.x)CJB.NET Proxy
389Claria (ex Gator) SearchScout robot (64.152.73.xx)ClariaBot/1.0
390Claria (ex Gator) SearchScout robot (64.152.73.xx)Diamond/x.0
391Claria (ex Gator) SearchScout robot (64.152.73.xx)DiamondBot
392Claymont Search robotClaymont.com
393ClickagyClickagy Intelligence Bot v2
394Cliqzbot\/Mozilla/5.0 (compatible; Cliqzbot/2.0; +http://cliqz.com/company/cliqzbot)
395Cliqzbot\\/Mozilla/5.0 (compatible; Cliqzbot/2.0; +http://cliqz.com/company/cliqzbot)
396Cliqzbot\\/Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)
397Cliqzbot\\/Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)
398Cliqzbot\\/Cliqzbot/0.1 (+http://cliqz.com +cliqzbot@cliqz.com)
399Cliqzbot\\/Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)
400CloudFlare-AlwaysOnlineMozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34
401Clush clustered search robotClushbot/2.x (+http://www.clush.com/bot.html)
402Clush clustered search robotClushbot/3.x-BinaryFury (+http://www.clush.com/bot.html)
403Clush clustered search robotClushbot/3.xx-Ajax (+http://www.clush.com/bot.html)
404Clush clustered search robotClushbot/3.xx-Hector (+http://www.clush.com/bot.html)
405Clush clustered search robotClushbot/3.xx-Peleus (+http://www.clush.com/bot.html)
406COAST Webmaster - Web management and maintenance softwareCOAST WebMaster Pro/4.x.x.xx (Windows NT)
407coccocMozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)
408coccocMozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)
409coccocMozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/searchengine)
410coccocMozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/searchengine)
411coccocMozilla/5.0 (compatible; image.coccoc/1.0; +http://help.coccoc.com/)
412coccocMozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/)
413coccocMozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)
414coccoccoccoc/1.0 (http://help.coccoc.vn/)
415coccoccoccoc/1.0 (http://help.coccoc.com/)
416coccoccoccoc/1.0 ()
417Cocoa del.icio.us (social bookmarks manager) client for Mac OS XCocoal.icio.us/1.0 (v36) (Mac OS X; http://www.scifihifi.com/cocoalicious)
418Cogent Search Bot from Cogent Software Solutions for unknown purposesCogentbot/1.X (+http://www.cogentsoftwaresolutions.com/bot.html)
419Cold Fusion server used by Bookmark Tracker - online favourites managmentColdFusion (BookmarkTracker.com)
420Cold Fusion server used by various IPs i.e.: - NetWORLD web catalogue link checkingColdFusion
421collection@infegy.comMozilla/5.0 (compatible) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36 collection@infegy.com
422Combine harvesting & indexing robotcombine/0.0
423Combine harvesting & indexing robotCombine/2.0 http://combine.it.lth.se/
424Combine harvesting & indexing robotCombine/3 http://combine.it.lth.se/
425Combine harvesting & indexing robotCombine/x.0
426cometrics Web Content Mining solution - Germanycometrics-bot, http://www.cometrics.de
427Cometsystems Comet Search robot via Findwhat (now Miva)Crawler (cometsearch@cometsystems.com)
428Commision Junction link checking spiderCJ Spider/
429CommonCrawl Foundation search crawlerCCBot/1.0 (+http://www.commoncrawl.org/bot.html)
430Companybook-CrawlerCompanybook-Crawler (+https://www.companybooknetworking.com/)
431Comperio Web Miner based on Fast ESPFAST Enterprise Crawler 6 used by Comperio AS (sts@comperio.no)
432ComRite Chinese Search Engine for Oversea Web Sites (69.248.26.xx)Comrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
433ContentSmartz e-mail harvesting toolsContentSmartz
434ContextAd BotMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;.NET CLR 1.0.3705; ContextAd Bot 1.0)
435contxbotMozilla/5.0 (compatible;contxbot/1.0)
436converaConveraCrawler/0.9e (+http://ews.converasearch.com/crawl.htm)
437Converas RetrievalWare Internet SpiderCrawlConvera0.1 (CrawlConvera@yahoo.com)
438Converas RetrievalWare Internet Spider (63.241.61.x)Convera Internet Spider V6.x
439Converas RetrievalWare Internet Spider (63.241.61.x)ConveraCrawler/0.2
440Converas RetrievalWare Internet Spider (63.241.61.x)ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)
441Converas RetrievalWare Internet Spider (63.241.61.x)ConveraMultiMediaCrawler/0.1 (+http://www.authoritativeweb.com/crawl)
442Cooliris photo and video browserCooliris/1.5 CFNetwork/459 Darwin/10.0.0d3
443Coral Content Distribution NetworkCoralWebPrx/0.1.1x (See http://coralcdn.org/)
444CoreMedia player on Apple iPhoneApple iPhone v1.1.4 CoreMedia v1.0.0.4A102
445Cosmix project crawler (204.14.48.x / 38.113.234.xxx)carleson/1.0
446Cosmix project crawler (204.14.48.x / 38.113.234.xxx)cfetch/1.0
447Coteo.com - local French directory link checkingCoteoNutchCrawler/Nutch-0.9 (info [at] coteo [dot] com)
448Cougarsearch.com robotCougarSearch/0.x (+http://www.cougarsearch.com/faq.shtml)
449Covac Arachnid Web CrawlerArachBot
450Covac Arachnid Web CrawlerCovac TexAs Arachbot
451CoverScout for iTunes - CD cover search toolCoverScout%203/3.0.1 CFNetwork/339.5 Darwin/9.5.0 (i386) (iMac5,1)
452Crawler.de / Abacho robotCrawler admin@crawler.de
453Crawler.de / Abacho robotCrawler V 0.2.x admin@crawler.de
454crawler4jcrawler4j (https://github.com/yasserg/crawler4j/)
455crawler4jcrawler4j (http://code.google.com/p/crawler4j/)
456Crawlly Beta search - Germany (72.232.194.2xx)Crawllybot/0.1 (Crawllybot; +http://www.crawlly.com; crawler@crawlly.com)
457Creatice Commons using Nutch open source robotCreativeCommons/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)
458Cricket A100 cell phone browserCricket-A100/1.0 UP.Browser/6.3.0.7 (GUI) MMP/2.0
459Croccrawler robotCrocCrawler vx.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)
460CrunchBotCrunchBot/1.0 (+http://www.leadcrunch.com/crunchbot)
461CSE HTML Validator for WindowsCSE HTML Validator Professional (http://www.htmlvalidator.com/)
462Cuam - IE based browserCuam Ver0.050bx
463Cuasar (Spain) music / ringtone search spiderCuasarbot/0.9b http://www.cuasar.com/spider_beta/
464Curl file transferring toolcurl/7.10.x (i386-redhat-linux-gnu) libcurl/7.10.x OpenSSL/0.9.7a ipv6 zlib/1.1.4
465Curl file transferring toolcurl/7.7.x (i386--freebsd4.3) libcurl 7.7.x (SSL 0.9.6) (ipv6 enabled)
466Curl file transferring toolcurl/7.8 (i686-pc-linux-gnu) libcurl 7.8 (OpenSSL 0.9.6)
467Curl file transferring toolcurl/7.9.x (win32) libcurl 7.9.x
468CurryGuide UK link check robotCurryGuide SiteScan 1.1
469Custo web site spidering tool (link checking)Custo x.x (www.netwu.com)
470cXensebotcXensebot/1.1a
471cybercity.fr user robot / faked user agent ?Bot mailto:craftbot@yahoo.com
472Cyberdog Mac Browser (was www.cyberdog.org)Cyberdog/2.0 (Macintosh; 68k)
473CyberPatrolCyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)
474CyberPatrol LLC robot for web filtering softwareCyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)
475CyberSpyder Link Test softwareCyberSpyder Link Test/2.1.12 (admin@mspennyworth.com)
476Cydral image & site search spiderCydralSpider/1.x (Cydral Web Image Search; http://www.cydral.com)
477Cydral image & site search spiderCydralSpider/3.0 (Cydral Image Search; http://www.cydral.com)
478DareBoostMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36 DareBoost
479DataBecker Bilder Sauger (discontinued) web graphics downloaderDer große BilderSauger 2.00u
480DatafeedwatchDatafeedwatch/2.1.x
481datagnionbotdatagnionbot (+http://www.datagnion.com/bot.html)
482DatanyzeMozilla/5.0 (X11; Datanyze; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
483DataparkSearch open source search engineDataparkSearch/4.47 (+http://dataparksearch.org/bot)
484DataparkSearch open source search engineDataparkSearch/4.xx (http://www.dataparksearch.org/)
485Dataprovider.comMozilla/5.0 (compatible; Dataprovider.com)
486DataSpear Directory robot (24.109.29.xx)DataSpear/1.0 (Spider; http://www.dataspear.com/spider.html; spider@dataspear.com)
487DataSpear Directory robot (24.109.29.xx)DataSpearSpiderBot/0.2 (DataSpear Spider Bot; http://dssb.dataspear.com/bot.html; dssb@dataspear.com)
488Daum\/Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
489DAUMOA - RSS search robot of DaumELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net/aboutkr.html)
490Daypop blog - weblog - online mag search spiderdaypopbot/0.x
491dbDig search enginedbDig(http://www.prairielandconsulting.com)
492DC-Sakura download managerDC-Sakura/x.xx
493dcrawldcrawl/1.0
494DeadLinkCheckerwww.deadlinkchecker.com XMLHTTP/1.0
495DeadLinkCheckerwww.deadlinkchecker.com Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
496deadlinkcheckerwww.deadlinkchecker.com Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
497DeadLinkCheckerwww.deadlinkchecker.com XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
498deepak-USC/ISI robot from USC/Information Science Institutedeepak-USC/ISI
499Deepindex robotDeepIndex
500Deepindex robotDeepIndex (www.en.deepindex.com)
501Deepindex robot (via Paragon.net Canada)DeepIndexer.ca
502del.icio.us picture robot for thumbnail preview via Yahoodel.icio.us-thumbnails/1.0 Mozilla/5.0 (compatible; Konqueror/3.4; FreeBSD) KHTML/3.4.2 (like Gecko)
503DeleGate application level gateway / proxy serverDeleGate/9.0.5-fix1
504Denmex Websearch robot/link checkingDenmex websearch (http://search.denmex.com)
505DeuSu\/Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.org)
506DeuSu\\/Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.org)
507DeuSu\\/Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
508DevComponents HTMLDocument Class Library for Visual Studio.netDevComponents.com HtmlDocument Object
509DiaGem Japan web crawlerDiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html)
510Die Kraehe Meta-Search-Engine (Germany) link checking-DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.de
511Diffbot\/Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
512Diffbot\\/Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
513Digg DeeperDigg Deeper/v1 (http://digg.com/about)
514Diggit! robotDigger/1.0 JDK/1.3.0rc3
515Digincore botMozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.)
516Dillo Web BrowserDillo/0.8.5-i18n-misc
517Dillo Web BrowserDillo/0.x.x
518disastrous - a del.icio.us link checker based on Pythondisastrous/1.0.5 (running with Python 2.5.1; http://www.bortzmeyer.org/disastrous.html; archangel77@del.icio.us)
519DISCo Pump offline browser / website ripperDISCo Pump x.x
520discobotMozilla/5.0 (compatible; discobot/2.0; +http://discoveryengine.com/discobot.html)
521discobotMozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)
522discobotmozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
523DiscordbotMozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)
524DisqusDisqus/1.0
525Ditto picture search robotDittoSpyder
526DivX.com Movie Find robotFlickBot 2.0 RPT-HTTPClient/0.3-3
527DLC Perl HTTP link checkingDeadLinkCheck/0.4.0 libwww-perl/5.xx
528DNS Right - Online DNS toolsDNSRight.com WebBot Link Ckeck Tool. Report abuse to: dnsr@dnsright.com
529DNS Right - Online DNS toolsunknownght.com Web Server IIS vs Apache Survey. See Results at www.DNSRight.com
530DnyzBotMozilla/5.0 (compatible; DnyzBot/1.0)
531DocZilla - Mozilla-based SGML/XML/HTML- browserDocZilla/1.0 (Windows; U; WinNT4.0; en-US; rv:1.0.0) Gecko/20020804
532Domain Re-Animator BotDomain Re-Animator Bot (http://domainreanimator.com) - support@domainreanimator.com
533domaincrawlerCipaCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/www.example.com)
534DomainStatsBotDomainStatsBot/1.0 (http://domainstats.io/our-bot)
535Donut P - Japanese IE based browserDonutP; Windows98SE
536dotbotMozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
537dotbotdotbot
538Download Accelerator Plus download managerDAP x.x
539Download Demon/Netzip download managerDownload Demon/3.x.x.x
540Download Direct download managerDownloadDirect.1.0
541Download Druid IE plugin download managerDownload Druid 2.x
542Download Express download managerDownload Express 1.0
543Download Master download managerDownload Master
544Download Ninja download manager (Japan)Download Ninja 3.0
545Download Wonder download managerDownload Wonder
546Download-Tipp Germany link checkingDownload-Tipp Linkcheck (http://download-tipp.de/)
547Download-Tipp Germany robotEyeCatcher (Download-tipp.de)/1.0
548download.exe .NET based downloading toolDownload.exe(1.1) (+http://www.sql-und-xml.de/freeware-tools/)
549Downloadaccelerator download managerDA 3.5 (www.lidan.com)
550Downloadaccelerator download managerDA 4.0
551Downloadaccelerator download managerDA 4.0 (www.downloadaccelerator.com)
552Downloadaccelerator download managerDA 5.0
553Downloadaccelerator download managerDA 7.0
554Dr.WEB online virus scannerDr.Web (R) online scanner: http://online.drweb.com/
555Dragonfly CMS - Open Source content management systemDragonfly File Reader
556DreamCast DreamPassport browser(DreamPassport/3.0; isao/MyDiGiRabi)
557Drecom Japan (210.233.67.xxx) - Blog search ??Drecombot/1.0 (http://career.drecom.jp/bot.html)
558Drew Curtis' FARK.com link checkingFARK.com link verifier
559drupactdrupact/0.7; http://www.arocom.de/drupact
560Drupal - open source content management platformDrupal (+http://drupal.org/)
561dt Search Spider softwaredtSearchSpider
562DTAAgent Java object for data collectingDTAAgent
563Duck Duck Go search crawlerDuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)
564DuckDuckBotDuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)
565DuckDuckBotMozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)
566DuckDuckBotDuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
567DuckDuckGo-Favicons-BotMozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
568DumbFind.com robotDumbot(version 0.1 beta - dumbfind.com)
569DumbFind.com robotDumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html)
570DumbFind.com robotDumbot(version 0.1 beta)
571e-Society Project (Japan) crawler (133.9.238.xx)e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/)
572e-Style ISP search (Russia) robot (217.174.103.xxx)eStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0)
573Earthcom (Czech Republic) search robot (194.108.39.xx)EARTHCOM.info/1.x [www.earthcom.info]
574Earthcom (Czech Republic) search robot (194.108.39.xx)EARTHCOM.info/1.xbeta [www.earthcom.info]
575ec2linkfinderec2linkfinder
576eCatch (now Wysigot) offline browsereCatch/3.0
577Echo.fr robotEchO!/2.0
578Egoto Search robotEgotoBot/4.8 (+http://www.egoto.com/about.htm)
579eJupiter searcg robot (206.191.49.xx)ejupiter.com
580electricmonkMozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)
581Elftales crawler for uChoose theme based search (Germany)elfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/)
582ELinks text mode browserELinks (0.x.x; Linux 2.4.20 i586; 132x60)
583ELinks text mode browserELinks/0.x.x (textmode; NetBSD 1.6.2 sparc; 132x43)
584EmailSpider E-mail harvesting softwareEmailSpider
585EmbedlyEmbedly +support@embed.ly
586EmeraldShield spam and web filtration servicesEmeraldShield.com WebBot
587EmeraldShield spam and web filtration servicesEmeraldShield.com WebBot (http://www.emeraldshield.com/webbot.aspx)
588Empas search Korea robot (220.95.22x.xxx)EMPAS_ROBOT
589Empyreum Ccubee (Czech) search engine solutionccubee/x.x
590Enabot - Enaball semantic search project crawlerEnaBot/1.x (http://www.enaball.com/crawler.html)
591endo - Mac news site and blog aggregatorendo/1.0 (Mac OS X; ppc i386; http://kula.jp/endo)
592Enfish Personal search toolEnfish Tracker
593Enterprise Search engine software (64.202.165.xxx)Enterprise_Search/1.0
594Enterprise Search engine software (64.202.165.xxx)Enterprise_Search/1.0.xxx
595Enterprise Search engine software (64.202.165.xxx)Enterprise_Search/1.00.xxx;MSSQL (http://www.innerprise.net/es-spider.asp)
596Enterprise Search engine software (64.202.165.xxx)ES.NET_Crawler/2.0 (http://search.innerprise.net/)
597Envolk Web Search robotenvolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php)
598Envolk Web Search robotenvolk[ITS]spider/1.6(+http://www.envolk.com/envolkspider.html)
599epicbotMozilla/5.0 (compatible; epicbot; +http://www.epictions.com/epicbot)
600erightMozilla/5.0 (compatible; eright/1.0; +bot@eright.com)
601EroCrawler adult search robotEroCrawler
602ESI Smart-Spider toolkitESISmartSpider
603Eurip.com - European Internet Portal robotEuripBot/0.x (+http://www.eurip.com) GetFile
604Eurip.com - European Internet Portal robotEuripBot/0.x (+http://www.eurip.com) GetRobots
605Eurip.com - European Internet Portal robotEuripBot/0.x (+http://www.eurip.com) PreCheck
606europarchive.orgMozilla/5.0 (compatible; MSIE 7.0 +http://www.europarchive.org)
607Euroseek spiderArachnoidea (arachnoidea@euroseek.com)
608Evaal Search Engine robotEvaalSE - bot@evaal.com
609Eventax event search (Germany)eventax/1.3 (eventax; http://www.eventax.de/; info@eventax.de)
610EveryoneSocialBotMozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)
611Evolution integrated mail solution Camel TCP stream classCamelHttpStream/1.0
612ExabotMozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
613ExabotMozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
614ExabotMozilla/5.0 (compatible; Exabot/3.0;http://www.exabot.com/go/robot)
615ExabotMozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
616ExabotMozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; certifyscan@alexa.com)
617ExabotMozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot)
618ExabotMozilla/5.0 (compatible; Exabot PyExalead/3.0; +http://www.exabot.com/go/robot)
619exabotMozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; certifyscan@alexa.com)
620eXact Search Bar for IEExactSearch
621ExactSEEK (Jayde Online) roboteseek-larbin_2.6.2 (crawler@exactseek.com)
622ExactSEEK (Jayde Online) robotExactSeek Crawler/0.1
623ExactSEEK (Jayde Online) robotexactseek-crawler-2.63 (crawler@exactseek.com)
624ExactSEEK (Jayde Online) robotexactseek-pagereaper-2.63 (crawler@exactseek.com)
625ExactSEEK (Jayde Online) robot (69.9.181.1xx)exactseek.com
626Exalead (France) search robot (193.47.80.xx)Exabot-Test/1.0
627Exalead (France) search robot (193.47.80.xx)Exabot/2.0
628Exalead (France) search robot (193.47.80.xx)Exabot/3.0
629Exalead (France) search robot (193.47.80.xx)Exalead NG/MimeLive Client (convert/http/0.120)
630Exalead Websearch image crawler (193.47.80.xx)Exabot-Images/1.0
631Excalibur (now Convera) spider softwareExcalibur Internet Spider V6.5.4
632Excite spiderArchitextSpider
633Execrawl software search using nutchExecrawl/1.0 (Execrawl; http://www.execrawl.com/; bot@execrawl.com)
634Exisoftware image grabber and downloading toolExtreme Picture Finder
635exooba crawler for exooba search pre-alpha developmentexooba crawler/exooba crawler (crawler for exooba.com; http://www.exooba.com/; info at exooba dot com)
636exooba crawler for exooba search pre-alpha developmentexooba/exooba crawler (exooba; exooba)
637ExperibotExperibot-v2 http://goo.gl/ZAr8wX
638Expired Domain Sleuth domain name toolExpired Domain Sleuth
639Express Web Pictures image browserExpress WebPictures (www.express-soft.com)
640ExtLinksBotMozilla/5.0 (compatible; ExtLinksBot/1.5 +https://extlinks.com/Bot.html)
641Extractor Pro e-mail collectorExtractorPro
642EZIDEZID (EZID link checker; https://ezid.cdlib.org/)
643ezoomsMozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)
644Facebookfacebookexternalhit/1.1
645Facebookfacebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
646Facebookfacebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
647facebookexternalhitfacebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
648FacebotFacebot/1.0
649Factbites search robotFactbot 1.09 (see http://www.factbites.com/webmasters.php)
650Factbites search robotfactbot : http://www.factbites.com/robots
651FaEdit Professional - Japanese bookmark managerFaEdit/2.0.x
652FairAd userFairAd Client
653faked user agentAnonymizer/1.1
654FAST Enterprise CrawlerFAST Enterprise Crawler 6 used by Schibsted (webcrawl@schibstedsok.no)
655FAST Enterprise CrawlerFAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com/srsapp/contactus/
656Fast Enterprise Crawler (66.151.181.xx) for Scirus scienctific information searchFAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com/srsapp/contactus/
657FAST Enterprise Crawler used by WWU (University of Muenster - Germany)FAST Enterprise Crawler 6 used by WWU (wardi@uni-muenster.de)
658FAST-WebCrawlerFAST-WebCrawler/3.8
659FAST-WebCrawlerFAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
660FAST-WebCrawlerFAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
661FAST-WebCrawlerFAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
662Fast/Alltheweb crawlerFAST-WebCrawler/2.2.6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
663Fast/Alltheweb crawlerFAST-WebCrawler/2.2.7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)http://www.fast.no
664Fast/Alltheweb crawlerFAST-WebCrawler/2.2.8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)http://www.fast.no
665Fast/Alltheweb crawlerFAST-WebCrawler/3.2 test
666Fast/Alltheweb crawlerFAST-WebCrawler/3.3 (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)
667Fast/Alltheweb crawlerFAST-WebCrawler/3.4/Nirvana (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)
668Fast/Alltheweb crawlerFAST-WebCrawler/3.4/PartnerSite (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)
669Fast/Alltheweb crawlerFAST-WebCrawler/3.5 (atw-crawler at fast dot no; http://fast.no/support.php?c=faqs/crawler)
670Fast/Alltheweb crawlerFAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
671Fast/Alltheweb crawlerFAST-WebCrawler/3.6/FirstPage (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)
672Fast/Alltheweb crawlerFAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
673Fast/Alltheweb crawlerFAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
674Fast/Alltheweb crawlerFAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
675Fast/Alltheweb crawlerFAST-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
676Fast/Alltheweb crawler (66.151.181.xx)Fast Crawler Gold Edition
677Fast/Alltheweb crawler (66.151.181.xx)FAST Enterprise Crawler 6 (Experimental)
678Fast/Alltheweb crawler (66.151.181.xx)FAST Enterprise Crawler 6 used by Cobra Development (admin@fastsearch.com)
679Fast/Alltheweb crawler (66.151.181.xx)FAST Enterprise Crawler 6 used by FAST (FAST)
680Fast/Alltheweb crawler (66.151.181.xx)FAST Enterprise Crawler/6 (www.fastsearch.com)
681Fast/Alltheweb crawler (66.151.181.xx)FAST Enterprise Crawler/6.4 (helpdesk at fast.no)
682Fast/Alltheweb crawler (66.151.181.xx)FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0)
683Fast/Alltheweb crawler (66.151.181.xx)FAST MetaWeb Crawler (helpdesk at fastsearch dot com)
684Fast/Alltheweb crawler (66.151.181.xx)Fast PartnerSite Crawler
685Fast/Alltheweb crawler (66.151.181.xx)FastCrawler 3.0.1 (crawler@1klik.dk)
686Fast/Alltheweb crawler for Sensis.com.au Australian search (66.151.181.xx)FAST Enterprise Crawler 6 used by Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au)
687Fast/Alltheweb crawler used by SuperPages.comFastSearch Web Crawler for Verizon SuperPages (kevin.watters@fastsearch.com)
688Fast/Alltheweb multimedia crawlerFAST-WebCrawler/2.2.10 (Multimedia Search) (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
689Fast/Alltheweb multimedia crawlerFAST-WebCrawler/3.x Multimedia
690Fast/Alltheweb multimedia crawlerFAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)
691Fastbot search Germany crawler (80.252.104.1xx)fastbot crawler beta 2.0 (+http://www.fastbot.de)
692Favcollector Favicon collecting robotFavcollector/2.0 (info@favcollector.com http://www.favcollector.com/)
693FavIconizer - IE favorites icons refreshing toolFavIconizer
694favo.eu (Germany) search robotfavo.eu crawler/0.6 (http://www.favo.eu)
695Favorites Sweeper bookmark checkerFavorites Sweeper v.2.03
696Favourites managing programBookmark Renewal Check Agent [http://www.bookmark.ne.jp/]
697Favourites managing programBookmark Renewal Check Agent [http://www.bookmark.ne.jp/] (Version 2.0beta)
698FaXo Search robot (69.152.89.xx)Faxobot/1.0
699fedoraplanetvenus/fedoraplanet
700Feed::Find - Syndication feed (RSS/Atom) auto-discoveryFeed::Find/0.0x
701Feed24 news feed and blog searchFeed24.com
702Feedable beta web based RSS serviceFeedable/0.1 (compatible; MSIE 6.0; Windows NT 5.1)
703FeedDemon RSS readerFeedDemon/2.7 (http://www.newsgator.com/; Microsoft Windows XP)
704Feedfetcher-GoogleFeedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=728742641706423)
705FeedForAll RSS feed robotFeedForAll rss2html.php v2
706FeedHub news feed personalization engine powered by mSpokeFeedHub FeedDiscovery/1.0 (http://www.feedhub.com)
707FeedHub news feed personalization engine powered by mSpokeFeedHub MetaDataFetcher/1.0 (http://www.feedhub.com)
708Feedjit news feed service favicon crawlerFeedjit Favicon Crawler 1.0
709FeedlyFeedlyBot/1.0 (http://feedly.com)
710FeedlyFeedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)
711FeedMap / BlogMap geo coding serviceBlogMap (http://www.feedmap.net)
712FeedShow online RSS feed readerFeedshow/x.0 (http://www.feedshow.com; 1 subscriber)
713FeedShow online RSS feed readerFeedshowOnline (http://www.feedshow.com)
714FeedspotMozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot)
715FeedspotbotMozilla/5.0 (compatible; Feedspot/1.0 (+https://www.feedspot.com/fs/fetcher; like FeedFetcher-Google)
716FeedspotbotMozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot)
717FeedValidatorFeedValidator/1.3
718FeedZcollector - Feed (RSS, ATOM and RDF) capturing softwareFeedZcollector v1.x (Platinum) http://www.feeds4all.com/feedzcollector
719FemtosearchBotMozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com)
720Fetch\/Fetch/2.0a (CMS Detection/Web/SEO analysis tool, see http://guess.scritch.org)
721Fetch\\/Fetch/2.0a (CMS Detection/Web/SEO analysis tool, see http://guess.scritch.org)
722FeverFever/1.38 (Feed Parser; http://feedafever.com; Allow like Gecko)
723Filangy search and bookmark serviceFilangy/0.01-beta (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)
724Filangy search and bookmark serviceFilangy/1.0x (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; filangy-agent@filangy.com)
725Filangy search and bookmark serviceFilangy/1.0x (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)
726File Boost Network link checkingfileboost.net/1.0 (+http://www.fileboost.net)
727FileHound download managerFileHound x.x
728filterdb.iss.net\/crawlerMozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)
729filtrbox media content (news) monitoringFiltrbox/1.0
730Find An ISP robotFindAnISP.com_ISP_Finder_v99a
731Find the Best search robotFTB-Bot http://www.findthebest.co.uk/
732findlinkfindlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
733findlinkfindlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)
734findlinkfindlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/)
735findlinkfindlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)
736findlinkfindlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
737findlinkfindlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/)
738findlinkfindlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
739findlinkfindlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
740findlinkfindlinks/2.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
741findlinkfindlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
742findlinkfindlinks/1.1.6-beta5 (+http://wortschatz.uni-leipzig.de/findlinks/)
743findlinkfindlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/; YaCy 0.1; yacy.net)
744findlinkfindlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
745findlinkfindlinks/1.1.6-beta3 (+http://wortschatz.uni-leipzig.de/findlinks/)
746findlinkfindlinks/1.1.6-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)
747findlinkfindlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
748findlinkfindlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
749findlinkfindlinks/2.1.3 (+http://wortschatz.uni-leipzig.de/findlinks/)
750findlinkfindlinks/2.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
751findlinkfindlinks/2.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
752findlinkfindlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
753findlinkfindlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
754findlinkfindlinks/2.6 (+http://wortschatz.uni-leipzig.de/findlinks/)
755Findsame.com (site is offline) / Digital-Integrity robotDIIbot/1.2
756findxbotMozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)
757Finesearch robotFineBot
758Finjan Vital Security Web Appliance security solutionFinjan-prefetch
759Fireball.de robotFirefly/1.0
760Fireball.de robotFirefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)
761Firstsfind Germany robot / link checkingfirstsbot
762Flamingo_SearchEngineFlamingo_SearchEngine (+http://www.flamingosearch.com/bot)
763Flaptor information retrieval solutions robotFlapbot/0.7.2 (Flaptor Crawler; http://www.flaptor.com; crawler at flaptor period com)
764FlatArts Favorites Icon ToolFLATARTS_FAVICO
765Flexum.ru search serviceFlexum spider
766Flexum.ru search serviceFlexum/2.0
767FlipboardProxyMozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
768FlipboardProxyMozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)
769FlipboardProxyMozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
770FlipboardProxyMozilla/5.0 (compatible; FlipboardProxy/1.2; +http://flipboard.com/browserproxy)
771FlipboardProxyMozilla/5.0 (compatible; FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
772Floortransformed.com robot (link ckecking ??)Ceramic Tile Installation Guide (http://www.floorstransformed.com)
773FLY Index Metasearch link checkingflyindex.net 1.0/http://www.flyindex.net
774Fnoole news crawlerFnooleBot/2.5.2 (+http://www.fnoole.com/addurl.html)
775folkd.com social search robotFolkd.com Spider/0.1 beta 1 (www.folkd.com)
776FollowSite robot - website monitoringFollowSite Bot ( http://www.followsite.com/bot.html )
777FollowSite robot - website monitoringFollowSite.com ( http://www.followsite.com/b.html )
778Fooky search Scorpionbots robot (65.12.170.xxx)Fooky.com/ScorpionBot/ScoutOut; http://www.fooky.com/scorpionbots
779Fourelle Venturi proxy serverDual Proxy
780fr-crawlerMozilla/5.0 (compatible; fr-crawler/1.1)
781Free Download Manager (FDM) download acceleratorFDM 1.x
782Free Download Manager (FDM) download acceleratorFDM 2.x
783Free Downloads shareware directory link checkingfree-downloads.net download-link validator /0.1
784FreeBSD download toolfetch libfetch/2.0
785FreeFind.com robotFreeFind.com-SiteSearchEngine/1.0 (http://freefind.com; spiderinfo@freefind.com)
786Frelics backlink checking bot (beta)Frelicbot/1.0 +http://www.frelic.com/
787Frequent Finders spider via Sitefusion.comFFC Trap Door Spider
788Fresh Download download managerFreshDownload/x.xx
789FreshNotes - music related artist search (72.3.225.xx)FreshNotes crawler< report problems to crawler-at-freshnotes-dot-com
790FriendicaFriendica 'The Tazmans Flax-lily' 2019.01-1293; https://hoyer.xyz
791FS Consulting (was www.fsconsult.net) Black Widow web crawlerBlackWidow
792fuelbotfuelbot
793FuseBulb searchFuseBulb.Com
794FyberSearch FyberSpider robotFyberSpider (+http://www.fybersearch.com/fyberspider.php)
795FyrebotFyrebot/1.0
796G2 Web ServicesG2 Web Services/1.0 (built with StormCrawler Archetype 1.8; https://www.g2webservices.com/; developers@g2llc.com)
797g2reader-botg2reader-bot/1.0 (+http://www.g2reader.com/)
798GarlikCrawlerGarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)
799GenieoMozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)
800GigablastGigablastOpenSource/1.0
801GigabotGigabot/1.0
802GigabotGigabot/2.0 (http://www.gigablast.com/spider.html)
803GingerCrawlerGingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)
804Global Opto's eApollo Flash based search engine - TaiwaneApolloBot/2.0 (compatible; heritrix/2.0.0-SNAPSHOT-20071024.170148 +http://www.eapollo-opto.com)
805Gluten Free Crawler\/Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)
806GnowitNewsbotMozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0 / GnowitNewsbot / Contact information at http://www.gnowit.com
807Go-http-clientGo-http-client/2.0
808Go-http-clientGo-http-client/1.1
809Google (216.239.39.x) proxy serverDoCoMo/1.0/P502i/c10 (Google CHTML Proxy/1.0)
810Google FaviconMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
811Google Feedfetcher - RSS and Atom feed crawlerFeedfetcher-Google; (+http://www.google.com/feedfetcher.html)
812Google news feed feetcher for iGoogle gadgetsFeedfetcher-Google-iGoogleGadgets; (+http://www.google.com/feedfetcher.html)
813Google Web PreviewMozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/4.0 Mobile Safari/537.36
814Google-Adwords-InstantGoogle-Adwords-Instant (+http://www.google.com/adsbot.html)
815Google-PhysicalWebMozilla/5.0 (Google-PhysicalWeb)
816Google-Site-VerificationMozilla/5.0 (compatible; Google-Site-Verification/1.0)
817Google-Structured-Data-Testing-ToolMozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +https://search.google.com/structured-data/testing-tool)
818google-xrawlergoogle-xrawler
819GooglebotGooglebot/2.1 (+http://www.googlebot.com/bot.html)
820GooglebotGooglebot-News
821GooglebotGooglebot/2.1 (+http://www.google.com/bot.html)
822GooglebotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
823GooglebotMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
824GooglebotMozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
825GooglebotSAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
826GooglebotMozilla/5.0 (iPhone; U; CPU iPhone OS) (compatible; Googlebot-Mobile/2.1; http://www.google.com/bot.html)
827GooglebotMediapartners-Google
828GooglebotAdsBot-Google (+http://www.google.com/adsbot.html)
829GooglebotMozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
830GooglebotDoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
831GooglebotGooglebot-Video/1.0
832Googlebot-ImageGooglebot-Image/1.0
833Googlebot-MobileSAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
834Googlebot-MobileNokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
835Googlebot-MobileMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
836Googlebot-MobileMozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
837Googlebot-MobileDoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
838Googlebot-NewsGooglebot-News
839Googlebot-VideoGooglebot-Video/1.0
840Googlebot\/Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
841Googlebot\/Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
842Googlebot\/Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
843Googlebot\/Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
844Googlebot\/Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
845Googlebot\/Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
846Googlebot\/Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
847Googlebot\/Googlebot/2.1 (+http://www.google.com/bot.html)
848GoonGee.com link popularity checkingBig Fish v1.0
849GowikibotMozilla/5.0 (compatible; Gowikibot/1.0; +http://www.gowikibot.com)
850GrapeshotCrawlerMozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
851GrobbotMozilla/5.0 (compatible; Grobbot/2.2; +https://grob.it)
852GroupHigh\/Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/
853grub.orgMozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)
854grub.orgMozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)
855grub.orgMozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)
856grub.orgMozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)
857grub.orgMozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)
858grub.orgMozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)
859grub.orgMozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)
860grub.orgMozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)
861grub.orgMozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)
862grub.orgMozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)
863grub.orgMozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)
864HatenaHatena::Fetcher/0.01 (master) Furl/3.13
865HatenaHatenaBookmark/4.0 (Hatena::Bookmark; Analyzer)
866HatenaHatena::Scissors/0.01
867HatenaHatena-Favicon/2 (http://www.hatena.ne.jp/faq/)
868HatenaHatena::Russia::Crawler/0.01
869HatenaHatena Antenna/0.3
870HeadlessChromeMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/76.0.3803.0 Safari/537.36
871HeadlessChromeMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/69.0.3494.0 Safari/537.36
872HeadlessChromeMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/74.0.3729.169 Safari/537.36
873heritrixMozilla/5.0 (compatible; heritrix/3.1.1 +http://www.mixdata.com)
874heritrixMozilla/5.0 (compatible; heritrix/3.1.1 +http://places.tomtom.com/crawlerinfo)
875heritrixMozilla/5.0 (compatible; heritrix/3.1.0-RC1 +http://boston.lti.cs.cmu.edu/crawler_12/)
876heritrixMozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
877heritrixMozilla/5.0 (compatible; heritrix/3.0.0-SNAPSHOT-20091120.021634 +http://crawler.archive.org)
878heritrixMozilla/5.0 (compatible; heritrix/2.0.2 +http://seekda.com)
879heritrixMozilla/5.0 (compatible; heritrix/2.0.2 +http://aihit.com)
880heritrixMozilla/5.0 (compatible; heritrix/1.14.4 +http://www.exif-search.com)
881heritrixMozilla/5.0 (compatible; heritrix/1.14.4 +http://parsijoo.ir)
882heritrixMozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.buddybuzz.net/yptrino)
883heritrixMozilla/5.0 (compatible; heritrix/1.14.3 +http://www.webarchiv.cz)
884heritrixMozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com)
885heritrixMozilla/5.0 (compatible; heritrix/1.14.2 +http://www.webarchiv.cz)
886heritrixMozilla/5.0 (compatible; heritrix/1.14.2 +http://rjpower.org)
887heritrixMozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html)
888heritrixMozilla/5.0 (compatible; heritrix/1.12.1 +http://www.webarchiv.cz)
889heritrixMozilla/5.0 (compatible; heritrix/1.14.3 +http://archive.org)
890heritrixMozilla/5.0 (compatible; sukibot_heritrix/3.1.1 +http://suki.ling.helsinki.fi/eng/webmasters.html)
891heritrixMozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20160309-0050; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
892heritrixMozilla/5.0 (compatible; heritrix/3.2.0 +http://www.mixdata.com)
893heritrixMozilla/5.0 (compatible; heritrix/3.2.0 +http://www.exif-search.com)
894heritrixMozilla/5.0 (compatible; heritrix/3.2.0 +http://www.crim.ca)
895Heritrix - The Internet Archive's open-source crawler (207.241.225.2xx)archive.org_bot
896http_gethttp_get
897httpunithttpunit/1.x
898HttpUrlConnectionJersey/2.25.1 (HttpUrlConnection 1.8.0_141)
899HTTrackMozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)
900HubSpotHubSpot Connect 2.0 (http://dev.hubspot.com/) - BizOpsCompanies-Tq2-BizCoDomainValidationAudit
901HubSpotMozilla/5.0 (compatible; HubSpot Crawler; web-crawlers@hubspot.com)
902HubSpotMozilla/5.0 (X11; Linux x86_64; HubSpot Single Page link check; web-crawlers+links@hubspot.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
903HubSpotMozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler - web-crawlers@hubspot.com
904Hungarian Academy of Sciences data mining searchComputer_and_Automation_Research_Institute_Crawler crawler@ilab.sztaki.hu
905ia_archiveria_archiver-web.archive.org
906ia_archiveria_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
907IAS crawlerIAS crawler (ias_crawler; http://integralads.com/site-indexing-policy/)
908IBM's Almaden Research robotFocusedSampler/1.0
909ICBot\/Mozilla/5.0 (compatible; ICBot/0.1; +https://ideasandcode.xyz
910ICC-CrawlerICC-Crawler/2.0 (Mozilla-compatible; ; http://ucri.nict.go.jp/en/icccrawler.html)
911IceRocket Web search robotBlogSearch/1.x +http://www.icerocket.com/
912IceRocket Web search robotBlogzIce/1.0 (+http://icerocket.com; rhodes@icerocket.com)
913IceRocket Web search robotBlogzIce/1.0 +http://www.icerocket.com/
914ichiroichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)
915ichiroichiro/3.0 (http://help.goo.ne.jp/help/article/1142)
916ichiroichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
917ichiroKDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
918ichiroKDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
919ichiroDoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo;+http://help.goo.ne.jp/door/crawler.html)
920ichiroDoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/door/crawler.html)
921ichiroichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)
922ichiroichiro/3.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)
923ichiroichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)
924ichiroichiro/2.0 (ichiro@nttr.co.jp)
925ichiroKDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
926ichiroDoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
927ichiroDoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
928ichiroDoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
929imrbotMozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)
930IndeedBotMozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0 (IndeedBot 1.1)
931INFOMINE/iVia Scholary Internet Resource Collections robotDataFountains/DMOZ Downloader
932INFOMINE/iVia Scholary Internet Resource Collections robotDataFountains/Dmoz Downloader (http://ivia.ucr.edu/useragents.shtml)
933INFOMINE/iVia Scholary Internet Resource Collections robotDataFountains/DMOZ Feature Vector Corpus Creator (http://ivia.ucr.edu/useragents.shtml)
934Inktomi (Hotbot-Lycos NBCi) robotBSDSeek/1.0
935integromedbwww.integromedb.org/Crawler
936Intelliseek's BlogPulse blog searchBlogpulse (info@blogpulse.com)
937Intelliseek's BlogPulse blog searchBlogPulseLive (support@blogpulse.com)
938Internet Explorer versions 4.x and 5 plugin contentcontype
939IPS AgentMozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1
940IPS AgentMozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4 Firefox/3.5.3
941IPS AgentMozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7
942IPS AgentBlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent
943IPS AgentMozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24; ips-agent) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24
944ips-agentMozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1
945ips-agentMozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24; ips-agent) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24
946ips-agentMozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4Firefox/3.5.3
947ips-agentMozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7
948ips-agentBlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent
949iskanieiskanie (+http://www.iskanie.com)
950IstellaBotMozilla/5.0 (compatible; IstellaBot/1.23.15 +http://www.tiscali.it/)
951it2media-domain-crawlerit2media-domain-crawler/2.0
952it2media-domain-crawlerit2media-domain-crawler/1.0 on crawler-prod.it2media.de
953James BOTMozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html
954Jamie's SpiderJamie's Spider (http://jamiembrown.com/)
955JavaOS app. for Sanyo Internet-TVAplix_SANYO_browser/1.x (Japanese)
956JavaOS app. for SEGA Saturn InternetAplix_SEGASATURN_browser/1.x (Japanese)
957JavaOS app. for SEGA Saturn Internet and Sanyo Internet-TVAplix HTTP/1.0.1
958JetCar/FlashGet download managerFlashGet
959JetslideMozilla/5.0 (compatible; Jetslide; +http://jetsli.de/crawler)
960JettyJetty/9.3.z-SNAPSHOT
961JooblebotMozilla/5.0 (compatible; Jooblebot/2.0; Windows NT 6.1; WOW64; +http://jooble.org/jooble-bot) AppleWebKit/537.36 (KHTML, like Gecko) Safari/537.36
962jpg-newsbotjpg-newsbot/2.0; (+https://vipnytt.no/bots/)
963Jugendschutzprogramm-CrawlerJugendschutzprogramm-Crawler; Info: http://www.jugendschutzprogramm.de
964K7MLWCBotK7MLWCBot/1.0 (+http://www.k7computing.com)
965KemvibotKemvibot/1.0 (http://kemvi.com, marco@kemvi.com)
966Keywen Encyclopedia BotEasyDL/3.xx
967Keywen Encyclopedia BotEasyDL/3.xx http://keywen.com/Encyclopedia/Bot
968KosmioBotMozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36 (compatible; KosmioBot/1.0; +http://kosm.io/bot.html)
969Landau-Media-SpiderLandau-Media-Spider/1.0(http://bots.landaumedia.de/bot.html)
970LaserlikebotMozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Laserlikebot/0.1)
971LeikibotMozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com)
972LibMaster.com Active Bookmark HTML page creatorActiveBookmark 1.x
973libwww-perl2Bone_LinkChecker/1.0 libwww-perl/6.03
974libwww-perl2Bone_LinkChkr/1.0 libwww-perl/6.03
975libwww-perlamibot - http://www.amidalla.de - tech@amidalla.com libwww-perl/5.831
976Linguee BotLinguee Bot (http://www.linguee.com/bot)
977Linguee BotLinguee Bot (http://www.linguee.com/bot; bot@linguee.com)
978linkapediabotlinkapediabot (+http://www.linkapedia.com)
979LinkArchiver@LinkArchiver twitter bot
980LinkcheckerBeepware (site is down) web directory link checkingBeebwareDirectory/v0.01
981linkdexMozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/about/bots/)
982linkdexMozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
983linkdexMozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)
984linkdexlinkdexbot/Nutch-1.0-dev (http://www.linkdex.com/; crawl at linkdex dot com)
985linkdexlinkdex.com/v2.0
986linkdexMozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)
987linkdexMozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/bots/)
988LinkedInBotLinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)
989LinkedInBotLinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.3 +http://www.linkedin.com)
990LinkedInBotLinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
991Linkomatic submission verifierAgentName/0.1 libwww-perl/5.48
992Linkscan tool from Elsopautohttp
993lipperheyMozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)
994lipperheyMozilla/5.0 (compatible; Lipperhey Site Explorer; http://www.lipperhey.com/)
995lipperheyMozilla/5.0 (compatible; Lipperhey SEO Service; http://www.lipperhey.com/)
996lipperheyMozilla/5.0 (compatible; Lipperhey Link Explorer; http://www.lipperhey.com/)
997Livelap[bB]otLivelapbot/0.1
998Livelap[bB]otLivelapBot/0.2 (http://site.livelap.com/crawler)
999Logitech Desktop Managers (LDM) Backweb (BW) update checkBW-C-2.0
1000ltx71ltx71 - (http://ltx71.com/)
1001Luminator-robotsMozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.13 (KHTML, like Gecko) Chrome/30.0.1599.66 Safari/537.13 Luminator-robots/2.0
1002magpie-crawlermagpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
1003Mail.RU_BotMozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
1004Mail.RU_BotMozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/
1005Mail.RU_BotMozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/
1006Mail.RU_BotMozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
1007mappydataMozilla/5.0 (compatible; Mappy/1.0; +http://mappydata.net/bot/)
1008Mastodonhttp.rb/2.2.2 (Mastodon/1.5.1; +https://example-masto-instance.org/)
1009MauiBotMauiBot (crawler.feedback+wc@gmail.com)
1010MaxOS X CoreFoundation CFNetwork APICFNetwork/x.x
1011MBCrawlerMBCrawler/1.0 (https://monitorbacklinks.com)
1012Mediapartners-GoogleMozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
1013Mediapartners-GoogleMozilla/5.0 (iPhone; U; CPU iPhone OS 10_0 like Mac OS X; en-us) AppleWebKit/602.1.38 (KHTML, like Gecko) Version/10.0 Mobile/14A5297c Safari/602.1 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
1014Mediapartners-GoogleMozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0 (Following Mediapartners-Google)
1015Mediapartners-GoogleMediapartners-Google
1016MediatoolkitbotMediatoolkitbot (complaints@mediatoolkit.com)
1017MegaIndexMozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)
1018Megaproxy userAnonymized by ProxyOS: http://www.megaproxy.com
1019MeltwaterNewsMeltwaterNews www.meltwater.com
1020memorybotMozilla/5.0 (compatible; memorybot/1.21.14 +http://mignify.com/bot.html)
1021Metacarta / Cogent robotflunky
1022MetaJobBotMozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler)
1023MetaURIMetaURI API/2.0 +metauri.com
1024Microsoft server information robot (see link)BunnySlippers
1025MiggiBot website crawler engine - A1 Sitemap GeneratorA1 Sitemap Generator/1.0 (+http://www.micro-sys.dk/products/sitemap-generator/) miggibot/2006.01.24
1026mindUpBotmindUpBot (datenbutler.de)
1027Miniflux\/Mozilla/5.0 (compatible; Miniflux/2.0.x-dev; +https://miniflux.net)
1028Miniflux\\/Mozilla/5.0 (compatible; Miniflux/2.0.12; +https://miniflux.app)
1029Miniflux\\/Mozilla/5.0 (compatible; Miniflux/3b6e44c; +https://miniflux.app)
1030Miniflux\\/Mozilla/5.0 (compatible; Miniflux/ae1dc1a; +https://miniflux.app)
1031Miniflux\\/Mozilla/5.0 (compatible; Miniflux/2.0.11; +https://miniflux.app)
1032Miniflux\\/Mozilla/5.0 (compatibl$; Miniflux/2.0.x-dev; +https://miniflux.app)
1033Miniflux\\/Mozilla/5.0 (compatible; Miniflux/2.0.10; +https://miniflux.net)
1034Miniflux\\/Mozilla/5.0 (compatible; Miniflux/2.0.7; +https://miniflux.net)
1035Miniflux\\/Mozilla/5.0 (compatible; Miniflux/2.0.3; +https://miniflux.net)
1036Miniflux\\/Mozilla/5.0 (compatible; Miniflux/2.0.x-dev; +https://miniflux.net)
1037Mirago UK RobotExperimentalHenrytheMiragoRobot
1038MixCat robotFelix - Mixcat Crawler (+http://mixcat.com)
1039MixnodeCache\/MixnodeCache/1.8(+https://cache.mixnode.com/)
1040MJ12botMozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)
1041MJ12botMozilla/5.0 (compatible; MJ12bot/v1.3.2; http://www.majestic12.co.uk/bot.php?+)
1042MJ12botMozilla/5.0 (compatible; MJ12bot/v1.3.1; http://www.majestic12.co.uk/bot.php?+)
1043MJ12botMozilla/5.0 (compatible; MJ12bot/v1.3.0; http://www.majestic12.co.uk/bot.php?+)
1044MJ12botMozilla/5.0 (compatible; MJ12bot/v1.2.5; http://www.majestic12.co.uk/bot.php?+)
1045MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
1046MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.7; http://www.majestic12.co.uk/bot.php?+)
1047MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)
1048MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.6; http://mj12bot.com/)
1049MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
1050MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
1051MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+)
1052MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)
1053MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+)
1054MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)
1055MJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.4 (domain ownership verifier); http://www.majestic12.co.uk/bot.php?+)
1056MJ12botMozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)
1057MJ12botMozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+)
1058MJ12botMJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
1059MJ12botMozilla/5.0 (compatible; MJ12bot/v1.2.4; http://www.majestic12.co.uk/bot.php?+)
1060mlbotMLBot (www.metadatalabs.com/mlbot)
1061MlbotMLBot (www.metadatalabs.com/mlbot)
1062moatbotMozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36 moatbot
1063MojeekBot\/MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)
1064MojeekBot\\/Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)
1065MojeekBot\\/Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
1066MojeekBot\\/Mozilla/5.0 (compatible; MojeekBot/0.5; http://www.mojeek.com/bot.html)
1067MojeekBot\\/Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html)
1068MojeekBot\\/Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html#relaunch)
1069MojeekBot\\/MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)
1070MoodleBotMoodleBot/1.0
1071MoreoverMozilla/5.0 Moreover/5.1 (+http://www.moreover.com)
1072msnbotmsnbot/2.0b (+http://search.msn.com/msnbot.htm).
1073msnbotmsnbot/1.1 (+http://search.msn.com/msnbot.htm)
1074msnbotmsnbot/2.0b (+http://search.msn.com/msnbot.htm)
1075msnbotadidxbot/2.0 (+http://search.msn.com/msnbot.htm)
1076msnbotadidxbot/1.1 (+http://search.msn.com/msnbot.htm)
1077msnbotmsnbot/2.0b (+http://search.msn.com/msnbot.htm)._
1078msnbotmsnbot/1.0 (+http://search.msn.com/msnbot.htm)
1079msnbotmsnbot-media/2.0b (+http://search.msn.com/msnbot.htm)
1080msnbotmsnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
1081msnbotmsnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
1082msnbotmsnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)
1083msnbotmsnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)
1084msnbotlibrabot/2.0 (+http://search.msn.com/msnbot.htm)
1085msnbotlibrabot/1.0 (+http://search.msn.com/msnbot.htm)
1086MuckRackMozilla/5.0 (compatible; MuckRack/1.0; +http://muckrack.com)
1087MultiviewbotMozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Multiviewbot
1088Nameprotect copyright search robot (24.177.134.x)abot/0.1 (abot; http://www.abot.com; abot@abot.com)
1089Nameprotect copyright search robot (24.177.134.x)aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com)
1090Nameprotect copyright search robot (24.177.134.x)aipbot/2-beta (aipbot dev; http://aipbot.com; aipbot@aipbot.com)
1091National Research Council Canada robotAVSearch-1.0(peter.turney@nrc.ca)
1092Naver Japan / Korea robotCowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
1093Naver Japan / Korea robotCowbot-0.1.x (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
1094Naver Japan / Korea robotdloader(NaverRobot)/1.0
1095neofonie search robot Germany:robot/1.0 (linux) ( admin e-mail: undefined http://www.neofonie.de/loesungen/search/robot.html )
1096Neomo Search (Germany) robot (85.10.197.1xx)Francis/1.0 (francis@neomo.de http://www.neomo.de/)
1097NerdByNature.BotMozilla/5.0 (compatible; NerdByNature.Bot; http://www.nerdbynature.net/bot)
1098nerdybotnerdybot
1099NetcraftSurveyAgentMozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)
1100netEstate NE CrawlernetEstate NE Crawler (+http://www.sengine.info/)
1101NetvibesNetvibes (crawler; http://www.netvibes.com)
1102NetvibesNetvibes (crawler/bot; http://www.netvibes.com
1103Newsbrain Feedreader3Feedreader 3.xx (Powered by Newsbrain)
1104newsharecountsMozilla/5.0 (compatible; NewShareCounts.com/1.0; +http://newsharecounts.com/crawler)
1105newspaper\/newspaper/0.1.0.7
1106newspaper\\/newspaper/0.2.8
1107newspaper\\/newspaper/0.2.6
1108newspaper\\/newspaper/0.2.5
1109newspaper\\/newspaper/0.1.0.7
1110NextCloudNextCloud-News/1.0
1111NextLinks - German vocabulary and hyperlink searchfindlinks/x.xxx (+http://wortschatz.uni-leipzig.de/findlinks/)
1112NF embedded browser on Amois SkypephoneAmoi 8512/R21.0 NF-Browser/3.3
1113Nimbostratus-BotMozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)
1114NING\/NING/1.0
1115Nmap Scripting EngineMozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)
1116Nordic semantic search engineCentiverseBot
1117Nordic semantic search engineCentiverseBot - investigator
1118Nordic semantic search engineCentiverseBot/3.0 (http://www.centiverse-project.net)
1119NTL user agentCheckUrl
1120NTT DoCoMo (Japan) proxy server (210.136.161.1xx)DoCoMo/2.0 SH901iS(c100;TB;W24H12),gzip(gfe) (via translate.google.com)
1121NTT DoCoMo (Japan) robotDoCoMo/1.0/Nxxxi/c10
1122NTT DoCoMo (Japan) robotDoCoMo/1.0/Nxxxi/c10/TB
1123NTT DoCoMo (Japan) robotDoCoMo/2.0 P900iV(c100;TB;W24H11)
1124nutchNutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
1125Nutchistellabot-nutch/Nutch-1.10
1126NuzzelNuzzel
1127OcarinabotOcarinabot
1128okhttpokhttp/2.7.5
1129okhttpokhttp/3.2.0
1130okhttpokhttp/2.5.0
1131okhttpokhttp/4.1.0
1132okhttpokhttp/3.5.0
1133omgiliomgili/0.5 +http://omgili.com
1134online-webceo-botMozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)
1135OpenHoseBotMozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)
1136OpenPortal4U robotDigOut4U
1137OrangeBot\/Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com
1138outbrainMozilla/5.0 (Java) outbrain
1139OutclicksBotOutclicksBot/2 +https://www.outclicks.net/agent/p2i4sNUh7eylJF1S6SGgRs5mP40ExlYvsr9GBxVQG6h
1140OutclicksBotOutclicksBot/2 +https://www.outclicks.net/agent/PryJzTl8POCRHfvEUlRN5FKtZoWDQOBEvFJ2wh6KH5J
1141OutclicksBotOutclicksBot/2 +https://www.outclicks.net/agent/VjzDygCuk4ubNmg40ZMbFqT0sIh7UfOKk8s8ZMiupUR
1142OutclicksBotOutclicksBot/2 +https://www.outclicks.net/agent/gIYbZ38dfAuhZkrFVl7sJBFOUhOVct6J1SvxgmBZgCe
1143page2rssMozilla/5.0 (compatible;Page2RSS/0.7; +http://page2rss.com/)
1144page2rssMozilla/5.0 (compatible; Page2RSS/0.7; +http://page2rss.com/)
1145Pages Jaunes business search (France) robot using Fast Enterprise CrawlerFAST Enterprise Crawler 6 used by Pages Jaunes (pvincent@pagesjaunes.fr)
1146panscientpanscient.com
1147PaperLiBotMozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)
1148Part of ButtMan remote access toolBMCLIENT
1149Pcore-HTTPPcore-HTTP/v0.40.3
1150PhantomJSMozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1 bl.uk_lddc_renderbot/2.0.0 (+ http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
1151Phone.com UP.Browser for mobiles on Alcatel cellphoneAlcatel-BG3/1.0 UP.Browser/5.0.3.1.2
1152phpcrawlphpcrawl
1153pingdomMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/59.0.3071.109 Chrome/59.0.3071.109 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; +http://www.pingdom.com/)
1154Pinpoint WAP search robotCrawlerBoy Pinpoint.com
1155pinterest.com.botMozilla/5.0 (compatible; Pinterestbot/1.0; +http://www.pinterest.com/bot.html)
1156PiplBotPiplBot (+http://www.pipl.com/bot/)
1157PocketParserPocketParser/2.0 (+https://getpocket.com/pocketparser_ua)
1158postrankPostRank/2.0 (postrank.com)
1159postrankPostRank/2.0 (postrank.com; 1 subscribers)
1160PowerTCP ActiveX control toolDart Communications PowerTCP
1161PR-CY.RUMozilla/5.0 (compatible; PR-CY.RU; + https://a.pr-cy.ru)
1162PrimalbotMozilla/5.0 (compatible; Primalbot; +https://www.primal.com;)
1163PrivacyAwareBotMozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)
1164Privoxy web proxy(Privoxy/1.0)
1165Probably E-Mail harvesting robot - same as LMQueueBotContactBot/0.2
1166Project BanBots Perl script robotBanBots/1.2 (spider@banbots.com)
1167Proposed Content-Based Image Tracking System (CoBITS) P2P crawlerCoBITSProbe
1168ProximicMozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)
1169proximicMozilla/5.0 (compatible; proximic; +http://www.proximic.com)
1170ProximicMozilla/5.0 (compatible; proximic; +http://www.proximic.com)
1171ProximicMozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
1172psbotpsbot-page (+http://www.picsearch.com/bot.html)
1173psbotpsbot-image (+http://www.picsearch.com/bot.html)
1174psbotpsbot/0.1 (+http://www.picsearch.com/bot.html)
1175PubSub - Mac OS X utility for managing RSS/Atom subscriptions via the PubSub frameworkApple-PubSub/65.1.1
1176PulsepointPulsepoint XT3 web scraper
1177python-requestspython-requests/2.9.2
1178python-requestspython-requests/2.11.1
1179python-requestspython-requests/2.22.0
1180python-requestspython-requests/2.21.0
1181python-requestspython-requests/2.20.0
1182python-requestspython-requests/2.19.1
1183python-requestspython-requests/2.18.4
1184Python-urllibPython-urllib/3.2
1185Python-urllibPython-urllib/3.7
1186Python-urllibPython-urllib/3.4
1187Python-urllibPython-urllib/1.17
1188Python-urllibPython-urllib/3.5
1189Python-urllibPython-urllib/2.6
1190Python-urllibPython-urllib/3.3
1191Python-urllibPython-urllib/2.7
1192Python-urllibPython-urllib/3.1
1193Python-urllibPython-urllib/2.5
1194Python-urllibPython-urllib/3.6
1195QwantifyMozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)/*
1196QwantifyMozilla/5.0 (compatible; Qwantify/Bleriot/1.1; +https://help.qwant.com/bot)
1197QwantifyMozilla/5.0 (compatible; Qwantify/2.4w; +https://www.qwant.com/)/2.4w
1198QwantifyMozilla/5.0 (compatible; Qwantify/Bleriot/1.2.1; +https://help.qwant.com/bot)
1199RankActiveLinkBotMozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)
1200redditbotMozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback)
1201RegionStuttgartBotMozilla/5.0 (compatible; RegionStuttgartBot/1.0; +http://it.region-stuttgart.de/competenzatlas/unternehmen-suchen/)
1202Relevare Portal software robotbumblebee@relevare.com
1203RetrevoPageAnalyzerMozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; RetrevoPageAnalyzer; +http://www.retrevo.com/content/about-us)
1204RivvaMozilla/5.0 (compatible; Rivva; http://rivva.de)
1205Robot for BravoBrian bSTOPBravoBrian SpiderEngine MarcoPolo
1206rogerbotrogerbot/1.2 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+phaser-testing-crawler-01@moz.com)
1207rogerbotrogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com
1208rogerbotMozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)
1209rogerbotrogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)
1210rogerbotrogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-11@moz.com)
1211rogerbotrogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-15@moz.com)
1212rogerbotrogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr2-crawler-05@moz.com)
1213rogerbotrogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-wherecat@moz.com)
1214rogerbotrogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler@seomoz.org)
1215rogerbotrogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler+shiny@seomoz.org)
1216rogerbotrogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)
1217rogerbotrogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com)
1218rogerbotrogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+partager@moz.com)
1219RSS Feed Seeker bot (68.225.95.2xx)Feed Seeker Bot (RSS Feed Seeker http://www.MyNewFavoriteThing.com/fsb.php)
1220RSSingBotRSSingBot (http://www.rssing.com)
1221S[eE][mM]rushBotMozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)
1222S[eE][mM]rushBotSEMrushBot
1223S[eE][mM]rushBotMozilla/5.0 (compatible; SemrushBot/0.98~bl; +http://www.semrush.com/bot.html)
1224S[eE][mM]rushBotMozilla/5.0 (compatible; SemrushBot-BA; +http://www.semrush.com/bot.html)
1225S[eE][mM]rushBotMozilla/5.0 (compatible; SemrushBot/3~bl; +http://www.semrush.com/bot.html)
1226S[eE][mM]rushBotMozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)
1227S[eE][mM]rushBotMozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
1228Safe-t.net web filtering serviceFANGCrawl/0.01
1229SafeDNSBotSafeDNSBot (https://www.safedns.com/searchbot)
1230SafeSearch microdata crawlerSafeSearch microdata crawler (https://safesearch.avira.com, safesearch-abuse@avira.com)
1231Samsung SPH-A660 phone with Sprint softwareAU-MIC/2.0 MMP/2.0
1232SBL-BOTSBL-BOT (http://sbl.net)
1233ScoutJetMozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
1234ScrapyScrapy/1.0.3 (+http://scrapy.org)
1235Screaming Frog SEO SpiderScreaming Frog SEO Spider/5.1
1236Searchengine.com (Germany) submission checking / robot (84.73.57.xx)de.searchengine.comBot 1.2 (http://de.searchengine.com/spider)
1237SearchEngineWorld's robots.txt validator( Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi )
1238Searchspider.com robot (72.245.225.xxx)dev-spider2.searchpsider.com/1.3b
1239seekbotSeekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2
1240Seekport CrawlerMozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/)
1241semanticbotsemanticbot
1242semanticbotsemanticbot (info@semanticaudience.com)
1243SemanticScholarBotSemanticScholarBot/1.0 (+http://s2.allenai.org/bot.html)
1244SEOkicksMozilla/5.0 (compatible; SEOkicks; +https://www.seokicks.de/robot.html)
1245seoscannersMozilla/5.0 (compatible; seoscanners.net/1; +spider@seoscanners.net)
1246serpstatbot\/serpstatbot/1.0 (advanced backlink tracking bot; http://serpstatbot.com/; abuse@serpstatbot.com)
1247Seznam Search (Czech Republic) robotCloakDetect/0.9 (+http://fulltext.seznam.cz/)
1248seznambotMozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
1249seznambotMozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://napoveda.seznam.cz/en/seznambot-intro/)
1250seznambotMozilla/5.0 (compatible; SeznamBot/3.2-test1-1; +http://napoveda.seznam.cz/en/seznambot-intro/)
1251seznambotMozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://napoveda.seznam.cz/en/seznambot-intro/)
1252seznambotMozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/)
1253SharewarePlaza link checkingAgent-SharewarePlazaFileCheckBot/2.0+(+http://www.SharewarePlaza.com)
1254Sicher-durchs-Netz German security related search (62.75.220.xxx)DatenBot( http://www.sicher-durchs-netz.de/bot.html)
1255SimpleCrawlerSimpleCrawler/0.1
1256SimpleScraperMozilla/5.0 (compatible; SimpleScraper)
1257Simpy Bookmarklet crawler (69.55.233.xx)Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com)
1258Singingfish media spider (64.12.186.2xx) via AOL searchasterias/2.0
1259sitebotMozilla/5.0 (compatible; Whoiswebsitebot/0.1; +http://www.whoiswebsite.net)
1260siteexplorer.infoMozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)
1261siteexplorer.infoMozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
1262Siteimprove.comMozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) LinkCheck by Siteimprove.com
1263Siteimprove.comMozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com
1264Siteimprove.comMozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com
1265Siteimprove.comMozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) Match by Siteimprove.com
1266SitiDi.net search (Germany) robot+SitiDi.net/SitiDiBot/1.0 (+Have Good Day)
1267SkypeUriPreviewMozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5
1268Slack-ImgProxySlack-ImgProxy 1.106 (+https://api.slack.com/robots)
1269Slack-ImgProxySlack-ImgProxy 0.59 (+https://api.slack.com/robots)
1270Slack-ImgProxySlack-ImgProxy 149 (+https://api.slack.com/robots)
1271Slack-ImgProxySlack-ImgProxy 1.138 (+https://api.slack.com/robots)
1272Slack-ImgProxySlack-ImgProxy 0.66 (+https://api.slack.com/robots)
1273Slack-ImgProxySlack-ImgProxy (+https://api.slack.com/robots)
1274SlackbotSlackbot-LinkExpanding (+https://api.slack.com/robots)
1275SlurpMozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
1276smtbotMozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
1277smtbotMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)
1278smtbotSMTBot (similartech.com/smtbot)
1279smtbotMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
1280smtbotMozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
1281SnacktoryMozilla/5.0 (compatible; Snacktory; +https://github.com/karussell/snacktory)
1282SocialRankIOBotSocialRankIOBot; http://socialrank.io/about
1283SogouSogou Pic Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
1284SogouSogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
1285SogouSogou News Spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
1286SogouSogou Orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
1287SogouSogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
1288some (website) downloading toolDDD
1289some download agentdlman
1290Some site scanning tool from 217.34.59.xxx (btopenworld.com)Demo Bot DOT 16b
1291Some site scanning tool from 66.255.6.xxx (uslec.com)Full Web Bot 2816B
1292Some site scanning tool from 68.154.96.xx (bellsouth.net)Demo Bot Z 16b
1293Some site scanning tool from diff. IPs i.e.: - 66.28.240.xx (cogentco.com) - 68.5.174.xx (cox.net)Full Web Bot 0416B
1294Some site scanning tool i.e. from - 68.154.96.xx (bellsouth.net)Full Web Bot 0516B
1295Some site scanning tool via diff. IPsESurf15a 15
1296Some site scanning tool via diff. IPsFSurf15a 01
1297Some site scanning tool via diff. IPs i.e.: - cox.net (68.4.xxx.xxx)DSurf15a 71
1298Some site scanning tool via diff. IPs i.e.: - cox.net (68.4.xxx.xxx)Educate Search VxB
1299Some site scanning tool via diff. IPs i.e.: - cox.net (68.5.xxx.xxx) - pacbell.net (64.16x.xxx.xxx)DSurf15a 01
1300Some site scanning tool via diff. IPs i.e.: - eastlink.ca (24.222.xxx.xxx) - cogeco.net (216.221.8x.xxx)DSurf15a VA
1301Some site scanning tool via diff. IPs i.e.: - pacbell.net (67.112.xxx.xxx)DBrowse 1.4d
1302Some site scanning tool via diff. IPs i.e.: - swbell.net (65.66.xxx.xxx)EBrowse 1.4b
1303Some site scanning tool via diff. IPs i.e.: - verizon.net (4.47.xxx.xxx)DSurf15a 81
1304Some site scanning tool via diff. IPs i.e.: - wanweb.net (208.6.163.xxx) - cox.net (68.4.xxx.xxx)DBrowse 1.4b
1305Some spam botFranklin Locator 1.8
1306Some user agentCancer Information and Support International;
1307SonicMozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
1308SonicMozilla/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
1309SonicMozilla/5.0 (compatible; RankSonicSiteAuditor/1.0; +https://ranksonic.com/ranksonic_sab.html)
1310Sonic E-mail collectorEmailSiphon
1311Sootle Web Directory deep link checkerDeep Link Calculator v1.0
1312SorgouSogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
1313spbotMozilla/5.0 (compatible; spbot/4.0.7; +https://www.seoprofiler.com/bot )
1314spbotMozilla/5.0 (compatible; spbot/4.4.1; +http://OpenLinkProfiler.org/bot )
1315spbotMozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )
1316spbotMozilla/5.0 (compatible; spbot/5.0.1; +http://OpenLinkProfiler.org/bot )
1317spbotMozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
1318spbotMozilla/5.0 (compatible; spbot/1.0; +http://www.seoprofiler.com/bot/ )
1319spbotMozilla/5.0 (compatible; spbot/4.0a; +http://www.seoprofiler.com/bot )
1320spbotMozilla/5.0 (compatible; spbot/5.0; +http://OpenLinkProfiler.org/bot )
1321spbotMozilla/5.0 (compatible; spbot/4.4.0; +http://OpenLinkProfiler.org/bot )
1322spbotMozilla/5.0 (compatible; spbot/4.3.0; +http://OpenLinkProfiler.org/bot )
1323spbotMozilla/5.0 (compatible; spbot/4.0; +http://www.seoprofiler.com/bot )
1324spbotMozilla/5.0 (compatible; spbot/4.0.5; +http://www.seoprofiler.com/bot )
1325spbotMozilla/5.0 (compatible; spbot/4.0.3; +http://www.seoprofiler.com/bot )
1326spbotMozilla/5.0 (compatible; spbot/4.0.2; +http://www.seoprofiler.com/bot )
1327spbotMozilla/5.0 (compatible; spbot/3.1; +http://www.seoprofiler.com/bot )
1328spbotMozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot )
1329spbotMozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ )
1330spbotMozilla/5.0 (compatible; spbot/2.0.4; +http://www.seoprofiler.com/bot )
1331spbotMozilla/5.0 (compatible; spbot/2.0.1; +http://www.seoprofiler.com/bot/ )
1332spbotMozilla/5.0 (compatible; spbot/1.2; +http://www.seoprofiler.com/bot/ )
1333spbotMozilla/5.0 (compatible; spbot/5.0.2; +http://OpenLinkProfiler.org/bot )
1334spbotMozilla/5.0 (compatible; spbot/4.2.0; +http://OpenLinkProfiler.org/bot )
1335spbotMozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )
1336spbotMozilla/5.0 (compatible; spbot/4.0b; +http://www.seoprofiler.com/bot )
1337spbotMozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )
1338spbotMozilla/5.0 (compatible; spbot/4.0.8; +http://OpenLinkProfiler.org/bot )
1339spbotMozilla/5.0 (compatible; spbot/4.0.7; +http://OpenLinkProfiler.org/bot )
1340spbotMozilla/5.0 (compatible; spbot/4.0.6; +http://www.seoprofiler.com/bot )
1341spbotMozilla/5.0 (compatible; spbot/4.0.4; +http://www.seoprofiler.com/bot )
1342spbotMozilla/5.0 (compatible; spbot/4.0.1; +http://www.seoprofiler.com/bot )
1343spbotMozilla/5.0 (compatible; spbot/2.1; +http://www.seoprofiler.com/bot )
1344spbotMozilla/5.0 (compatible; spbot/2.0.3; +http://www.seoprofiler.com/bot/ )
1345spbotMozilla/5.0 (compatible; spbot/2.0.2; +http://www.seoprofiler.com/bot/ )
1346spbotMozilla/5.0 (compatible; spbot/1.1; +http://www.seoprofiler.com/bot/ )
1347speedyMozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)
1348speedyMozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider for SpeedyAds (http://www.entireweb.com/about/search_tech/speedy_spider/)
1349speedySpeedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
1350speedySpeedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)
1351speedyMozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
1352SPH Search - Singapore related search using Fast crawlerFAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg)
1353StorygizeBotMozilla/5.0 (compatible; StorygizeBot; http://www.storygize.com)
1354Streamline3Bot\/Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) Streamline3Bot/1.0
1355Suchmaschine21 (Germany) robotCoolBot
1356summifySummify (Summify/1.0.1; +http://summify.com)
1357SurveyBotMozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)
1358SWIMGBotMozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36 SWIMGBot
1359Swsoft.net robot using AspseekASPseek/1.2.9d
1360Sygol Search (Italy) robot http://www.sygol.com
1361Sync2It bookmark management & clustering engine!Susie (http://www.sync2it.com/susie)
1362SysomosMozilla/5.0 (compatible; Sysomos/1.0; +http://www.sysomos.com/; Sysomos)
1363TangibleeBotTangibleeBot/1.0.0.0 (http://tangiblee.com/bot)
1364TelegramBotTelegramBot (like TwitterBot)
1365TeomaMozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
1366TeomaMozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)
1367TeomaMozilla/2.0 (compatible; Ask Jeeves/Teoma)
1368TeomaMozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
1369theoldreader.comMozilla/5.0 (compatible; theoldreader.com)
1370ThinklabThinklab (thinklab.com)
1371TimelyWeb web page monitoring toolEldoS TimelyWeb/3.x
1372Timewe mobile browser (WAP) simulator (Japan)CDR/1.7.1 Simulator/0.7(+http://timewe.net) Profile/MIDP-1.0 Configuration/CLDC-1.0
1373TinEyeMozilla/5.0 (compatible; TinEye-bot/1.31; +http://www.tineye.com/crawler.html)
1374Tiny Tiny RSSTiny Tiny RSS/1.15.3 (http://tt-rss.org/)
1375Tomba project: the Portuguese web archiveArquivo-web-crawler (compatible; heritrix/1.12.1 +http://arquivo-web.fccn.pt)
1376ToutiaoSpiderMozilla/5.0 (compatible; ToutiaoSpider/1.0; http://web.toutiao.com/media_cooperation/;)
1377Traackr.comTraackr.com
1378tracemyfileMozilla/5.0 (compatible; tracemyfile/1.0; +bot@tracemyfile.com)
1379Trellian EMailWolf E-mail collectorEmailWolf 1.00
1380trendictionbotMozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
1381TrendsmapResolverMozilla/5.0 (compatible; TrendsmapResolver/0.1)
1382TurnitinBotTurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
1383tweetedtimesMozilla/5.0 (compatible; +http://tweetedtimes.com)
1384TweetmemeBotMozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
1385TwinglyMozilla/5.0 (compatible; Twingly Recon; twingly.com)
1386TwitterbotTwitterbot/1.0
1387TwitterbotTwitterbot/0.1
1388TwurlyRuby, Twurly v1.1 (http://twurly.org)
1389UCLA Complex Networks Groups Complex Network Analysiscomplex_network_group/Nutch-0.9-dev (discovering the structure of the world-wide-web; http://cantor.ee.ucla.edu/~networks/crawl; nimakhaj@gmail.com)
1390UIUCs Cazoodle search based on MetaQuerierCazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com)
1391um-LNMozilla/5.0 (compatible; um-LN/1.0; mailto: techinfo@ubermetrics-technologies.com)
1392UnCHAOS search robot UnChaos From Chaos To Order Hybrid Web Search Engine.(vadim_gonchar@unchaos.com)
1393UnCHAOS search robot UnChaos Bot Hybrid Web Search Engine. (vadim_gonchar@unchaos.com)
1394UnCHAOS search robot UnChaosBot From Chaos To Order UnChaos Hybrid Web Search Engine at www.unchaos.com (info@unchaos.com)
1395unknownContact
1396unknownfly/6.01 libwww/4.0D
1397Unknown bot from Kornet Korea (218.149.129.xxx) scans for Perl AwstatsDataCha0s/2.0
1398unknown divine/Openmarket.com robotdCSbot/1.1
1399Unknown guestbook spamming or harvesting tool from diff. IPs8484 Boston Project v 1.0
1400Unknown robot from 205.203.108.xx (telerate.com)AVSearch-2.0-fusionIdx-14-CompetitorWebSites
1401unknown robot from AGMLAB Information Technologies (Information retrieval system ?)dodgebot/experimental
1402Unknown robot from Allrati.comAllrati/1.1 (+)
1403Unknown robot from Czech Technical University Prague (147.32.141.xx)Firefox (kastaneta03@hotmail.com)
1404Unknown robot from Czech Technical University Prague (147.32.141.xx)Firefox_1.0.6 (kasparek@naparek.cz)
1405Unknown robot from douban search (China) - maybe image crawlingDoubanbot/1.0 (bot@douban.com http://www.douban.com)
1406Unknown robot from the University of Tokyo (157.82.157.xx)FeedChecker/0.01
1407unknown robot via Level3.netCrawler
1408Unknown spam bot from telekom.com.my (218.111.83.xxx)China Local Browse 2.6
1409Unknown UA looking for Awstats Perl componentsDataCha0s/2.0
1410Unknown user agentdds explorer v1.0 beta
1411Unknown user agent from MicrosoftbCentral Billing Post-Process
1412Unknown Yahoo robot*/Nutch-0.9-dev
1413Unkown robot from Discovery Engine Corp.disco/Nutch-0.9 (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com)
1414Unkown robot from Discovery Engine Corp.disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com)
1415UpflowUpflow/1.0
1416Uptimebot\.orgUptimebot.org - Free website monitoring
1417UptimeRobotMozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
1418urlappendbotMozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)
1419Validator\.nuValidator.nu/LV
1420vebidoobotMozilla/5.0 (compatible; vebidoobot/1.0; +https://blog.vebidoo.de/vebidoobot/
1421VelenPublicWebCrawlerVelenPublicWebCrawler (velen.io)
1422VeoozbotMozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)
1423Vestris robotAlkalineBOT/1.3
1424Vestris robotAlkalineBOT/1.4 (1.4.0326.0 RTM)
1425Vigiltech e-Sense user research robot (website is offline)e-sense 1.0 ea(www.vigiltech.com/esensedisclaim.html)
1426vkShareMozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)
1427voilabotMozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
1428voilabotMozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
1429Voyager - Amiga browserAmigaVoyager/3.4.4 (MorphOS/PPC native)
1430Vulcan Inc. Everest crawler (in development)Everest-Vulcan Inc./0.1 (R&D project; host=e-1-24; http://everest.vulcan.com/crawlerhelp)
1431Vulcan Inc. Everest crawler (in development)Everest-Vulcan Inc./0.1 (R&D project; http://everest.vulcan.com/crawlerhelp)
1432W3C_CSS_ValidatorJigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0
1433W3C_I18n-CheckerW3C_I18n-Checker/1.0
1434W3C_UnicornW3C_Unicorn/1.0
1435W3C_ValidatorW3C_Validator/1.3
1436W3C-checklinkW3C-checklink/4.5 [4.154] libwww-perl/5.823
1437W3C-checklinkW3C-checklink/4.3 [4.42] libwww-perl/5.820
1438W3C-checklinkW3C-checklink/4.2 [4.20] libwww-perl/5.803
1439W3C-checklinkW3C-checklink/2.90 libwww-perl/5.64
1440W3C-checklinkW3C-checklink/4.3 [4.42] libwww-perl/5.805
1441W3C-checklinkW3C-checklink/4.2.1 [4.21] libwww-perl/5.803
1442W3C-checklinkW3C-checklink/4.5 [4.160] libwww-perl/5.823
1443W3C-checklinkW3C-checklink/4.3 [4.42] libwww-perl/5.808
1444W3C-checklinkW3C-checklink/3.6.2.3 libwww-perl/5.64
1445W3C-mobileOKW3C-mobileOK/DDC-1.0
1446Walhello Internet Search robotappie 1.1 (www.walhello.com)
1447Web mining project from CSCI 659 (computer science course) at Indiana Univeritycsci_b659/0.13
1448Webarchive Project Bruinbot crawlerBruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html)
1449WebCollage Syndicator graphics crawler/collectorcollage.cgi/1.xx
1450WebDataStatsMozilla/5.0 (compatible; WebDataStats/1.0 ; +https://webdatastats.com/policy.html)
1451website directory adaxas link checkingAdaxas Spider (http://www.adaxas.net/)
1452WeSEE:SearchWeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
1453WeSEE:SearchWeSEE:Search
1454WhatsAppWhatsApp/2.19.308 A
1455WhatsAppWhatsApp/2.19.244 A
1456WhatsAppWhatsApp/2.19.92 i
1457WhatsAppWhatsApp/2.16.1/i
1458WhatsAppWhatsApp/2.12.555 A
1459WhatsAppWhatsApp/2.12.510 A
1460WhatsAppWhatsApp/2.12.449 A
1461WhatsAppWhatsApp/2.12.15/i
1462WhatsAppWhatsApp/0.3.4941 N
1463WhatsAppWhatsApp/0.3.4679 N
1464WhatsAppWhatsApp/0.3.4479 N
1465WhatsAppWhatsApp/2.16.42 A
1466WhatsAppWhatsApp/2.12.548 A
1467WhatsAppWhatsApp/2.19.330 A
1468WhatsAppWhatsApp/2.19.258 A
1469WhatsAppWhatsApp/2.19.175 A
1470WhatsAppWhatsApp/2.16.57 A
1471WhatsAppWhatsApp/2.16.13 A
1472WhatsAppWhatsApp/2.12.556 A
1473WhatsAppWhatsApp/2.12.540 A
1474WhatsAppWhatsApp/2.12.453 A
1475WhatsAppWhatsApp/2.12.17/i
1476WhatsAppWhatsApp/2.12.16/i
1477WhatsAppWhatsApp
1478WhatsAppWhatsApp/2.16.2/i
1479Wholeweb robotDaviesBot/1.7 (www.wholeweb.net)
1480Wildsoft Germany (closed) Internet client system user-agent (was www.oskarweb.de)Commerce Browser Center
1481woobotwoobot
1482WordupInfoSearchWordupInfoSearch/1.0
1483WoriobotMozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)
1484woriobotMozilla/5.0 (compatible; woriobot +http://worio.com)
1485WoriobotMozilla/5.0 (compatible; woriobot +http://worio.com)
1486wotboxWotbox/2.01 (+http://www.wotbox.com/bot/)
1487wotboxWotbox/2.0 (bot@wotbox.com; http://www.wotbox.com)
1488www\.uptime\.comMozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)
1489Xdefine text search engine robot - based on Egothor open source crawleregothor/3.0a (+http://www.xdefine.org/robot.html)
1490Xenu Link SleuthXenu Link Sleuth/1.3.8
1491xovibotMozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)
1492Xyleme SA France robotcosmos/0.8_(robot@xyleme.com)
1493Xyleme SA France robotcosmos/0.9_(robot@xyleme.com)
1494Y!JMozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html))
1495Y!JMozilla/5.0 (iPhone; Y!J-BRY/YATSH crawler; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
1496Y!JY!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
1497Y!JY!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
1498Y!JY!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
1499Y!JY!J-PSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
1500yacybotyacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
1501yacybotyacybot (/global; amd64 Linux 2.6.32-573.3.1.el6.x86_64; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
1502yacybotyacybot (/global; amd64 Linux 2.6.32-042stab094.8; java 1.7.0_79; America/en) http://yacy.net/bot.html
1503yacybotyacybot (-global; amd64 FreeBSD 9.2-RELEASE-p10; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
1504yacybotyacybot (/global; amd64 FreeBSD 10.3-RELEASE; java 1.8.0_77; GMT/en) http://yacy.net/bot.html
1505yacybotyacybot (-global; amd64 Linux 5.2.11-Jinsol; java 12.0.2; Europe/en) http://yacy.net/bot.html
1506yacybotyacybot (-global; amd64 Linux 5.2.9-Jinsol; java 12.0.2; Europe/en) http://yacy.net/bot.html
1507yacybotyacybot (-global; amd64 Linux 5.2.8-Jinsol; java 12.0.2; Europe/en) http://yacy.net/bot.html
1508yacybotyacybot (-global; amd64 Windows 8 6.2; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
1509yacybotyacybot (-global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
1510yacybotyacybot (-global; amd64 Linux 4.4.0-57-generic; java 9-internal; Europe/en) http://yacy.net/bot.html
1511yacybotyacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
1512yacybotyacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
1513yacybotyacybot (-global; amd64 Linux 3.19.0-15-generic; java 1.8.0_45-internal; Europe/de) http://yacy.net/bot.html
1514yacybotyacybot (/global; amd64 Linux 3.16-0.bpo.2-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
1515yacybotyacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_111; Europe/en) http://yacy.net/bot.html
1516yacybotyacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
1517yacybotyacybot (-global; amd64 Linux 2.6.32-042stab116.1; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
1518yacybotyacybot (-global; amd64 Linux 2.6.32-042stab111.11; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
1519yacybotyacybot (/global; amd64 Linux 2.6.32-042stab108.8; java 1.7.0_91; America/en) http://yacy.net/bot.html
1520yacybotyacybot (/global; amd64 FreeBSD 10.3-RELEASE-p7; java 1.7.0_95; GMT/en) http://yacy.net/bot.html
1521yacybotyacybot (/global; amd64 Linux 2.6.32-042stab093.4; java 1.7.0_65; Etc/en) http://yacy.net/bot.html
1522yacybotyacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
1523yacybotyacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
1524yacybotyacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
1525yacybotyacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
1526yacybotyacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
1527yacybotyacybot (-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
1528yacybotyacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; America/en) http://yacy.net/bot.html
1529yacybotyacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_111; Europe/de) http://yacy.net/bot.html
1530yacybotyacybot (-global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.8.0_111; Europe/de) http://yacy.net/bot.html
1531yacybotyacybot (/global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
1532yacybotyacybot (/global; amd64 Linux 3.14-0.bpo.1-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
1533yacybotyacybot (/global; amd64 Linux 3.13.0-88-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
1534yacybotyacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
1535yacybotyacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
1536yacybotyacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
1537yacybotyacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/de) http://yacy.net/bot.html
1538yacybotyacybot (/global; amd64 Linux 3.13.0-74-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
1539yacybotyacybot (-global; amd64 Linux 3.13.0-61-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
1540yacybotyacybot (/global; amd64 Linux 3.13.0-45-generic; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
1541yacybotyacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
1542yacybotyacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
1543yacybotyacybot (/global; amd64 Linux 3.12.1; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
1544yacybotyacybot (/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; America/en) http://yacy.net/bot.html
1545yacybotyacybot (/global; amd64 Linux 3.10.0-327.22.2.el7.x86_64; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
1546yacybotyacybot (/global; amd64 Linux 3.10.0-229.7.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
1547YahooMozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
1548YahooMozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
1549YahooMozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
1550Yahoo Link PreviewMozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)
1551Yahoo Search Japan robot (203.216.197.xxx)DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html)
1552Yahoo Search Japan robot (203.216.197.xxx)DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html)
1553YaK\/Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com)
1554YandexMozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
1555YandexMozilla/5.0 (compatible; YandexVertis/3.0; +http://yandex.com/bots)
1556YandexMozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)
1557YandexMozilla/5.0 (compatible; YandexMetrika/3.0; +http://yandex.com/bots)
1558YandexMozilla/5.0 (compatible; YandexSitelinks; Dyatel; +http://yandex.com/bots)
1559YandexMozilla/5.0 (compatible; YandexCalendar/1.0; +http://yandex.com/bots)
1560YandexMozilla/5.0 (compatible; YaDirectFetcher/1.0; Dyatel; +http://yandex.com/bots)
1561YandexMozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)
1562YandexMozilla/5.0 (compatible; YandexPagechecker/1.0; +http://yandex.com/bots)
1563YandexMozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)
1564YandexMozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)
1565YandexMozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)
1566YandexMozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
1567YandexMozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)
1568YandexMozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
1569YandexMozilla/5.0 (compatible; YandexDirectDyn/1.0; +http://yandex.com/bots
1570YandexMozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
1571YandexMozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots)
1572YandexMozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)
1573YandexMozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
1574YandexAccessibilityBotMozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots
1575YandexBotMozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
1576YandexImagesMozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
1577YandexMobileBotMozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
1578YangaYanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)
1579yangaYanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)
1580Yelo.no business search (Norway) via FindexaFindexa Crawler (http://www.findexa.no/gulesider/article26548.ece)
1581YetiMozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)
1582YisouSpiderYisouSpider
1583yoozBotMozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; info@yooz.ir)
1584ZabbixZabbix
1585ZD's FavOrg favourites managing programFavOrg
1586Zetbot search Belgium (213.41.128.xx) using Deepindex robotDeepIndex ( http://www.zetbot.com )
1587zgrabMozilla/5.0 zgrab/0.x
1588ZoomBotZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html)
1589ZoominfoBotZoominfoBot (zoominfobot at zoominfo dot com)
1590ZumBotMozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)
1591ZuperlistBot\/Mozilla/5.0 (compatible; ZuperlistBot/1.0)
1592ZuperlistBot\\/Mozilla/5.0 (compatible; ZuperlistBot/1.0)
1593µTorrent BitTorrent ClientBTWebClient/180B(9704)
1594\[ZSEBOT\]ZoomSpider - zoomsearchengine.com [ZSEBOT]
1595!Susie (http://www.sync2it.com/susie)Sync2It bookmark management & clustering engine
1596 UnChaos From Chaos To Order Hybrid Web Search Engine.(vadim_gonchar@unchaos.com)UnCHAOS search robot
1597 UnChaos Bot Hybrid Web Search Engine. (vadim_gonchar@unchaos.com)UnCHAOS search robot
1598 UnChaosBot From Chaos To Order UnChaos Hybrid Web Search Engine at www.unchaos.com (info@unchaos.com)UnCHAOS search robot
1599 http://www.sygol.comSygol Search (Italy) robot
1600( Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi )SearchEngineWorld's robots.txt validator
1601(DreamPassport/3.0; isao/MyDiGiRabi)DreamCast DreamPassport browser
1602(Privoxy/1.0)Privoxy web proxy
1603*/Nutch-0.9-devUnknown Yahoo robot
1604+SitiDi.net/SitiDiBot/1.0 (+Have Good Day)SitiDi.net search (Germany) robot
1605-DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.deDie Kraehe Meta-Search-Engine (Germany) link checking
1606123spider-Bot (Version: 1.02, powered by www.123spider.de123spider.de (Germany) web directory link checking
1607192.comAgent192.com - UK web directory
16081st ZipCommander (Net) - http://www.zipcommander.com/1st ZipCommander Net - IE based browser
16092Bone_LinkChecker/1.0 libwww-perl/5.642Bone online link checker
16104anything.com LinkChecker v2.04Anything robot
16118484 Boston Project v 1.0Unknown guestbook spamming or harvesting tool from diff. IPs
1612:robot/1.0 (linux) ( admin e-mail: undefined http://www.neofonie.de/loesungen/search/robot.html )neofonie search robot Germany
1613A-Online SearchA-Online.at robot - now Jet2Web Search
1614A1 Keyword Research/1.0.2 (+http://www.micro-sys.dk/products/keyword-research/) miggibot/2007.03.27A1 Keyword Research - search engine and keyword optimization software
1615A1 Sitemap Generator/1.0 (+http://www.micro-sys.dk/products/sitemap-generator/) miggibot/2006.01.24MiggiBot website crawler engine - A1 Sitemap Generator
1616aardvark-crawlerAardvark web crawler for Sun's Blog recommendations
1617AbachoBOTAbacho / Crawler.de robot
1618AbachoBOT (Mozilla compatible)Abacho / Crawler.de robot
1619ABCdatos BotLink/5.xx.xxx#BBLABCdatos - Castilian program & tutorial directory
1620Aberja CheckomatAberja Hybridsuchmaschine (Germany) link checking
1621abot/0.1 (abot; http://www.abot.com; abot@abot.com)Nameprotect copyright search robot (24.177.134.x)
1622About/0.1libwww-perl/5.47About robot
1623Accelatech RSSCrawler/0.4Accela Technology RSS feed crawler
1624accoonaAccoona Search robot
1625Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)Accoona Search robot
1626Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)Accoona Search robot
1627Ace ExplorerAce Explorer - IE based browser
1628Ack (http://www.ackerm.com/)Ackerm search robot
1629AcoiRobotAcoi picture finder robot
1630Acoon Robot v1.50.001Acoon.de search (Germany) robot
1631Acoon Robot v1.52 (http://www.acoon.de)Acoon.de search (Germany) robot
1632Acoon-Robot 4.0.x.[xx] (http://www.acoon.de)Acoon.de search (Germany) robot
1633Acoon-Robot v3.xx (http://www.acoon.de and http://www.acoon.com)Acoon.de search (Germany) robot
1634Acorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot org)Acorn Search Project
1635ActiveBookmark 1.xLibMaster.com Active Bookmark HTML page creator
1636ActiveworldsActiveworlds 3D homepage browser
1637ActiveWorlds/3.xx (xxx)Activeworlds 3D homepage browser
1638Ad Muncher v4.xx.xAd Muncher - banner killer
1639Ad Muncher v4x Build xxxxxAd Muncher - banner killer
1640Adaxas Spider (http://www.adaxas.net/)website directory adaxas link checking
1641Advanced Browser (http://www.avantbrowser.com)Avant Browser - IE based browser
1642AESOP_com_SpiderManAesop robot
1643agadine/1.x.x (+http://www.agada.de)Agada search (Germany) robot
1644Agent-SharewarePlazaFileCheckBot/2.0+(+http://www.SharewarePlaza.com)SharewarePlaza link checking
1645AgentName/0.1 libwww-perl/5.48Linkomatic submission verifier
1646AIBOT/2.1 By +(www.21seek.com A Real artificial intelligence search engine China)21seek.com (China) robot (218.17.90.xxx)
1647AideRSS/1.0 (aiderss.com)AideRss - Postrank RSS and Blog filtering
1648aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com)Nameprotect copyright search robot (24.177.134.x)
1649aipbot/2-beta (aipbot dev; http://aipbot.com; aipbot@aipbot.com)Nameprotect copyright search robot (24.177.134.x)
1650Akregator/1.2.9; librss/remnantsAkregator news feed reader for KDE
1651Aladin/3.324Aladin robot
1652Alcatel-BG3/1.0 UP.Browser/5.0.3.1.2Phone.com UP.Browser for mobiles on Alcatel cellphone
1653Aleksika Spider/1.0 (+http://www.aleksika.com/)Aleksika Danmark - Search engine optimization spider
1654AlertInfo 2.0 (Powered by Newsbrain)Alertinfo - French version of Feedreader 3.xx
1655AlkalineBOT/1.3Vestris robot
1656AlkalineBOT/1.4 (1.4.0326.0 RTM)Vestris robot
1657Allesklar/0.1 libwww-perl/5.46Allesklar.de robot
1658Alligator 1.31 (www.nearsoftware.com)Alligator download manager
1659Allrati/1.1 (+)Unknown robot from Allrati.com
1660AltaVista Intranet V2.0 AVS EVAL search@freeit.comAltavista robot
1661AltaVista Intranet V2.0 Compaq Altavista Eval sveand@altavista.netAltavista robot
1662AltaVista Intranet V2.0 evreka.com crawler@evreka.comAltavista robot
1663AltaVista V2.0B crawler@evreka.comAltavista robot
1664amaya/x.xx libwww/x.x.xAmaya - W3C's Editor/Browser
1665AmfibiBOTAmfibi Search robot
1666Amfibibot/0.06 (Amfibi Web Search; http://www.amfibi.com; agent@amfibi.com)Amfibi Search robot
1667Amfibibot/0.07 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)Amfibi Search robot
1668amibotamibot - Amidalla search engine robot (62.241.33.xx)
1669Amiga-AWeb/3.4.167SEAWeb Amiga browser
1670AmigaVoyager/3.4.4 (MorphOS/PPC native)Voyager - Amiga browser
1671AmiTCP Miami (AmigaOS 2.04)Amiga Miami TCP Stack
1672Amoi 8512/R21.0 NF-Browser/3.3NF embedded browser on Amois Skypephone
1673amzn_assocAmazon.com robot for checking their affiliate sites
1674AnnoMille spider 0.1 alpha - http://www.annomille.itAnnomille Italian historical oriented robot
1675annotate_google; http://ponderer.org/download/annotate_google.user.jsannotate Google - Firefox extension for annotating Google search results
1676Anonymized by ProxyOS: http://www.megaproxy.comMegaproxy user
1677Anonymizer/1.1faked user agent
1678AnswerBus (http://www.answerbus.com/)AnswerBus natural language search using COLLATE technology
1679AnswerChase PROve x.0AnswerChase search tool
1680AnswerChase x.0AnswerChase search tool
1681ANTFresco/x.xxANT Fresco Browser
1682antibot-V1.1.5/i586-linux-2.2Antibot (discontinued) robot
1683AnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au;Engine)Anzwers (Yahoo) Australia robot
1684Apexoo Spider 1.xApexoo Search spider
1685Aplix HTTP/1.0.1JavaOS app. for SEGA Saturn Internet and Sanyo Internet-TV
1686Aplix_SANYO_browser/1.x (Japanese)JavaOS app. for Sanyo Internet-TV
1687Aplix_SEGASATURN_browser/1.x (Japanese)JavaOS app. for SEGA Saturn Internet
1688AportAport robot
1689appie 1.1 (www.walhello.com)Walhello Internet Search robot
1690Apple iPhone v1.1.4 CoreMedia v1.0.0.4A102CoreMedia player on Apple iPhone
1691Apple-PubSub/65.1.1PubSub - Mac OS X utility for managing RSS/Atom subscriptions via the PubSub framework
1692ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby.com;)Araby search - Arabia
1693ArachBotCovac Arachnid Web Crawler
1694Arachnoidea (arachnoidea@euroseek.com)Euroseek spider
1695aranhabotAmazon.com robot for checking their affiliate sites
1696ArchitextSpiderExcite spider
1697archive.org_botHeritrix - The Internet Archive's open-source crawler (207.241.225.2xx)
1698Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com)Simpy Bookmarklet crawler (69.55.233.xx)
1699Arikus_SpiderArikus inContext search engine software
1700Arquivo-web-crawler (compatible; heritrix/1.12.1 +http://arquivo-web.fccn.pt)Tomba project: the Portuguese web archive
1701ASAHA Search Engine Turkey V.001 (http://www.asaha.com/)Asaha search robot (Turkey)
1702Asahina-Antenna/1.xASAHINA Antenna information detecting agent
1703Asahina-Antenna/1.x (libhina.pl/x.x ; libtime.pl/x.x)ASAHINA Antenna information detecting agent
1704ask.24x.infoAsk 24x Info robot
1705AskAboutOil/0.06-rcp (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@askaboutoil.com)Ask About Oil - Petroleum related search (24.227.212.xxx) using Nutch
1706asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com)askEd! / Inferret search (Japan) robot using Nutch
1707ASPSeek/1.2.5ASPSeek search engine software -Yahoo-Inc. / Telecom Canada robot
1708ASPseek/1.2.9dSwsoft.net robot using Aspseek
1709ASPSeek/1.2.xASPSeek search engine software
1710ASPSeek/1.2.xaASPSeek search engine software
1711ASPseek/1.2.xxASPSeek search engine software
1712ASPSeek/1.2.xxpreASPSeek search engine software
1713ASSORT/0.10Associative Sort robot
1714asterias/2.0Singingfish media spider (64.12.186.2xx) via AOL search
1715AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html)Atlocal local business search robot
1716Atomic_Email_Hunter/4.0Atomic Email Hunter email extracing and harvesting
1717Atomz/1.0Atomz robot
1718atSpider/1.0atSpider (ceased) email harvester / spambot
1719Attentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; info@attentio.com)Attentio social media monitoring and analysing
1720AU-MIC/2.0 MMP/2.0Samsung SPH-A660 phone with Sprint software
1721AUDIOVOX-SMT5600Audiovox SMT5600 (AT&T) Smartphone mobile phone browser
1722augurfindAugurnet Swiss (was www.augurnet.ch) search robot
1723augurnfind V-1.xAugurnet Swiss (was www.augurnet.ch) search robot
1724autoemailspiderAuto Email Pro Email harvester
1725autohttpLinkscan tool from Elsop
1726autowebdir 1.1 (www.autowebdir.com)Autowebdir - The Automatically Generated Web Directory
1727AV Fetch 1.0Altavista robot ??
1728Avant Browser (http://www.avantbrowser.com)Avant Browser - IE based browser
1729AVSearch-1.0(peter.turney@nrc.ca)National Research Council Canada robot
1730AVSearch-2.0-fusionIdx-14-CompetitorWebSitesUnknown robot from 205.203.108.xx (telerate.com)
1731AVSearch-3.0(AltaVista/AVC)Altavista robot
1732AWebAWeb Amiga browser
1733axadine/ (Axadine Crawler; http://www.axada.de/; )Axada search Germany robot
1734AxmoRobot - Crawling your site for better indexing on www.axmo.com search engine.Axmo search robot
1735Azureus 2.x.x.xAzureus Java BitTorrent Client
1736BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)Babaloo search robot (Slovenia)
1737BaboomBot/1.x.x (+http://www.baboom.us)BaBoom Web Portal (ODP) robot (66.98.254.xx)
1738BackStreet Browser 3.xBackStreet Browser - Offline browser / website downloader
1739BaiduImagespider+(+http://www.baidu.jp/search/s308.html)Baidu search (Japan) image crawler
1740BaiDuSpiderBaidu spidering engine - used by diff. IPs
1741Baiduspider+(+http://help.baidu.jp/system/05.html)Baidu spidering engine - used by diff. IPs
1742Baiduspider+(+http://www.baidu.com/search/spider.htm)Baidu spidering engine - used by diff. IPs
1743Baiduspider+(+http://www.baidu.com/search/spider_jp.html)Baidu search (Japan) crawler
1744Balihoo/Nutch-1.0-dev (Crawler for Balihoo.com search engine - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com)Balihoo - Vertical search engine crawler (beta)
1745BanBots/1.2 (spider@banbots.com)Project BanBots Perl script robot
1746Barca/2.0.xxxxBarca Pro email & PIM software
1747BarcaPro/1.4.xxxxBarca Pro email & PIM software
1748BarraHomeCrawler (albertof@barrahome.org)Barrahome crawler
1749bCentral Billing Post-ProcessUnknown user agent from Microsoft
1750bdcindexer_2.6.2 (research@bdc)Business.com robot
1751BDFetchBrandimensions Brand Protection robot
1752BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html) (X11; I; Linux 2.0.44 i686)Bdncentral Sitesearch robot
1753BeamMachine/0.5 (dead link remover of www.beammachine.net)beammachine web directory (Germany) link checking
1754beautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/)Beauty robot for Cosmoty - German beauty and wellness search
1755BebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html )A Passion for Jazz music related search robot
1756BeebwareDirectory/v0.01LinkcheckerBeepware (site is down) web directory link checking
1757Big Brother (http://pauillac.inria.fr/~fpottier/)Big Brother link checking tool
1758Big Fish v1.0GoonGee.com link popularity checking
1759BigBrother/1.6eBB4 network monitoring
1760BigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com; bot@bigclique.com)BigClique Search robot
1761BIGLOTRON (Beta 2;GNU/Linux)Biglotron search (France) robot
1762Bigsearch.ca/Nutch-x.x-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com)Bigsearch.ca search robot
1763Bilbo/2.3b-UNIXBilbo - web frontend for the Nessus Security Scanner
1764BilgiBetaBot/0.8-dev (bilgi.com (Beta) ; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)Bilgi.com (Beta) search robot - Turkey
1765BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com)Bilgi.com (Beta) search robot - Turkey
1766billbot wjj@cs.cmu.eduCarnegie Mellon School robot/link checking ?
1767Bitacle bot/1.1Bitacle Blog Search Archive robot
1768Bitacle Robot (V:1.0;) (http://www.bitacle.com)Bitacle Blog Search Archive robot
1769Biyubi/x.x (Sistema Fenix; G11; Familia Toledo; es-mx)Biyubi Navigator - Mexican browser for Fenix OS
1770BlackBerry7520/4.0.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/5.0.3.3 UP.Link/5.1.2.12 (Google WAP Proxy/1.0)Blackberry Wireless Internet browser via Google WAP Proxy
1771BlackWidowFS Consulting (was www.fsconsult.net) Black Widow web crawler
1772BlackWidowBlackWidow web site scanner / downloading tool
1773Blaiz-Bee/1.0 (+http://www.blaiz.net)Blaiz Enterprises RawGrunt search
1774Blaiz-Bee/2.00.8222 (BE Internet Search Engine http://www.rawgrunt.com)Blaiz Enterprises RawGrunt search
1775Blaiz-Bee/2.00.xxxx (+http://www.blaiz.net)Blaiz Enterprises RawGrunt search
1776BlitzBOT@tricus.netBlitzsuche Germany robot
1777BlitzBOT@tricus.net (Mozilla compatible)Blitzsuche Germany robot
1778BlockNote.NetBlockNote web page editor
1779BlogBot/1.xblogdex robot from MIT.edu
1780BlogBridge 2.13 (http://www.blogbridge.com/)BlogBridge RSS reader
1781Bloglines Title Fetch/1.0 (http://www.bloglines.com)Bloglines article search
1782Bloglines-Images/0.1 (http://www.bloglines.com)Bloglines graphics crawler
1783Bloglines/3.1 (http://www.bloglines.com)Bloglines news crawler
1784BlogMap (http://www.feedmap.net)FeedMap / BlogMap geo coding service
1785Blogpulse (info@blogpulse.com)Intelliseek's BlogPulse blog search
1786BlogPulseLive (support@blogpulse.com)Intelliseek's BlogPulse blog search
1787BlogSearch/1.x +http://www.icerocket.com/IceRocket Web search robot
1788blogsearchbot-pumpkin-3Art of Computing blog search project
1789BlogsNowBot, V 2.01 (+http://www.blogsnow.com/)BlogsNow realtime link tracker robot
1790BlogVibeBot-v1.1 (spider@blogvibe.nl)BlogVipe news and Blog crawler (Netherlands)
1791blogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWatcher/)blogWatcher robot from Okumura Group Tokyo (131.112.182.xxx)
1792BlogzIce/1.0 (+http://icerocket.com; rhodes@icerocket.com)IceRocket Web search robot
1793BlogzIce/1.0 +http://www.icerocket.com/IceRocket Web search robot
1794BloobyBotBlooby search (beta) robot
1795Bloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com)Balihoo - Search Engine for Advertising Media
1796bluefish 0.6 HTML editorBluefish HTML-editor for Linux
1797BMCLIENTPart of ButtMan remote access tool
1798BMLAUNCHERBookmark Express bookmark manager
1799Bobby/4.0.x RPT-HTTPClient/0.3-3EBobby web accessibility desktop testing tool
1800boitho.com-dc/0.xx (http://www.boitho.com/dcbot.html)Boitho search (Norway) robot via 80.202.212.xx / 80.80.111.xx
1801boitho.com-robot/1.xBoitho search (Norway) robot via 80.202.212.xx / 80.80.111.xx
1802boitho.com-robot/1.x (http://www.boitho.com/bot.html)Boitho search (Norway) robot via 80.202.212.xx / 80.80.111.xx
1803Bookdog/x.xBookdog - Mac bookmark manager
1804Bookmark Buddy bookmark checker (http://www.bookmarkbuddy.net/)Bookmark Buddy - favorite bookmark manager
1805Bookmark Renewal Check Agent [http://www.bookmark.ne.jp/]Favourites managing program
1806Bookmark Renewal Check Agent [http://www.bookmark.ne.jp/] (Version 2.0beta)Favourites managing program
1807BookmarkBase(2/;http://bookmarkbase.com)Bookmark Base bookmark manager
1808Bot mailto:craftbot@yahoo.comcybercity.fr user robot / faked user agent ?
1809BPImageWalker/2.0 (www.bdbrandprotect.com)BD-Brandprotect copyright infringement crawler
1810BravoBrian bstop.bravobrian.itBravoBrian bSTOP parental control
1811BravoBrian SpiderEngine MarcoPoloRobot for BravoBrian bSTOP
1812BrightCrawler (http://www.brightcloud.com/brightcrawler.asp)BrightCloud web filtering for classifying websites
1813BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html) Webarchive Project Bruinbot crawler
1814BSDSeek/1.0Inktomi (Hotbot-Lycos NBCi) robot
1815BStop.BravoBrian.it Agent DetectorBravoBrian bSTOP parental control
1816BTbot/0.x (+http://www.btbot.com/btbot.html)BitTorrent Search Engine btbot robot
1817BTWebClient/180B(9704)µTorrent BitTorrent Client
1818BuildCMS crawler (http://www.buildcms.com/crawler)BuildCMS crawler - market monitoring project of BuildCMS
1819Bulkfeeds/r1752 (http://bulkfeeds.net/)Bulkfeeds: RSS directory link checking
1820BullsEyeBullsEye/Intelliseek robot
1821bumblebee@relevare.comRelevare Portal software robot
1822BunnySlippersMicrosoft server information robot (see link)
1823BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawler@burstfind.com)BurstFind search crawler (64.34.172.xx)
1824Buscaplus Robi/1.0 (http://www.buscaplus.com/robi/)Buscaplus (Spain) robot
1825BW-C-2.0Logitech Desktop Managers (LDM) Backweb (BW) update check
1826bwh3_user_agentBasic Web Hacking 3 fake user-agent from Hellbound Hackers challenges
1827Cabot/Nutch-0.9 (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; agent@amfibi.com)Amfibi Search robot
1828Cabot/Nutch-1.0-dev (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; agent@amfibi.com)Amfibi Search robot
1829CamelHttpStream/1.0Evolution integrated mail solution Camel TCP stream class
1830Cancer Information and Support International;Some user agent
1831carleson/1.0Cosmix project crawler (204.14.48.x / 38.113.234.xxx)
1832Carnegie_Mellon_University_Research_WebBOT-->PLEASE READ-->http://www.andrew.cmu.edu/~brgordon/webbot/index.html http://www.andrew.cmu.edu/~brgordon/webbot/index.htmlCarnegie Mellon University WebBOT
1833Carnegie_Mellon_University_WebCrawler http://www.andrew.cmu.edu/~brgordon/webbot/index.htmlCarnegie Mellon University WebBOT
1834Catall SpiderCatall.de search & web directory (Germany)
1835CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com)UIUCs Cazoodle search based on MetaQuerier
1836CCBot/1.0 (+http://www.commoncrawl.org/bot.html)CommonCrawl Foundation search crawler
1837ccubee/x.xEmpyreum Ccubee (Czech) search engine solution
1838CDR/1.7.1 Simulator/0.7(+http://timewe.net) Profile/MIDP-1.0 Configuration/CLDC-1.0Timewe mobile browser (WAP) simulator (Japan)
1839CE-PreloadCisco Content Engine
1840CentiverseBotNordic semantic search engine
1841CentiverseBot - investigatorNordic semantic search engine
1842CentiverseBot/3.0 (http://www.centiverse-project.net)Nordic semantic search engine
1843Ceramic Tile Installation Guide (http://www.floorstransformed.com)Floortransformed.com robot (link ckecking ??)
1844CERN-LineMode/2.15CERN Line Mode Browser
1845cfetch/1.0Cosmix project crawler (204.14.48.x / 38.113.234.xxx)
1846CFNetwork/x.xMaxOS X CoreFoundation CFNetwork API
1847cg-eye interactivecg-eye CGI checker
1848Charon/1.x (Amiga)Charon Amiga download manager
1849Chat Catcher/1.0Chat Catcher blog monitoring robot
1850Checkbot/1.xx LWP/5.xxCheckbot link validation
1851CheckLinks/1.x.xChecklinks - Perl link checker
1852CheckUrlNTL user agent
1853CheckWebCheckWeb link validation
1854Chilkat/1.0.0 (+http://www.chilkatsoft.com/ChilkatHttpUA.asp)Chilkat HTTP component user-agent
1855China Local Browse 2.6Unknown spam bot from telekom.com.my (218.111.83.xxx)
1856Chitika ContentHit 1.0Chitika Inc. Blog advertising
1857ChristCRAWLER 2.0Christcentral.com Christcrawler (was www.christcrawler.com)
1858CHttpClient by Open Text CorporationCHttpClient - C++ class using WinInet
1859CipinetBot (http://www.cipinet.com/bot.html)CipinetBot -Cipinet Search Engine Web Crawler
1860Cityreview Robot (+http://www.cityreview.org/crawler/)Cityreview regional search (Germany) link checking
1861CJ Spider/Commision Junction link checking spider
1862CJB.NET ProxyCJB Net anonymous socks proxy service (216.194.70.x)
1863ClariaBot/1.0Claria (ex Gator) SearchScout robot (64.152.73.xx)
1864Claymont.comClaymont Search robot
1865CloakDetect/0.9 (+http://fulltext.seznam.cz/)Seznam Search (Czech Republic) robot
1866Clushbot/2.x (+http://www.clush.com/bot.html)Clush clustered search robot
1867Clushbot/3.x-BinaryFury (+http://www.clush.com/bot.html)Clush clustered search robot
1868Clushbot/3.xx-Ajax (+http://www.clush.com/bot.html)Clush clustered search robot
1869Clushbot/3.xx-Hector (+http://www.clush.com/bot.html)Clush clustered search robot
1870Clushbot/3.xx-Peleus (+http://www.clush.com/bot.html)Clush clustered search robot
1871COAST WebMaster Pro/4.x.x.xx (Windows NT)COAST Webmaster - Web management and maintenance software
1872CoBITSProbeProposed Content-Based Image Tracking System (CoBITS) P2P crawler
1873Cocoal.icio.us/1.0 (v36) (Mac OS X; http://www.scifihifi.com/cocoalicious)Cocoa del.icio.us (social bookmarks manager) client for Mac OS X
1874Cogentbot/1.X (+http://www.cogentsoftwaresolutions.com/bot.html)Cogent Search Bot from Cogent Software Solutions for unknown purposes
1875ColdFusionCold Fusion server used by various IPs i.e.: - NetWORLD web catalogue link checking
1876ColdFusion (BookmarkTracker.com)Cold Fusion server used by Bookmark Tracker - online favourites managment
1877collage.cgi/1.xxWebCollage Syndicator graphics crawler/collector
1878combine/0.0Combine harvesting & indexing robot
1879Combine/2.0 http://combine.it.lth.se/Combine harvesting & indexing robot
1880Combine/3 http://combine.it.lth.se/Combine harvesting & indexing robot
1881Combine/x.0Combine harvesting & indexing robot
1882cometrics-bot, http://www.cometrics.decometrics Web Content Mining solution - Germany
1883Commerce Browser CenterWildsoft Germany (closed) Internet client system user-agent (was www.oskarweb.de)
1884complex_network_group/Nutch-0.9-dev (discovering the structure of the world-wide-web; http://cantor.ee.ucla.edu/~networks/crawl; nimakhaj@gmail.com)UCLA Complex Networks Groups Complex Network Analysis
1885Computer_and_Automation_Research_Institute_Crawler crawler@ilab.sztaki.huHungarian Academy of Sciences data mining search
1886Comrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)ComRite Chinese Search Engine for Oversea Web Sites (69.248.26.xx)
1887Contactunknown
1888ContactBot/0.2Probably E-Mail harvesting robot - same as LMQueueBot
1889ContentSmartzContentSmartz e-mail harvesting tools
1890contypeInternet Explorer versions 4.x and 5 plugin content
1891Convera Internet Spider V6.xConveras RetrievalWare Internet Spider (63.241.61.x)
1892ConveraCrawler/0.2Converas RetrievalWare Internet Spider (63.241.61.x)
1893ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)Converas RetrievalWare Internet Spider (63.241.61.x)
1894ConveraMultiMediaCrawler/0.1 (+http://www.authoritativeweb.com/crawl)Converas RetrievalWare Internet Spider (63.241.61.x)
1895CoolBotSuchmaschine21 (Germany) robot
1896Cooliris/1.5 CFNetwork/459 Darwin/10.0.0d3Cooliris photo and video browser
1897CoralWebPrx/0.1.1x (See http://coralcdn.org/)Coral Content Distribution Network
1898cosmos/0.8_(robot@xyleme.com)Xyleme SA France robot
1899cosmos/0.9_(robot@xyleme.com)Xyleme SA France robot
1900CoteoNutchCrawler/Nutch-0.9 (info [at] coteo [dot] com)Coteo.com - local French directory link checking
1901CougarSearch/0.x (+http://www.cougarsearch.com/faq.shtml)Cougarsearch.com robot
1902Covac TexAs ArachbotCovac Arachnid Web Crawler
1903CoverScout%203/3.0.1 CFNetwork/339.5 Darwin/9.5.0 (i386) (iMac5,1)CoverScout for iTunes - CD cover search tool
1904Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)Naver Japan / Korea robot
1905Cowbot-0.1.x (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)Naver Japan / Korea robot
1906CrawlConvera0.1 (CrawlConvera@yahoo.com)Converas RetrievalWare Internet Spider
1907Crawlerunknown robot via Level3.net
1908Crawler (cometsearch@cometsystems.com)Cometsystems Comet Search robot via Findwhat (now Miva)
1909Crawler admin@crawler.deCrawler.de / Abacho robot
1910Crawler V 0.2.x admin@crawler.deCrawler.de / Abacho robot
1911crawler@alexa.comAlexa crawler
1912CrawlerBoy Pinpoint.comPinpoint WAP search robot
1913Crawllybot/0.1 (Crawllybot; +http://www.crawlly.com; crawler@crawlly.com)Crawlly Beta search - Germany (72.232.194.2xx)
1914CreativeCommons/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)Creatice Commons using Nutch open source robot
1915Cricket-A100/1.0 UP.Browser/6.3.0.7 (GUI) MMP/2.0Cricket A100 cell phone browser
1916CrocCrawler vx.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)Croccrawler robot
1917csci_b659/0.13Web mining project from CSCI 659 (computer science course) at Indiana Univerity
1918CSE HTML Validator Professional (http://www.htmlvalidator.com/)CSE HTML Validator for Windows
1919Cuam Ver0.050bxCuam - IE based browser
1920Cuasarbot/0.9b http://www.cuasar.com/spider_beta/Cuasar (Spain) music / ringtone search spider
1921curl/7.10.x (i386-redhat-linux-gnu) libcurl/7.10.x OpenSSL/0.9.7a ipv6 zlib/1.1.4Curl file transferring tool
1922curl/7.7.x (i386--freebsd4.3) libcurl 7.7.x (SSL 0.9.6) (ipv6 enabled)Curl file transferring tool
1923curl/7.8 (i686-pc-linux-gnu) libcurl 7.8 (OpenSSL 0.9.6)Curl file transferring tool
1924curl/7.9.x (win32) libcurl 7.9.xCurl file transferring tool
1925CurryGuide SiteScan 1.1CurryGuide UK link check robot
1926Custo x.x (www.netwu.com)Custo web site spidering tool (link checking)
1927Custom Spider www.bisnisseek.com /1.0Bisnisseek (was www.bisnisseek.com) robot
1928Cyberdog/2.0 (Macintosh; 68k)Cyberdog Mac Browser (was www.cyberdog.org)
1929CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)CyberPatrol LLC robot for web filtering software
1930CyberSpyder Link Test/2.1.12 (admin@mspennyworth.com)CyberSpyder Link Test software
1931CydralSpider/1.x (Cydral Web Image Search; http://www.cydral.com)Cydral image & site search spider
1932CydralSpider/3.0 (Cydral Image Search; http://www.cydral.com)Cydral image & site search spider
1933DA 3.5 (www.lidan.com)Downloadaccelerator download manager
1934DA 4.0Downloadaccelerator download manager
1935DA 4.0 (www.downloadaccelerator.com)Downloadaccelerator download manager
1936DA 5.0Downloadaccelerator download manager
1937DA 7.0Downloadaccelerator download manager
1938DAP x.xDownload Accelerator Plus download manager
1939Dart Communications PowerTCPPowerTCP ActiveX control tool
1940DataCha0s/2.0Unknown bot from Kornet Korea (218.149.129.xxx) scans for Perl Awstats
1941DataCha0s/2.0Unknown UA looking for Awstats Perl components
1942DataFountains/DMOZ DownloaderINFOMINE/iVia Scholary Internet Resource Collections robot
1943DataFountains/Dmoz Downloader (http://ivia.ucr.edu/useragents.shtml)INFOMINE/iVia Scholary Internet Resource Collections robot
1944DataFountains/DMOZ Feature Vector Corpus Creator (http://ivia.ucr.edu/useragents.shtml)INFOMINE/iVia Scholary Internet Resource Collections robot
1945DataparkSearch/4.47 (+http://dataparksearch.org/bot)DataparkSearch open source search engine
1946DataparkSearch/4.xx (http://www.dataparksearch.org/)DataparkSearch open source search engine
1947DataSpear/1.0 (Spider; http://www.dataspear.com/spider.html; spider@dataspear.com)DataSpear Directory robot (24.109.29.xx)
1948DataSpearSpiderBot/0.2 (DataSpear Spider Bot; http://dssb.dataspear.com/bot.html; dssb@dataspear.com)DataSpear Directory robot (24.109.29.xx)
1949DatenBot( http://www.sicher-durchs-netz.de/bot.html)Sicher-durchs-Netz German security related search (62.75.220.xxx)
1950DaviesBot/1.7 (www.wholeweb.net)Wholeweb robot
1951daypopbot/0.xDaypop blog - weblog - online mag search spider
1952dbDig(http://www.prairielandconsulting.com)dbDig search engine
1953DBrowse 1.4bSome site scanning tool via diff. IPs i.e.: - wanweb.net (208.6.163.xxx) - cox.net (68.4.xxx.xxx)
1954DBrowse 1.4dSome site scanning tool via diff. IPs i.e.: - pacbell.net (67.112.xxx.xxx)
1955DC-Sakura/x.xxDC-Sakura download manager
1956dCSbot/1.1unknown divine/Openmarket.com robot
1957DDDsome (website) downloading tool
1958dds explorer v1.0 betaUnknown user agent
1959de.searchengine.comBot 1.2 (http://de.searchengine.com/spider)Searchengine.com (Germany) submission checking / robot (84.73.57.xx)
1960DeadLinkCheck/0.4.0 libwww-perl/5.xxDLC Perl HTTP link checking
1961Deep Link Calculator v1.0Sootle Web Directory deep link checker
1962deepak-USC/ISIdeepak-USC/ISI robot from USC/Information Science Institute
1963DeepIndexDeepindex robot
1964DeepIndex ( http://www.zetbot.com )Zetbot search Belgium (213.41.128.xx) using Deepindex robot
1965DeepIndex (www.en.deepindex.com)Deepindex robot
1966DeepIndexer.caDeepindex robot (via Paragon.net Canada)
1967del.icio.us-thumbnails/1.0 Mozilla/5.0 (compatible; Konqueror/3.4; FreeBSD) KHTML/3.4.2 (like Gecko)del.icio.us picture robot for thumbnail preview via Yahoo
1968DeleGate/9.0.5-fix1DeleGate application level gateway / proxy server
1969Demo Bot DOT 16bSome site scanning tool from 217.34.59.xxx (btopenworld.com)
1970Demo Bot Z 16bSome site scanning tool from 68.154.96.xx (bellsouth.net)
1971Denmex websearch (http://search.denmex.com)Denmex Websearch robot/link checking
1972Der große BilderSauger 2.00uDataBecker Bilder Sauger (discontinued) web graphics downloader
1973dev-spider2.searchpsider.com/1.3bSearchspider.com robot (72.245.225.xxx)
1974DevComponents.com HtmlDocument ObjectDevComponents HTMLDocument Class Library for Visual Studio.net
1975DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html)DiaGem Japan web crawler
1976Diamond/x.0Claria (ex Gator) SearchScout robot (64.152.73.xx)
1977DiamondBotClaria (ex Gator) SearchScout robot (64.152.73.xx)
1978Digger/1.0 JDK/1.3.0rc3Diggit! robot
1979DigOut4UOpenPortal4U robot
1980DIIbot/1.2Findsame.com (site is offline) / Digital-Integrity robot
1981Dillo/0.8.5-i18n-miscDillo Web Browser
1982Dillo/0.x.xDillo Web Browser
1983disastrous/1.0.5 (running with Python 2.5.1; http://www.bortzmeyer.org/disastrous.html; archangel77@del.icio.us)disastrous - a del.icio.us link checker based on Python
1984DISCo Pump x.xDISCo Pump offline browser / website ripper
1985disco/Nutch-0.9 (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com)Unkown robot from Discovery Engine Corp.
1986disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com)Unkown robot from Discovery Engine Corp.
1987DittoSpyderDitto picture search robot
1988dlmansome download agent
1989dloader(NaverRobot)/1.0Naver Japan / Korea robot
1990DNSRight.com WebBot Link Ckeck Tool. Report abuse to: dnsr@dnsright.comDNS Right - Online DNS tools
1991DoCoMo/1.0/Nxxxi/c10NTT DoCoMo (Japan) robot
1992DoCoMo/1.0/Nxxxi/c10/TBNTT DoCoMo (Japan) robot
1993DoCoMo/1.0/P502i/c10 (Google CHTML Proxy/1.0)Google (216.239.39.x) proxy server
1994DoCoMo/2.0 P900iV(c100;TB;W24H11)NTT DoCoMo (Japan) robot
1995DoCoMo/2.0 SH901iS(c100;TB;W24H12),gzip(gfe) (via translate.google.com)NTT DoCoMo (Japan) proxy server (210.136.161.1xx)
1996DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html)Yahoo Search Japan robot (203.216.197.xxx)
1997DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html)Yahoo Search Japan robot (203.216.197.xxx)
1998DocZilla/1.0 (Windows; U; WinNT4.0; en-US; rv:1.0.0) Gecko/20020804DocZilla - Mozilla-based SGML/XML/HTML- browser
1999dodgebot/experimentalunknown robot from AGMLAB Information Technologies (Information retrieval system ?)
2000DonutP; Windows98SEDonut P - Japanese IE based browser
2001Doubanbot/1.0 (bot@douban.com http://www.douban.com)Unknown robot from douban search (China) - maybe image crawling
2002Download Demon/3.x.x.xDownload Demon/Netzip download manager
2003Download Druid 2.xDownload Druid IE plugin download manager
2004Download Express 1.0Download Express download manager
2005Download MasterDownload Master download manager
2006Download Ninja 3.0Download Ninja download manager (Japan)
2007Download WonderDownload Wonder download manager
2008Download-Tipp Linkcheck (http://download-tipp.de/)Download-Tipp Germany link checking
2009Download.exe(1.1) (+http://www.sql-und-xml.de/freeware-tools/)download.exe .NET based downloading tool
2010DownloadDirect.1.0Download Direct download manager
2011Dr.Web (R) online scanner: http://online.drweb.com/Dr.WEB online virus scanner
2012Dragonfly File ReaderDragonfly CMS - Open Source content management system
2013Drecombot/1.0 (http://career.drecom.jp/bot.html)Drecom Japan (210.233.67.xxx) - Blog search ??
2014Drupal (+http://drupal.org/)Drupal - open source content management platform
2015DSurf15a 01Some site scanning tool via diff. IPs i.e.: - cox.net (68.5.xxx.xxx) - pacbell.net (64.16x.xxx.xxx)
2016DSurf15a 71Some site scanning tool via diff. IPs i.e.: - cox.net (68.4.xxx.xxx)
2017DSurf15a 81Some site scanning tool via diff. IPs i.e.: - verizon.net (4.47.xxx.xxx)
2018DSurf15a VASome site scanning tool via diff. IPs i.e.: - eastlink.ca (24.222.xxx.xxx) - cogeco.net (216.221.8x.xxx)
2019DTAAgentDTAAgent Java object for data collecting
2020dtSearchSpiderdt Search Spider software
2021Dual ProxyFourelle Venturi proxy server
2022DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)Duck Duck Go search crawler
2023Dumbot(version 0.1 beta - dumbfind.com)DumbFind.com robot
2024Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html)DumbFind.com robot
2025Dumbot(version 0.1 beta)DumbFind.com robot
2026e-sense 1.0 ea(www.vigiltech.com/esensedisclaim.html)Vigiltech e-Sense user research robot (website is offline)
2027e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/)e-Society Project (Japan) crawler (133.9.238.xx)
2028eApolloBot/2.0 (compatible; heritrix/2.0.0-SNAPSHOT-20071024.170148 +http://www.eapollo-opto.com)Global Opto's eApollo Flash based search engine - Taiwan
2029EARTHCOM.info/1.x [www.earthcom.info]Earthcom (Czech Republic) search robot (194.108.39.xx)
2030EARTHCOM.info/1.xbeta [www.earthcom.info]Earthcom (Czech Republic) search robot (194.108.39.xx)
2031EasyDL/3.xxKeywen Encyclopedia Bot
2032EasyDL/3.xx http://keywen.com/Encyclopedia/BotKeywen Encyclopedia Bot
2033EBrowse 1.4bSome site scanning tool via diff. IPs i.e.: - swbell.net (65.66.xxx.xxx)
2034eCatch/3.0eCatch (now Wysigot) offline browser
2035EchO!/2.0Echo.fr robot
2036Educate Search VxBSome site scanning tool via diff. IPs i.e.: - cox.net (68.4.xxx.xxx)
2037egothor/3.0a (+http://www.xdefine.org/robot.html)Xdefine text search engine robot - based on Egothor open source crawler
2038EgotoBot/4.8 (+http://www.egoto.com/about.htm)Egoto Search robot
2039ejupiter.comeJupiter searcg robot (206.191.49.xx)
2040EldoS TimelyWeb/3.xTimelyWeb web page monitoring tool
2041elfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/)Elftales crawler for uChoose theme based search (Germany)
2042ELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net/aboutkr.html)DAUMOA - RSS search robot of Daum
2043ELinks (0.x.x; Linux 2.4.20 i586; 132x60)ELinks text mode browser
2044ELinks/0.x.x (textmode; NetBSD 1.6.2 sparc; 132x43)ELinks text mode browser
2045EmailSiphonSonic E-mail collector
2046EmailSpiderEmailSpider E-mail harvesting software
2047EmailWolf 1.00Trellian EMailWolf E-mail collector
2048EmeraldShield.com WebBotEmeraldShield spam and web filtration services
2049EmeraldShield.com WebBot (http://www.emeraldshield.com/webbot.aspx)EmeraldShield spam and web filtration services
2050EMPAS_ROBOTEmpas search Korea robot (220.95.22x.xxx)
2051EnaBot/1.x (http://www.enaball.com/crawler.html)Enabot - Enaball semantic search project crawler
2052endo/1.0 (Mac OS X; ppc i386; http://kula.jp/endo)endo - Mac news site and blog aggregator
2053Enfish TrackerEnfish Personal search tool
2054Enterprise_Search/1.0Enterprise Search engine software (64.202.165.xxx)
2055Enterprise_Search/1.0.xxxEnterprise Search engine software (64.202.165.xxx)
2056Enterprise_Search/1.00.xxx;MSSQL (http://www.innerprise.net/es-spider.asp)Enterprise Search engine software (64.202.165.xxx)
2057envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php)Envolk Web Search robot
2058envolk[ITS]spider/1.6(+http://www.envolk.com/envolkspider.html)Envolk Web Search robot
2059EroCrawlerEroCrawler adult search robot
2060ES.NET_Crawler/2.0 (http://search.innerprise.net/)Enterprise Search engine software (64.202.165.xxx)
2061eseek-larbin_2.6.2 (crawler@exactseek.com)ExactSEEK (Jayde Online) robot
2062ESISmartSpiderESI Smart-Spider toolkit
2063eStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0)e-Style ISP search (Russia) robot (217.174.103.xxx)
2064ESurf15a 15Some site scanning tool via diff. IPs
2065EuripBot/0.x (+http://www.eurip.com) GetFileEurip.com - European Internet Portal robot
2066EuripBot/0.x (+http://www.eurip.com) GetRobotsEurip.com - European Internet Portal robot
2067EuripBot/0.x (+http://www.eurip.com) PreCheckEurip.com - European Internet Portal robot
2068Eurobot/1.0 (http://www.ayell.eu)Ayell Euronet business directory robot
2069EvaalSE - bot@evaal.comEvaal Search Engine robot
2070eventax/1.3 (eventax; http://www.eventax.de/; info@eventax.de)Eventax event search (Germany)
2071Everest-Vulcan Inc./0.1 (R&D project; host=e-1-24; http://everest.vulcan.com/crawlerhelp)Vulcan Inc. Everest crawler (in development)
2072Everest-Vulcan Inc./0.1 (R&D project; http://everest.vulcan.com/crawlerhelp)Vulcan Inc. Everest crawler (in development)
2073Exabot-Images/1.0Exalead Websearch image crawler (193.47.80.xx)
2074Exabot-Test/1.0Exalead (France) search robot (193.47.80.xx)
2075Exabot/2.0Exalead (France) search robot (193.47.80.xx)
2076Exabot/3.0Exalead (France) search robot (193.47.80.xx)
2077ExactSearcheXact Search Bar for IE
2078ExactSeek Crawler/0.1ExactSEEK (Jayde Online) robot
2079exactseek-crawler-2.63 (crawler@exactseek.com)ExactSEEK (Jayde Online) robot
2080exactseek-pagereaper-2.63 (crawler@exactseek.com)ExactSEEK (Jayde Online) robot
2081exactseek.comExactSEEK (Jayde Online) robot (69.9.181.1xx)
2082Exalead NG/MimeLive Client (convert/http/0.120)Exalead (France) search robot (193.47.80.xx)
2083Excalibur Internet Spider V6.5.4Excalibur (now Convera) spider software
2084Execrawl/1.0 (Execrawl; http://www.execrawl.com/; bot@execrawl.com)Execrawl software search using nutch
2085exooba crawler/exooba crawler (crawler for exooba.com; http://www.exooba.com/; info at exooba dot com)exooba crawler for exooba search pre-alpha development
2086exooba/exooba crawler (exooba; exooba)exooba crawler for exooba search pre-alpha development
2087ExperimentalHenrytheMiragoRobotMirago UK Robot
2088Expired Domain SleuthExpired Domain Sleuth domain name tool
2089Express WebPictures (www.express-soft.com)Express Web Pictures image browser
2090ExtractorProExtractor Pro e-mail collector
2091Extreme Picture FinderExisoftware image grabber and downloading tool
2092EyeCatcher (Download-tipp.de)/1.0Download-Tipp Germany robot
2093Factbot 1.09 (see http://www.factbites.com/webmasters.php)Factbites search robot
2094factbot : http://www.factbites.com/robotsFactbites search robot
2095FaEdit/2.0.xFaEdit Professional - Japanese bookmark manager
2096FairAd ClientFairAd user
2097FANGCrawl/0.01Safe-t.net web filtering service
2098FARK.com link verifierDrew Curtis' FARK.com link checking
2099Fast Crawler Gold EditionFast/Alltheweb crawler (66.151.181.xx)
2100FAST Enterprise Crawler 6 (Experimental)Fast/Alltheweb crawler (66.151.181.xx)
2101FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com/srsapp/contactus/Fast Enterprise Crawler (66.151.181.xx) for Scirus scienctific information search
2102FAST Enterprise Crawler 6 used by Cobra Development (admin@fastsearch.com)Fast/Alltheweb crawler (66.151.181.xx)
2103FAST Enterprise Crawler 6 used by Comperio AS (sts@comperio.no)Comperio Web Miner based on Fast ESP
2104FAST Enterprise Crawler 6 used by FAST (FAST)Fast/Alltheweb crawler (66.151.181.xx)
2105FAST Enterprise Crawler 6 used by Pages Jaunes (pvincent@pagesjaunes.fr)Pages Jaunes business search (France) robot using Fast Enterprise Crawler
2106FAST Enterprise Crawler 6 used by Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au)Fast/Alltheweb crawler for Sensis.com.au Australian search (66.151.181.xx)
2107FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg)SPH Search - Singapore related search using Fast crawler
2108FAST Enterprise Crawler 6 used by WWU (wardi@uni-muenster.de)FAST Enterprise Crawler used by WWU (University of Muenster - Germany)
2109FAST Enterprise Crawler/6 (www.fastsearch.com)Fast/Alltheweb crawler (66.151.181.xx)
2110FAST Enterprise Crawler/6.4 (helpdesk at fast.no)Fast/Alltheweb crawler (66.151.181.xx)
2111FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0)Fast/Alltheweb crawler (66.151.181.xx)
2112FAST MetaWeb Crawler (helpdesk at fastsearch dot com)Fast/Alltheweb crawler (66.151.181.xx)
2113Fast PartnerSite CrawlerFast/Alltheweb crawler (66.151.181.xx)
2114FAST-WebCrawler/2.2.10 (Multimedia Search) (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)Fast/Alltheweb multimedia crawler
2115FAST-WebCrawler/2.2.6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)Fast/Alltheweb crawler
2116FAST-WebCrawler/2.2.7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)http://www.fast.noFast/Alltheweb crawler
2117FAST-WebCrawler/2.2.8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)http://www.fast.noFast/Alltheweb crawler
2118FAST-WebCrawler/3.2 testFast/Alltheweb crawler
2119FAST-WebCrawler/3.3 (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)Fast/Alltheweb crawler
2120FAST-WebCrawler/3.4/Nirvana (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)Fast/Alltheweb crawler
2121FAST-WebCrawler/3.4/PartnerSite (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)Fast/Alltheweb crawler
2122FAST-WebCrawler/3.5 (atw-crawler at fast dot no; http://fast.no/support.php?c=faqs/crawler)Fast/Alltheweb crawler
2123FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)Fast/Alltheweb crawler
2124FAST-WebCrawler/3.6/FirstPage (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)Fast/Alltheweb crawler
2125FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)Fast/Alltheweb crawler
2126FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)Fast/Alltheweb crawler
2127FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)Fast/Alltheweb crawler
2128FAST-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)Fast/Alltheweb crawler
2129FAST-WebCrawler/3.x MultimediaFast/Alltheweb multimedia crawler
2130FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)Fast/Alltheweb multimedia crawler
2131fastbot crawler beta 2.0 (+http://www.fastbot.de)Fastbot search Germany crawler (80.252.104.1xx)
2132FastBug http://www.ay-up.comAy-Up geo sync search robot
2133FastCrawler 3.0.1 (crawler@1klik.dk)Fast/Alltheweb crawler (66.151.181.xx)
2134FastSearch Web Crawler for Verizon SuperPages (kevin.watters@fastsearch.com)Fast/Alltheweb crawler used by SuperPages.com
2135Favcollector/2.0 (info@favcollector.com http://www.favcollector.com/)Favcollector Favicon collecting robot
2136FavIconizerFavIconizer - IE favorites icons refreshing tool
2137favo.eu crawler/0.6 (http://www.favo.eu)favo.eu (Germany) search robot
2138FavOrgZD's FavOrg favourites managing program
2139Favorites Checking (http://campulka.net)Campulka.net Favorites checking tool
2140Favorites Sweeper v.2.03Favorites Sweeper bookmark checker
2141Faxobot/1.0FaXo Search robot (69.152.89.xx)
2142FDM 1.xFree Download Manager (FDM) download accelerator
2143FDM 2.xFree Download Manager (FDM) download accelerator
2144Feed Seeker Bot (RSS Feed Seeker http://www.MyNewFavoriteThing.com/fsb.php)RSS Feed Seeker bot (68.225.95.2xx)
2145Feed24.comFeed24 news feed and blog search
2146Feed::Find/0.0xFeed::Find - Syndication feed (RSS/Atom) auto-discovery
2147Feedable/0.1 (compatible; MSIE 6.0; Windows NT 5.1)Feedable beta web based RSS service
2148FeedChecker/0.01Unknown robot from the University of Tokyo (157.82.157.xx)
2149FeedDemon/2.7 (http://www.newsgator.com/; Microsoft Windows XP)FeedDemon RSS reader
2150Feedfetcher-Google-iGoogleGadgets; (+http://www.google.com/feedfetcher.html)Google news feed feetcher for iGoogle gadgets
2151Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)Google Feedfetcher - RSS and Atom feed crawler
2152FeedForAll rss2html.php v2FeedForAll RSS feed robot
2153FeedHub FeedDiscovery/1.0 (http://www.feedhub.com)FeedHub news feed personalization engine powered by mSpoke
2154FeedHub MetaDataFetcher/1.0 (http://www.feedhub.com)FeedHub news feed personalization engine powered by mSpoke
2155Feedjit Favicon Crawler 1.0Feedjit news feed service favicon crawler
2156Feedreader 3.xx (Powered by Newsbrain)Newsbrain Feedreader3
2157Feedshow/x.0 (http://www.feedshow.com; 1 subscriber)FeedShow online RSS feed reader
2158FeedshowOnline (http://www.feedshow.com)FeedShow online RSS feed reader
2159FeedZcollector v1.x (Platinum) http://www.feeds4all.com/feedzcollectorFeedZcollector - Feed (RSS, ATOM and RDF) capturing software
2160Felix - Mixcat Crawler (+http://mixcat.com)MixCat robot
2161fetch libfetch/2.0FreeBSD download tool
2162FFC Trap Door SpiderFrequent Finders spider via Sitefusion.com
2163Filangy/0.01-beta (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)Filangy search and bookmark service
2164Filangy/1.0x (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; filangy-agent@filangy.com)Filangy search and bookmark service
2165Filangy/1.0x (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)Filangy search and bookmark service
2166fileboost.net/1.0 (+http://www.fileboost.net)File Boost Network link checking
2167FileHound x.xFileHound download manager
2168Filtrbox/1.0filtrbox media content (news) monitoring
2169FindAnISP.com_ISP_Finder_v99aFind An ISP robot
2170Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece)Yelo.no business search (Norway) via Findexa
2171findlinks/x.xxx (+http://wortschatz.uni-leipzig.de/findlinks/) NextLinks - German vocabulary and hyperlink search
2172FineBotFinesearch robot
2173Finjan-prefetchFinjan Vital Security Web Appliance security solution
2174Firefly/1.0Fireball.de robot
2175Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)Fireball.de robot
2176Firefox (kastaneta03@hotmail.com)Unknown robot from Czech Technical University Prague (147.32.141.xx)
2177Firefox_1.0.6 (kasparek@naparek.cz)Unknown robot from Czech Technical University Prague (147.32.141.xx)
2178FirstGov.gov Search - POC:firstgov.webmasters@gsa.govAT&T/Fast Search robot for FirstGov (U.S.Government) portal
2179firstsbotFirstsfind Germany robot / link checking
2180Flapbot/0.7.2 (Flaptor Crawler; http://www.flaptor.com; crawler at flaptor period com)Flaptor information retrieval solutions robot
2181FlashGetJetCar/FlashGet download manager
2182FLATARTS_FAVICOFlatArts Favorites Icon Tool
2183Flexum spiderFlexum.ru search service
2184Flexum/2.0Flexum.ru search service
2185FlickBot 2.0 RPT-HTTPClient/0.3-3DivX.com Movie Find robot
2186flunkyMetacarta / Cogent robot
2187fly/6.01 libwww/4.0Dunknown
2188flyindex.net 1.0/http://www.flyindex.netFLY Index Metasearch link checking
2189FnooleBot/2.5.2 (+http://www.fnoole.com/addurl.html)Fnoole news crawler
2190FocusedSampler/1.0IBM's Almaden Research robot
2191Folkd.com Spider/0.1 beta 1 (www.folkd.com)folkd.com social search robot
2192FollowSite Bot ( http://www.followsite.com/bot.html )FollowSite robot - website monitoring
2193FollowSite.com ( http://www.followsite.com/b.html )FollowSite robot - website monitoring
2194Fooky.com/ScorpionBot/ScoutOut; http://www.fooky.com/scorpionbotsFooky search Scorpionbots robot (65.12.170.xxx)
2195Francis/1.0 (francis@neomo.de http://www.neomo.de/)Neomo Search (Germany) robot (85.10.197.1xx)
2196Franklin Locator 1.8Some spam bot
2197free-downloads.net download-link validator /0.1Free Downloads shareware directory link checking
2198FreeFind.com-SiteSearchEngine/1.0 (http://freefind.com; spiderinfo@freefind.com)FreeFind.com robot
2199Frelicbot/1.0 +http://www.frelic.com/Frelics backlink checking bot (beta)
2200FreshDownload/x.xxFresh Download download manager
2201FreshNotes crawler< report problems to crawler-at-freshnotes-dot-comFreshNotes - music related artist search (72.3.225.xx)
2202FSurf15a 01Some site scanning tool via diff. IPs
2203FTB-Bot http://www.findthebest.co.uk/Find the Best search robot
2204Full Web Bot 0416BSome site scanning tool from diff. IPs i.e.: - 66.28.240.xx (cogentco.com) - 68.5.174.xx (cox.net)
2205Full Web Bot 0516BSome site scanning tool i.e. from - 68.154.96.xx (bellsouth.net)
2206Full Web Bot 2816BSome site scanning tool from 66.255.6.xxx (uslec.com)
2207FuseBulb.ComFuseBulb search
2208FyberSpider (+http://www.fybersearch.com/fyberspider.php)FyberSearch FyberSpider robot
2209GagglebotSome user from bbnplanet.net (4.63.218.2xx) using an Innerprise robot tool
2210GAIS Robot/1.0B2Seed Search robot
2211Gaisbot/3.0 (indexer@gais.cs.ccu.edu.tw; http://gais.cs.ccu.edu.tw/robot.php)Gaislab Taiwan robot
2212Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)Gaislab Taiwan robot
2213GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)Galaxy robot (63.121.41.xxx)
2214Gallent Search Spider v1.4 Robot 2 (http://robot.GallentSearch.com)Gallent Search directory (UK)
2215gamekitbot/1.0 (+http://www.uchoose.de/crawler/gamekitbot/)Gamekit game search engine - Germany
2216Gamespy_ArcadeGameSpyHTTP/1.0
2217GammaSpider/1.0GammaWare GammaSpider
2218gazz/x.x (gazz@nttrd.com)nttrd.com / Infobee.ne.jp robot
2219geckobotGeckobot user robot
2220Generic Mobile Phone (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)Google Mobile Search crawler
2221generic_crawler/01.0217/Unknown robot from Carnegie Mellon University (128.2.211.xxx)
2222GenesisBrowser (HTTP 1.1; 0.9; XP SP2; .NET CLR 2.0.50727)Lunascape Genesis browser
2223genieBot (http://64.5.245.11/faq/faq.html)GenieKnows.com search
2224geniebot wgao@genieknows.comGenieKnows.com search
2225GeoBot/1.0Unknown robot from wavepath.com (65.254.33.1xx)
2226GeonaBot 1.x; http://www.geona.com/Geona Search robot / link checking
2227geourl/2.0b2GeoURL ICBM Address Server - a location-to-URL reverse directory
2228GeoURLBot 1.0 (http://geourl.org)GeoURL ICBM Address Server - a location-to-URL reverse directory
2229GetBotGetbot web downloading tool / site grabber
2230GetRight/3.x.xGetRight download manager
2231GetRight/4.5xxGetRight download manager
2232GetRight/4.xGetRight download manager
2233GetRight/4.x[a-e]GetRight download manager
2234GetRight/6.1 (Pro)GetRight download manager
2235GetRightPro/6.0beta2GetRight download manager
2236GetWeb/0.1 libwww-perl/5.16GetWeb - web page to email service
2237GhostRouteHunter/20021130 (https://www.sixxs.net/tools/grh/; info@sixxs.net)Sixxs Ghost Route Hunter
2238gigabaz/3.1x (baz@gigabaz.com; http://gigabaz.com/gigabaz/)GigaBaz Brainbot (Germany) robot
2239Gigabot/2.0 (gigablast.com)Gigablast robot (64.62.168.xx)
2240Gigabot/2.0/gigablast.com/spider.htmlGigablast robot
2241Gigabot/2.0; http://www.gigablast.com/spider.htmlGigablast robot
2242Gigabot/2.0attGigablast robot
2243Gigabot/3.0 (http://www.gigablast.com/spider.html)Gigablast robot
2244Gigabot/x.0Gigablast robot (64.62.168.xx)
2245GigabotSiteSearch/2.0 (sitesearch.gigablast.com)Gigablast robot (64.62.168.xx)
2246GNODSPIDER (www.gnod.net)www.gnod.net spider
2247Go!Zilla 3.x (www.gozilla.com)Go!Zilla download manager
2248Go!Zilla/4.x.x.xxGo!Zilla download manager
2249Go-Ahead-Got-It/1.1GotIt web accelerator (discontinued)
2250Goblin/0.9 (http://www.goguides.org/)GoGuides.Org (195.226.137.xx) robot
2251Goblin/0.9.x (http://www.goguides.org/goblin-info.html)GoGuides.Org (195.226.137.xx) robot
2252GoForIt.comGoForIt Search robot
2253GOFORITBOT ( http://www.goforit.com/about/ )GoForIt Search robot
2254GoGuides.Org Link CheckGoGuides.org directory & search link checking
2255GoldenFeed Spider 1.0 (http://www.goldenfeed.com)GoldenFeed.com - RSS search engine
2256Goldfire ServerInvention Machines Goldfire Server
2257gonzo1[P] +http://www.suchen.de/popups/faq.jspsuchen.de German local search robot
2258gonzo2[P] +http://www.suchen.de/faq.htmlsuchen.de German local search robot
2259Goofer/0.2Some private robot (Wanadoo.fr client)
2260Google TalkGoogle instant messenger
2261googlebot (larbin2.6.0@unspecified.mail)Packard Bell Net user robot (*not* Google)
2262Googlebot-Image/1.0Google image crawler (66.249.72.xxx)
2263Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html)Google image crawler (66.249.72.xxx)
2264Googlebot/2.1 ( http://www.google.com/bot.html)Google robot 66.249.64.XXX
2265Googlebot/2.1 ( http://www.googlebot.com/bot.html)Google robot 66.249.64.XXX
2266Googlebot/Test ( http://www.googlebot.com/bot.html)Google robot 66.249.64.XXX
2267Gordon's Spider/Nutch-0.9 (http://www.sharethis.com; gordon@sharethis.com)ShareThis social networking service via Amazon Web Services
2268GrapeFX/0.3 libwww/5.4.0Grapeshot web search system API
2269great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com)Flatland Industries vertical search solution
2270GreatNews/1.0GreatNews 1.0 Beta RSS reader
2271GreenBrowserGreenBrowser - IE based browser (China)
2272gridwell (http://search.gridwell.com)search gridwell favicon display
2273GrigorBot 0.8 (http://www.grigor.biz/bot.html)Grigor Search bot
2274Gromit/1.0Australasian Legal Information Institute (AustLII) robot
2275grub crawler(http://www.grub.org)Grub open source crawler
2276grub-clientGrub open source crawler
2277gsa-crawler (Enterprise; GID-01422; jplastiras@google.com)Google Search Appliance robot (216.239.xx.xx)
2278gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com)Google Search Appliance robot (216.239.xx.xx)
2279gsa-crawler (Enterprise; GIX-02057; dm@enhesa.com)Google Enterprise Search Appliance used by Enhesa (212.35.100.1xx)
2280gsa-crawler (Enterprise; GIX-03519; cknuetter@stubhub.com)Google Enterprise Search Appliance used by IBM (129.41.20.1xx)
2281gsa-crawler (Enterprise; GIX-0xxxx; enterprise-training@google.com)Google Search Appliance robot (216.239.xx.xx)
2282GSiteCrawler/v1.xx rev. xxx (http://gsitecrawler.com/)GSiteCrawler - Google sitemap generator for Windows
2283Guestbook Auto SubmitterGuestbook spamming tool
2284Gulliver/1.3Northernlight robot
2285Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)Yuntis Collaborative Web Resource Categorization and Ranking Project robot
2286Gungho/0.08004 (http://code.google.com/p/gungho-crawler/wiki/Index)Gungho - Extensible web crawler written in Perl by Google Code
2287GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html)guruji : the Indian search engine robot
2288GurujiImageBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)guruji : the Indian search engine picture crawler
2289Haier-T10C/1.0 iPanel/2.0 WAP2.0 (compatible; UP.Browser/6.2.2.4; UPG1; UP/4.0; Embedded)Openwave Mobile Browser on Haier T10C mobile
2290HappyFunBot/1.1Happy Fun Search robot
2291Harvest-NG/1.0.2Harvest-NG web crawler used by search.yahoo.com
2292Haste/0.12 (HOME: http://haste.kytoon.com/)Haste - web mapping and monitoring system
2293Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot)Hatena::Antenna Japan robot
2294Hatena Mobile Gateway/1.0Hatena Japan proxy for handheld/mobile clients
2295Hatena Pagetitle Agent/1.0Hatena Japan robot
2296Hatena RSS/0.3 (http://r.hatena.ne.jp)Hatena Japan RSS feed robot
2297HatenaScreenshot/1.0 (checker)Hatena::Diary (Japan) web page screenshot robot
2298hbtronix.spider.2 -- http://hbtronix.de/spider.phphbtronix.spider - Domain name spider (Germany)
2299HeinrichderMiragoRobotMirago Germany robot
2300HeinrichderMiragoRobot (http://www.miragorobot.com/scripts/deinfo.asp)Mirago Germany robot
2301Helix/1.x ( http://www.sitesearch.ca/helix/)Helix - The SiteSearch (Canada) web crawler
2302HenriLeRobotMirago (http://www.miragorobot.com/scripts/frinfo.asp)Mirago France robot
2303HenrytheMiragoRobotMirago search (UK) robot
2304HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp)Mirago search (UK) robot
2305hgrepurl/1.0O'Reilly's Perl LWP example client program from Web Client Programming with Perl
2306Hi! I'm CsCrawler my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss2005/googlespam/crawler.html RPT-HTTPClient/0.3-3University of Kassel Germany CsCrawler using the HTTPClient library
2307HiDownloadHiDownload download manager
2308Hippias/0.9 BetaHippias robot
2309HitListPilot Hitlist web analytics solution
2310Hitwise Spider v1.0 http://www.hitwise.comHitwise spider
2311HLoaderdiff. IPs / unknown services
2312holmes/3.11 (http://morfeo.centrum.cz/bot)Morfeo / Centrum Search (Czech Republic) robot from 65.102.46.xxx
2313holmes/3.9 (onet.pl)Onet.pl (Poland) search robot
2314holmes/3.xx (OnetSzukaj/5.0; +http://szukaj.onet.pl)Onet.pl (Poland) search robot
2315holmes/x.xMorfeo / Centrum Search (Czech Republic) robot from 65.102.46.xxx
2316HolmesBot (http://holmes.ge)Holes search robot (Georgia)
2317HomePageSearch(hpsearch.uni-trier.de)HomePageSearch robot
2318Homerbot: www.homerweb.comHomerweb search robot
2319Honda-Search/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; search@honda-search.com)Honda-Search.com - Honda cars related search robot
2320HooWWWer/2.1.3 (debugging run) (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-infohiit.fi)HooWWer - Next Generation Information Retrieval robot
2321HooWWWer/2.1.x ( http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-infohiit.fi)HooWWer - Next Generation Information Retrieval robot (128.214.112.xx)
2322HotJava/1.0.1/JRE1.1.xHotJava browser plus HTML Component 1.1.x
2323Hotzonu/x.0Hotzuno - Japanese BBS reader client
2324HPL/Nutch-0.9 -Unknown robot from HP Labs
2325htdig/3.1.6 (http://computerorgs.com)COMPUTERorgs.com robot (205.134.190.xxx) using htdig
2326htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer)htdig used by the Academie de Toulouse
2327htdig/3.1.x (root@localhost)htdig search tool
2328Html Link Validator (www.lithopssoft.com)Lithops Software link validation tool
2329HTML2JPG Blackbox, http://www.html2jpg.comHTML2JPG webpage to image converter
2330HTML2JPG EnterpriseHTML2JPG webpage to image converter
2331HTMLParser/1.xHTML Parser Java library to parse HTML
2332HTTP RetrieverPHP HTTP client to access Web servers
2333http://Anonymouse.org/ (Unix)Anonymous web proxy service
2334http://Ask.24x.Info/ (http://narres.it/)Ask 24x Info (Germany) DMOZ related robot
2335http://hilfe.acont.de/bot.html ACONTBOTACONTBOT - Acont search Germany robot
2336http://OzySoftware.com/Index.htmlOzySoftware.com software directory link checking
2337http://www.almaden.ibm.com/cs/crawlerIBM's Almaden Research robot
2338http://www.almaden.ibm.com/cs/crawler [rc1.wf.ibm.com]IBM's Almaden Research robot
2339http://www.almaden.ibm.com/cs/crawler [wf216]IBM's Almaden Research robot
2340http://www.istarthere.com_spider@istarthere.comIstarthere.com search robot
2341http://www.monogol.deMonogol - German open source search engine project (195.226.167.1xx)
2342http://www.trendtech.dk/spider.asp)TrendTech Search Engine (Denmark) robot
2343HTTP::Lite/2.x.xHTTP::Lite - Standalone Perl module for retreiving HTTP documents
2344HTTPEyesHTTPEyes - Web proxy cache
2345HTTPResume v. 1.xHTTPResume Amiga download manager
2346httpunit/1.5HttpUnit - Java test code for emulating browser behaviour
2347httpunit/1.xHttpUnit - Java browser behavior simulation tool
2348Hybrid/1.2 [en] (OS Independent)Hybrid Share mono C#/Gtk# application for file sharing
2349HyperEstraier/1.x.xxHyper Estraier full-text search system
2350i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search.com)i1search robot
2351IAArchiver-1.0Alexa / The Internet Archive (209.237.238.1xx)
2352iaskspiderUnknown robot (reads robots.txt) from chinatelecom (219.142.78.xx)
2353iaskspider2 (iask@staff.sina.com.cn)Iask search / Sina portal robot (China)
2354ia_archiverAlexa / The Internet Archive (209.237.238.1xx)
2355ia_archiver-web.archive.orgAlexa / The Internet Archive (209.237.238.1xx)
2356ia_archiver/1.6Alexa / The Internet Archive (209.237.238.1xx)
2357IBrowse/2.2 (AmigaOS 3.5)IOSpirit iBrowse Amiga Browser
2358IBrowse/2.2 (Windows 3.1)IOSpirit iBrowse Amiga Browser
2359iCab/2.5.2 (Macintosh; I; PPC)iCab MAC Web browser
2360ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(at)ml(dot)nict(dot)go(dot)jp)Knowledge Clustered Group ICC-Crawler (University of Tokyo - Japan)
2361ICC-Crawler(Mozilla-compatible;http://kc.nict.go.jp/icc/crawl.html;icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp)Knowledge Clustered Group ICC-Crawler (University of Tokyo - Japan)
2362iCCrawler (http://www.iccenter.net)ICJobs - Intelligence Competence Center (Germany) robot
2363ICCrawler - ICjobs (http://www.icjobs.de/bot.htm)ICJobs - Intelligence Competence Center (Germany) robot
2364ICE Browser/5.05 (Java 1.4.0; Windows 2000 5.0 x86)ICE Java browser
2365ichiro/x.0 (http://help.goo.ne.jp/door/crawler.html)Goo Japan / Inktomi robot (210.173.179.xx)
2366ichiro/x.0 (ichiro@nttr.co.jp)Goo Japan / Inktomi robot (210.173.179.xx)
2367IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html)Iconsurf.com - Visual Surf Engine / favicon finder
2368IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html)Iconsurf.com - Visual Surf Engine / favicon finder
2369ICOO Loader v.x.x.xicooLoader download manager
2370ICRA_label_spider/x.0ICRA (Internet Content Rating Association) label spider
2371icsbot-0.1ICS Robot Search Engine (International Christian school of Seoul)
2372IDAInternet Download Accelerator
2373ideare - SignSite/1.xJanas (Ideare.com / Tiscali.it) robot
2374iearthworm/1.0, iearthworm@yahoo.com.cnUnknown UA from Yahoo China
2375IEFav172FreeSome bookmark manager
2376iFeed.jp/2.0 (www.psychedelix.com/agents/agents.rss; 0 subscribers)iFeed.jp - online rss aggregator (in development)
2377igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html)Igde search (Russia) robot
2378iGetter/1.x (Macintosh;G;PPC)iGetter download manager
2379iGetter/2 (Macintosh; U; PPC Mac OS X; en)iGetter download manager
2380IIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net; pvvpr at iiit dot ac dot in)Webkhoj - Indian language search engine
2381ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. For more information please visit http://www.ilial.com/crawler; http://www.ilial.com/crawler; crawl@ilial.com)Ilial Knowledge Search robot
2382ilial/Nutch-0.9-devUnknown robot from UCLA using Nutch
2383IlseBot/1.xIlse Netherlands robot (62.69.178.xx)
2384IlTrovatore-Setaccio ( http://www.iltrovatore.it)Il Trovatore - Italian search engine robot
2385Iltrovatore-Setaccio/0.3-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)Il Trovatore - Italian search engine robot
2386IlTrovatore-Setaccio/1.2 ( http://www.iltrovatore.it/aiuto/faq.html)Il Trovatore - Italian search engine robot
2387Iltrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)Il Trovatore - Italian search engine robot
2388iltrovatore-setaccio/1.2-dev (spidering; http://www.iltrovatore.it/aiuto/.....)Il Trovatore - Italian search engine robot
2389IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it)Il Trovatore - Italian search engine robot
2390ImageVisu/v4.x.xImageVisu image and graphics viewer - display files from the Web (HTTP and ECWP)
2391ImageWalker/2.0 (www.bdbrandprotect.com)BD-Brandprotect copyright infringement crawler
2392Incutio HttpClient v0.xHttpClient - a PHP Web Client Class
2393IncyWincy data gatherer(webmaster@loopimprovements.comIncyWincy search engine using DMOZ Open Directory database
2394IncyWincy page crawler(webmaster@loopimprovements.comIncyWincy search engine using DMOZ Open Directory database
2395IncyWincy(http://www.look.com)Look.com robot using IncyWincy search engine
2396IncyWincy(http://www.loopimprovements.com/robot.html)IncyWincy search engine using DMOZ Open Directory database
2397IncyWincy/2.1(loopimprovements.com/robot.html)IncyWincy search engine using DMOZ Open Directory database
2398IndexTheWeb.com Crawler7Index the Web (69.57.134.xx) crawler
2399Industry Program 1.0.xSpam bot from diff. IPs
2400Inet libraryInet Library Resource Center robot
2401InetURL/1.0InetURL IVM (phone software) plugin for web server access ?
2402info@pubblisito.com- (http://www.pubblisito.com) il Sud dei Motori di RicercaPubblisito.com search - Italia
2403Infoaxe./Nutch-0.9Infoaxe - search history and bookmark service
2404infoConveraCrawler/0.8 ( http://www.authoritativeweb.com/crawl)Converas RetrievalWare Internet Spider (63.241.61.x)
2405InfoFly/1.0 (http://www.versions-project.org/)Versions-project.org Ingelin spider
2406InfoLink/1.xInfoLink link checking tool
2407INFOMINE/8.0 AddersINFOMINE Scholary Internet Resource Collection crawler
2408INFOMINE/8.0 RemoteServicesINFOMINE Scholary Internet Resource Collection crawler
2409INFOMINE/8.0 VLCrawler (http://infomine.ucr.edu/useragents)INFOMINE Scholary Internet Resource Collection crawler
2410InfoNaviRobot(F107)164.71.1.1xx jp.co.fujitsu.t2 Robot
2411InfoSeek Sidewinder/0.9Infoseek robot
2412InfoSeek Sidewinder/1.0AInfoseek robot
2413InfoSeek Sidewinder/1.1AInfoseek robot
2414Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX)Infoseek robot
2415Infoseek SideWinder/2.0B (Linux 2.4 i686)Infoseek Japan robot
2416INGRID/3.0 MT (webcrawler@NOSPAMexperimental.net; http://webmaster.ilse.nl/jsp/webmaster.jsp)Ilse Netherlands robot (62.69.178.xx)
2417Inktomi SearchInktomi (Hotbot-Lycos NBCi etc.) robot
2418InnerpriseBot/1.0 (http://www.innerprise.com/)Enterprise Search engine software (64.202.165.xxx)
2419Insitor.com search and find world wide!Insitor Search robot (80.67.20.1xx)
2420InsitornautInsitor Search robot (80.67.20.1xx)
2421InstallShield DigitalWizarddownload manager
2422integrity/1.6Integrity - website broken link checker for MAC OSx
2423Intelix/0.x (cs; http://www.microton.cz/intelix/; microton@@microton.cz)Microton Intelix robot for Eurotran translation software ?
2424Interarchy/x.x.x (InterarchyCrawler)Interarchy file transfer software - SFTP/FTP client for Mac OS X
2425Internet Ninja x.0Dream Train (Japan) Internet search robot
2426InternetArchive/0.8-dev(Nutch;http://lucene.apache.org/nutch/bot.html;nutch-agent@lucene.apacheHeritrix - The Internet Archive's open-source crawler based on Nutch (207.241.225.2xx)
2427InternetLinkAgent/3.1Internet Link Agent - link checking tool
2428InternetSeer.comInternetseer Web site monitoring / Claymont robot
2429intraVnews/1.xintraVNews - Feed reader & RSS aggregator for Outlook
2430IOI/2.0 (ISC Open Index crawler; http://index.isc.org/; bot@index.isc.org)Internet Open Index crawler using Nutch
2431IP*Works! V5 HTTP/S Component - by /n software - www.nsoftware.comIP*Works! HTTP Component
2432http://www.ip2location.comIP2Location - Reverse lookup geographical data and ISP by IP
2433IP2MapBot/1.1 http://www.ip2map.comIP2Map - geographical IP mapping
2434IPiumBot laurion(dot)comLaurions Ipium robot
2435IpselonBot/0.xx-beta (Ipselon; http://www.ipselon.com; ipselonbot@ipselon.com)Ipselon Web Search robot
2436Iria/1.xxaIria download manager
2437IRLbot/1.0 ( http://irl.cs.tamu.edu/crawler)IRL-crawler - Texas A&M University research project crawler
2438IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler/)IRL-crawler - Texas A&M University research project crawler
2439IrssiUrlLog/0.2url_log - Irssi Perl url grabber
2440Irvine/1.x.xIrvine downloading tool
2441ISC Systems iRc Search 2.1Unknown spambot / harvester from diff. IPs
2442iSiloX/4.xx Windows/32iSiloX document converter for iSilo reader
2443isurf (tszhu@canada.com)Unknown University of Alberta link-checking ?
2444iTunes/x.x.xiTunes UA name for access and decrypt the iTunes music store pages
2445IUPUI Research Bot v 1.9aSome spam bot from 66.139.78.xx(x)
2446iVia Page Fetcher (http://ivia.ucr.edu/useragents.shtml)iVia robot - Open source Internet portal & virtual library system software
2447iVia/4.0 CanonizeUrl (http://infomine.ucr.edu/iVia/useragents.shtmliVia robot - Open source Internet portal & virtual library system software
2448IWAgent/ 1.0 - www.brandprotect.comBD BrandProtect - brand, company or trademarks online monitoring
2449J-PHONE/3.0/J-SH07Proxy message from jp-q.ne.jp
2450Jabot/6.x (http://odin.ingrid.org/)ODIN Directory Japan robot (163.138.95.xx)
2451Jabot/7.x.x (http://odin.ingrid.org/)ODIN Directory Japan robot (163.138.95.xx)
2452JackGerman Domanova (offline since Feb.02) robot
2453Jakarta Commons-HttpClient/2.0xxxJakarta Commons (Java based) HTTP client
2454Jakarta Commons-HttpClient/3.0-rcxJakarta Commons (Java based) HTTP client
2455Jambot/0.1.x (Jambot; http://www.jambot.com/blog; crawler@jambot.com)JamBot search robot (70.146.82.xx)
2456Jambot/0.2.1 (Jambot; http://www.jambot.com/blog/static.php?page=webmaster-robot; crawler@jambot.com)JamBot search robot
2457Java 1.1Java VM
2458Java/1.4.1_01Java VM
2459Java1.0.21.0Java VM
2460Java1.1.xx.xJava VM
2461Java1.3.0rc1Java VM
2462Java1.3.xJava VM
2463Java1.4.0Java VM
2464Jayde Crawler. http://www.jayde.comJayde B2B Search robot (66.28.139.xx)
2465JBH Agent 2.0some site downloading tool ? via 61.77.51.xxx
2466jBrowser/J2ME Profile/MIDP-1.0 Configuration/CLDC-1.0 (Google WAP Proxy/1.0)WAP 2.0 / jBrowser for handhelds
2467JCheckLinks/0.1 RPT-HTTPClient/0.3-1JCheckLinks Java hyperlink validator
2468JDK/1.1Java Development Kit
2469Jeode/1.x.xInsignias Jeode (PDA) Java platform
2470Jetbot/1.0JetEye Search robot (64.62.142.xxx / 64.71.144.xxx)
2471JetBrains Omea Reader 1.0.x (http://www.jetbrains.com/omea_reader/)Omea RSS - Atom - newsgroups web page reader
2472JetBrains Omea Reader 2.0 Release Candidate 1 (http://www.jetbrains.com/omea_reader/)Omea RSS - Atom - newsgroups web page reader
2473JetCarJetCar / Flashget download manager
2474Jigsaw/2.2.x W3C_CSS_Validator_JFouffa/2.0Jigsaw - W3C's CSS Validator Server
2475JoBo/1.x (http://www.matuschek.net/jobo.html)Jobo website downloading program
2476JoBo/@JOBO_VERSION@(http://www.matuschek.net/jobo.html)Jobo website downloading program
2477JobSpider_BA/1.1Finacialbot.com - German (213.61.218.xx) job search JobRoboter
2478JOC Web SpiderJocsoft Web Spider - website downloading tool
2479JordoMedia/1.0 RSS File Reader (http://www.jordomedia.com)Jordo Media RSS / Atom feed directory link checking
2480Journster [alpha] (http://journster.com/)Journster.com RSS/Atom aggregator
2481Journster.com RSS/Atom aggregator 0.5 (http://www.journster.com/bot.phtml)Journster.com RSS/Atom aggregator
2482JRTS Check Favorites UtilityCheck Favorites bookmark checking
2483JRTwine Software Check Favorites UtilityCheck Favorites bookmark checking
2484Jyxobot/xJyxo search (Czech Republic) robot (212.71.128.xx)
2485K-Meleon/0.6 (Windows; U; Windows NT 5.1; en-US; rv:0.9.5) Gecko/20011011K-meleon browser - Windows 2000
2486k2spiderVerity K2 Spider ( Network search software)
2487KAIST AITrc CrawlerUnknown robot from AITrc (Advanced Information Technology Research Center) - Korea
2488KakleBot - www.kakle.com/0.1 (KakleBot - www.kakle.com; http:// www.kakle.com/bot.html; support@kakle.com)Kakle ranked metasearch robot
2489kalooga/kalooga-4.0-dev-datahouse (Kalooga; http://www.kalooga.com; info@kalooga.com)Kalooga image crawler
2490kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler; crawler@kalooga.com)Kalooga image crawler
2491Kapere (http://www.kapere.com)Kapere site grapper / web downloader
2492Kazehakase/0.x.x.[x]Kazehakase - Gecko based browser (Japan)
2493KDDI-SN22 UP.Browser/6.0.7 (GUI) MMP/1.1 (Google WAP Proxy/1.0)Openwave UP.Browser for mobiles via Google WAP Proxy (216.239.33.x)
2494Kenjin SpiderKenjin Spider search agent
2495Kevin http://dznet.com/kevin/Dznet.com Kevin crawler (link checking ?) via 68.39.148.xx (nj.comcast.net)
2496Kevin http://websitealert.net/kevin/Website AlertsKevin crawler (website monitoring) via 68.39.148.xx (nj.comcast.net)
2497KE_1.0/2.0 libwww/5.2.8Voila.fr robot
2498KFSW-Bot (Version: 1.01 powered by KFSW www.kfsw.de)Some Perl search script from KFSW (Germany)
2499kinja-imagebot (http://www.kinja.com/)kinja weblog search robot
2500kinjabot (http://www.kinja.com)kinja weblog search robot
2501KIT-Fireball/2.0Fireball search (Germany) robot
2502KIT-Fireball/2.0 (compatible; Mozilla 4.0; MSIE 5.5)Fireball search (Germany) robot
2503Klondike/1.50 (WSP Win32) (Google WAP Proxy/1.0)Klondike WAP Browser
2504KnowItAll(knowitall@cs.washington.edu)University of Washington KnowItAll - web information extraction
2505Knowledge.com/0.xThe knowledge.com (ODP) directory robot
2506Kontiki Client x.xxKontiki Client download manager
2507Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info.html; webcrawler@krugle.com)Krugle source code search engine for developers (64.71.164.1xx)
2508KSbot/1.0 (KnowledgeStorm crawler; http://www.knowledgestorm.com/resources/content/crawler/index.html; crawleradmin@knowledgestorm.com)KnowledgeStorm technology industry crawler for Findtech.com
2509kuloko-bot/0.xKuloko contextual search robot
2510kulokobot www.kuloko.com kuloko@backweave.comKuloko contextual search robot
2511kulturarw3/0.1National Library of Sweden Heritage Project robot
2512KummHttp/1.1 (compatible; KummClient; Linux rulez)Link or server checking from Sanoma Budapest (195.70.35.xxx)
2513KWC-KX9/1109 UP.Browser/6.2.3.9.g.1.107 (GUI) MMP/2.0 UP.Link/6.3.0.0.0UP.Browser for mobiles on Kyocera KWC-KX9 cellphone
2514Labrador/0.2; http://ir.dcs.gla.ac.uk/labrador; craigm@dcs.gla.ac.ukTREC Blog Track - Blog and news feed crawler
2515LachesisIntels Lachesis web site response time monitoring tool
2516lanshanbot/1.0Unknown robot from Easten Network China (202.96.51.1xx)
2517lanshanbot/1.0 (+http://search.msn.com/msnbot.htm)Unknown robot from Easten Network China (202.96.51.1xx)
2518LapozzBot/1.4 ( http://robot.lapozz.com)Lapozz search (Hungary) robot (82.131.195.xx)
2519LapozzBot/1.5 (+http://robot.lapozz.hu)Lapozz search (Hungary) robot (82.131.195.xx)
2520larbin (samualt9@bigfoot.com)Larbin indexer used as Metacarta.com (66.28.xx.xxx) robot
2521LARBIN-EXPERIMENTAL (efp@gmx.net)Unknown robot from 66.230.140.xx (argon.oxeo.com)
2522larbin_2.1.1 larbin2.1.1@somewhere.comLarbin indexer used by Mitsubishi Electric Research Labs
2523larbin_2.2.0 (crawl@compete.com)Larbin indexer used as Compete.com crawler
2524larbin_2.2.1_de_Viennot (Laurent.Viennot@inria.fr)Larbin indexer used as Inria robot
2525larbin_2.2.2 (sugayama@lab7.kuis.kyoto-u.ac.jp)Larbin indexer used as Kyoto University robot
2526larbin_2.2.2_guillaume (guillaume@liafa.jussieu.fr)Larbin indexer used as Inria robot
2527larbin_2.6.0 (larbin2.6.0@unspecified.mail)Larbin indexer used by an unknown dsl.net client
2528larbin_2.6.1 (larbin2.6.1@unspecified.mail)Larbin indexer used by diff. IPs / services
2529larbin_2.6.2 (hamasaki@grad.nii.ac.jp)Larbin indexer used by National Institut of Informatics (NII/Japan)
2530larbin_2.6.2 (larbin2.6.2@unspecified.mail)Larbin indexer used by diff. IPs
2531larbin_2.6.2 (listonATccDOTgatechDOTedu)Larbin indexer used as robot by Georgia Institute of Technology http://www.gatech.edu/
2532larbin_2.6.2 (pimenas@systems.tuc.gr)Larbin indexer used by Technical University of Crete
2533larbin_2.6.2 (tom@lemurconsulting.com)Larbin indexer used as Lemur Consulting robot
2534larbin_2.6.2 (vitalbox1@hotmail.com)Larbin indexer used as robot via cloud9.net (168.100.192.xxx)
2535larbin_2.6.3 (ltaa_web_crawler@groupes.epfl.ch)Unknown robot from EPFL Lausanne Switzerland (128.178.155.1xx)
2536larbin_2.6.3 (wgao@genieknows.com)Larbin indexer used by GenieKnows.com search
2537larbin_2.6.3_for_(http://cosco.hiit.fi/search/) tsilande@hiit.fiLarbin indexer used by Next Generation Information Retrieval (NGIR)
2538larbin_2.6_basileocaml (basile.starynkevitch@cea.fr)Larbin indexer used by CEA / DCom Rechercher
2539larbin_devel (http://pauillac.inria.fr/~ailleret/prog/larbin/)Larbin indexer used as Inria robot
2540lawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo.com; webmaster@lawinfo.com)LawInfo - Lawyer and attorney directory
2541lc/$ROADS::Version libwww-perl/5.00ROADS - Perl web based subject based gateway tool
2542lcabotAccept: */*unknown robot via MTT.ca / Aliant.ca
2543LeapTag/0.8.1.beta081.r3750 (compatible; Mozilla 4.0; MSIE 5.5; robot@yoriwa.com)LeapTag news reader and content discovery tool
2544LECodeChecker/3.0 libgetdoc/1.0Linkexchange crawler
2545LeechGet 200x (www.leechget.de)Leechget download manager
2546LEIA/2.90Gseek.com (site is offline) robot
2547LEIA/3.01pr (LEIAcrawler; [SNIP])Gseek.com (site is offline) robot
2548LetsCrawl.com/1.0 +http://letscrawl.com/Maybe logfile spamming for Lets crawl! search (Germany)
2549LexiBot/1.00Lexibot (exMataHari) search software
2550LG-LX260 POLARIS-LX260/2.0 MMP/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.1Polaris mobile browser on LG LX 260 Sprint Rumor phone
2551LG/U8138/v1.0LG 8138 Mobile Phone browser
2552Libby_1.1/libwww-perl/5.47About.com robot
2553libcurl-agent/1.0libcurl's (multiprotocol file transfer library) standard user-agent name
2554LibertyW (+http://www.lw01.com)LibertyW search for mobile (France)
2555libWeb/clsHTTP -- hiongun@kt.co.krKorea Telecom Search robot
2556libwww-perl/5.41CMP United Media robot
2557libwww-perl/5.45SplatSearch robot (207.44.142.xx)
2558libwww-perl/5.48Alexa robot
2559libwww-perl/5.50diff. IPs / services
2560libwww-perl/5.52 FP/2.1Fast Search robot
2561libwww-perl/5.52 FP/4.0Fast Search robot
2562libwww-perl/5.53diff. IPs / services
2563libwww-perl/5.63Profile for You internet profiling (?)
2564libwww-perl/5.64unknown link checking from Wanadoo.fr (193.253.33.xxx)
2565libwww-perl/5.65Amidalla search engine robot (62.241.33.xx)
2566libwww-perl/5.800SplatSearch robot (72.36.210.xx)
2567libwww/5.3.2Mediater Rechercher robot
2568Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/)Liferea - Linux feed reader
2569Liferea/1.x.x (Linux; es_ES.UTF-8; http://liferea.sf.net/)Liferea (Linux Feed Reader) news aggregator for Unix and Linux
2570LightningDownload/1.0beta2Lightning Download manager
2571LightningDownload/1.x.xLightning Download manager
2572LightningDownload/1.x.x [Accelerated x]Lightning Download manager
2573LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)com)Lijit blog search spider
2574Lincoln State Web BrowserSome spam bot
2575Link Valet Online 1.xLink Valet online link checking
2576LinkAlarm/2.xLinkalarm link validation
2577LinkbotLinkbot Pro link checking software
2578linkbotRpsoft 2000 Site-Crawler
2579Linkbot x.0Linkbot Pro link checking software
2580LinkCheck (linkcheck@inter7.com http://www.inter7.com/linkcheck)Linkcheck - linkchecking tool for Unix/Linux
2581LinkLint-checkonly/2.x.xLinklint - Perl html link checker
2582LinkLint-spider/2.x.xLinklint - Perl html link checker
2583linknzbotLinknz - The Kiwi Search Engine (New Zealand)
2584LinkPimpin v1.0Link-Pimp web directory link checking
2585LinkProver 2.1TafWeb link checking program
2586Links (0.9x; Linux 2.4.7-10 i686)Links text browser for Unix & OS/2
2587Links (0.9xpre12; Linux 2.2.14-5.0 i686; 80x24)Links text browser for Unix & OS/2
2588Links (2.xpre7; Linux 2.4.18 i586; x)Links text browser for Unix & OS/2
2589Links - http://gossamer-threads.com/scripts/links/Links SQL directory management program
2590Links 2.0 (http://gossamer-threads.com/scripts/links/)Links SQL directory management program
2591Links SQL (http://gossamer-threads.com/scripts/links-sql/)Links SQL directory management program
2592Links4US-Crawler, (+http://links4us.com/)Links4us ODP based directory link checking
2593LinkScan/11.0beta2 UnixShareware robot from Elsop.com (used by Indiafocus/Indiainfo)Shareware robot from Elsop.com (used by Indiafocus/Indiainfo and others)
2594LinkScan/9.0g UnixShareware robot from Elsop.com (used by Indiafocus/Indiainfo and others)
2595LinkScan/x.x UnixShareware robot from Elsop.com
2596LinksManager.com (http://linksmanager.com/linkchecker.html)Linksmanager bookmark checking
2597LinkSonar/1.35LinkSonar (Japan) link checking tool
2598LinkSweeper/1.xLeft Side Software's LinkSweeper (ceased) bookmark utility
2599LinkWalkerSeventwentyfour link checking robot
2600link_check3.plx libwww-perl/5.65link check 3 - Perl HTML link checker (from Perl for Web Site Management)
2601ListBidBot (freelance job spider http://listbid.com)FreelanceListbid.com / Directnic.com link checking ?
2602LiveTrans/Nutch-0.9 (maintainer: cobain at iis dot sinica dot edu dot tw; http://wkd.iis.sinica.edu.tw/LiveTrans/)WKD Lab: LiveTrans - Online query and terminology translation service
2603Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html)llaut robot - Universitat de les Illes Balears (Spain)
2604LMQueueBot/0.2E-Mail harvesting robot - same as ContactBot
2605lmspider (lmspider@scansoft.com)lmspider from Scansoft (192.133.61.xx) - Web text collector
2606LNSpiderguyLexis-Nexis robot
2607LocalBot/1.0 ( http://www.localbot.co.uk/)LocalBot company information collector
2608LocalcomBot/1.2.x ( http://www.local.com/bot.htm)Local.com local search robot (216.52.252.xxx)
2609Lockstep Spider/1.0Lockstep (website content protection tool) user agent
2610Look.comGlobalQueue spider (64.40.105.xxx)
2611Lotus-Notes/4.5 ( Windows-NT )Lotus Notes browser
2612LotusDiscovery/x.0 (compatible; Mozilla 4.0; MSIE 4.01; Windows NT)IBM Lotus Discovery Server
2613Lovel as 1.0 ( +http://www.everatom.com)Everatom.com song lyrics search
2614LTI/LemurProject Nutch Spider/Nutch-1.0-dev (lti crawler for CMU; http://www.lti.cs.cmu.edu; changkuk at cmu dot edu)LTI - The Lemur Toolkit for Language Modeling and Information Retrieval via Yahoo
2615LTI/LemurProject Nutch Spider/Nutch-1.0-dev (Research spider using Nutch; http://www.lemurproject.org; mhoy@cs.cmu.edu)LTI - The Lemur Toolkit for Language Modeling and Information Retrieval via Yahoo
2616luchs.at URL checkerLuchs.at (Linux Wiki) link checking
2617LunascapeLunascape IE based browser (Japan)
2618lwp-trivial/1.32Ultimate Search / Smartdesk (no website) robot
2619lwp-trivial/1.34Search4free robot
2620lwp-trivial/1.35Expert HTML online source viewer
2621LWP::Simple/5.22Perl LWP:Collective module
2622LWP::Simple/5.36Perl LWP:Collective module
2623LWP::Simple/5.48Perl LWP:Collective module - Linkomatic robot
2624LWP::Simple/5.50Perl LWP:Collective module - secure-netz.de link checking (in conjunction w. libwww-perl/5.50)
2625LWP::Simple/5.51Perl LWP:Collective module - Inktomi (62.253.64.x) robot
2626LWP::Simple/5.53Perl LWP:Collective module - Only.com
2627LWP::Simple/5.63Perl LWP:Collective module
2628LWP::Simple/5.803ThePlanet/jaja-jak-globusy.com Google Adsense refferer spam bot from 70.85.116.* / 70.84.128.xxx / 70.85.193.xxx
2629Lycos_Spider_(modspider)Lycos spider
2630Lycos_Spider_(T-Rex)Lycos spider
2631Lynx/2-4-2 (Bobcat/0.5 [DOS] Jp Beta04)Bobcat - Text and Lynx based DOS browser
2632Lynx/2.6 libwww-FM/2.14Lynx 2.x text mode browser
2633Lynx/2.8 (;http://seebot.org)seebot.org online service - uses lynx browser for crawlers view of web pages
2634Lynx/2.8.3dev.9 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6Lynx 2.x text mode browser
2635Lynx/2.8.4rel.1 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6c (human-guided@lerly.net)Lynx 2.x text mode browser used as robot via cogentco.com
2636Mac Finder 1.0.xxSome spam bot
2637Mackster( http://www.ukwizz.com )UKWizz search robot
2638Mag-NetWind.it client user-agent ?
2639MagicWML/1.0 (forcewml)All Magic/Wap wml service for mobile devices
2640MagpieRSS/0.7x (+http://magpierss.sf.net)Magpie RSS - PHP RSS Parser
2641Mahiti.Com/Mahiti Crawler-1.0 (Mahiti.Com; http://mahiti.com ; mahiti.com)Mahiti.com India search crawler
2642Mail.Ru/1.0Mail.ru search
2643mailto:webcraft@bea.comUnknown Bea robot
2644mammoth/1.0 ( http://www.sli-systems.com/)SLI Systems mammoth robot
2645MantraAgentLooksmart robot
2646MapoftheInternet.com ( http://MapoftheInternet.com)Map of the Internet visual search engine index robot
2647Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot)Kolibri.de robot
2648Marketwave Hit ListPilot Hitlist web site analysis
2649MartiniLooksmart directory page analysis
2650MARTINILooksmart directory page analysis
2651Marvin v0.3Marvin Medhunt robot
2652MaSagool/1.0 (MaSagool; http://sagool.jp/; info@sagool.jp)Sagool search Japan robot
2653Mass Downloader 2.xMass Downloader download manager
2654MasterSeekMasterseek (Scandinavia) Beta business search ?
2655Mata Hari/2.00Lexibot (exMataHari) search software
2656Matrix S.p.A. - FAST Enterprise Crawler 6 (Unknown admin e-mail address)Virgilio Italy robot (212.48.11.xxx) using Fast Enterprise Search
2657maxomobot/dev-20051201 (maxomo; http://67.102.134.34:4047/MAXOMO/MAXOMObot.html; maxomobot@maxomo.com)Maxomo multimedia search robot
2658McBot/5.001 (windows; U; NT4.0; en-us)Unknown robot from McAfee Austria (80.123.144.xx)
2659MDbot/1.0 (+http://www.megadownload.net/bot.html)MegaDownload files search robot
2660Media Player ClassicSubstitute for Windows Media Player
2661MediaCrawler-1.0 (Experimental)Media Find crawler
2662Mediapartners-Google/2.1 ( http://www.googlebot.com/bot.html)Google AdSense robot
2663MediaSearch/0.1WWW.fi Media Search
2664MegaSheep v1.0 (www.searchuk.com internet sheep)Search UK robot
2665Megite2.0 (http://www.megite.com)Megite web2.0 RSS and news service software
2666Mercator-1.xMercator crawler software (used by Altavista)
2667Mercator-2.0Mercator crawler software (used by Altavista)
2668Mercator-Scrub-1.1Mercator crawler software (used by Altavista)
2669Metaeuro Web Crawler/0.2 (MetaEuro Web Search Clustering Engine; http://www.metaeuro.com; crawler at metaeuro dot com)MetaEuro.com Clustering Web Search Engine crawler
2670MetaGer-LinkCheckerMetaGer search robot (Germany)
2671MetagerBot/0.8-dev (MetagerBot; http://metager.de; )MetaGer search robot (Germany)
2672MetaGer_PreChecker0.1MetaGer search robot (Germany)
2673MetaProducts Download Express/1.xDownload Express download manager
2674Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner.de/)Metaspinner search robot - Germany
2675metatagsdir/0.7 (+http://metatagsdir.com/directory/)Metatagsdir.com directory index spider
2676MFC Foundation Class Library 4.0Microsoft Foundation Class Library - i.e. used for e-mail harvesting from 68.154.96.xx (bellsouth.net)
2677MFC_Tear_SampleMicrosoft.com user agent
2678MFHttpScanAdvanced Site Crawler web site ripper and extractor
2679MicroBazGigaBaz Brainbot (Germany) robot (213.139.152.xx)
2680Microsoft Data Access Internet Publishing Provider Cache ManagerMS Office 2000 acting as WebDAV client
2681Microsoft Data Access Internet Publishing Provider DAVMS Office 2000 acting as WebDAV client
2682Microsoft Data Access Internet Publishing Provider Protocol DiscoveryServer probe for data access operations using MS Frontpage with OPTION header
2683Microsoft Data Access Internet Publishing Provider Protocol DiscoveryMS Office 2000 acting as WebDAV client
2684Microsoft Log Parser 2.2Microsoft Log Parser text query tool
2685Microsoft Small Business IndexerMS Small Business Server content indexer
2686Microsoft URL Control - 6.00.8xxxuser agent looks for form-mail components (spam-bot)
2687MicrosoftPrototypeCrawler (How's my crawling? mailto:newbiecrawler@hotmail.com)Unknown robot from Microsoft.com (131.107.163.xx)
2688Microsoft_Internet_Explorer_5.00.438 (fjones@isd.net)Secure Computing SmartFilterWhere / Bess web filter (192.55.214.xx)
2689MIIxpc/4.2xpc-mii.net HTTP server message
2690Mindjet MindManagerThe Mindjet blog MindManager category
2691minibotunknown robot via Korea Telecom (211.218.xxx.xxx)
2692miniRank/1.6 (Website ranking; www.minirank.com; robot)mini- Rank website popularity tool
2693MiracleAlphaTestunknown robot via nec.co.jp Telecom (210.143.35.xx)
2694Missauga Locate 1.0.0Some spam bot
2695Missigua Locator 1.9Some spam bot
2696Missouri College BrowseSome spam bot
2697Mister Pix II 2.02aMister PiX picture finding software
2698Mister PiX version.dllMister PiX picture finding software
2699Misterbot-Nutch/0.7.1 (Misterbot-Nutch; http://www.misterbot.fr; admin@misterbot.fr)Misterbot search France robot
2700Miva (AlgoFeedback@miva.com)Miva / ex Findwhat.com search robot (66.150.55.2xx)
2701Mizzu Labs 2.2Some spam bot from Jasmine Internet - Bangkok (203.147.0.xx)
2702MJ12bot/vx.x.x (http://majestic12.co.uk/bot.php?+)Majestic-12 DSearch MJ12bot (Experimental distributed crawler)
2703MJ12bot/vx.x.x (http://www.majestic12.co.uk/projects/dsearch/mj12bot.php)Majestic-12 DSearch MJ12bot (Experimental distributed crawler)
2704MJBot (SEO assessment)MJB SEO Club MJBot
2705MLBot (www.metadatalabs.com)MLBot - metadata labs web crawler for building a media index (beta)
2706MnogoSearch/3.2.xxmnoGoSearch (ex UdmSearch) software robot
2707Mo College 1.9Unknown bad bot - maybe guestbook spamming or email harvesting
2708moget/x.x (moget@goo.ne.jp)Goo Japan / Inktomi robot (210.173.179.xx)
2709mogimogi/1.0Goo Japan / Inktomi robot (210.173.179.xx)
2710moiNAG 0.02moiNag - net.art generator
2711MojeekBot/0.x (archi; http://www.mojeek.com/bot.html)Mojeek Search Preview robot (217.155.205.xx)
2712monkeyagentGreasemonkey Firefox extension
2713MoonBrowser (version 0.41 Beta4)Moonbrowser - IE based browser (Japan)
2714Moreoverbot/x.00 (+http://www.moreover.com)Moreover / FeedDirect RSS feed robot
2715Morris - Mixcat Crawler ( http://mixcat.com)MixCat robot s. also Felix
2716Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml)Robots.txt online checker
2717Motorola-V3m ObigoObigo WAP browser for mobiles on Motorola V3
2718Mouse-House/7.4 (spider_monkey spider info at www.mobrien.com/sm.shtml)MPRM Group Ltd. Spider Monkey robot
2719MovableType/x.xMovable Type web-based personal publishing system
2720mozDex/0.xx-dev (mozDex; http://www.mozdex.com/en/bot.html; spider@mozdex.com)Mozdex Open search engine spider (65.98.100.2xx)
2721Mozi!Bell Nexxia / Sympatico Canada user robot & spoofed referer from diff. IPs
2722MozillaUnknown robots from diff. IPs
2723Mozilla (libwhisker/2.4)libwhisker - HTTP client and utility - Perl library
2724Mozilla (Mozilla@somewhere.com)http://www.somewhere.com robot
2725Mozilla 4.0(compatible; BotSeer/1.0; +http://botseer.ist.psu.edu)BotSeer search engine for robots.txt
2726Mozilla/1.1 (compatible; MSPIE 2.0; Windows CE)PDA Pocket IE 2.x Windows CE
2727Mozilla/1.10 [en] (Compatible; RISC OS 3.70; Oregano 1.10)Oregano browser for RISC OS
2728Mozilla/1.22 (compatible; MSIE 2.0d; Windows NT)IE 2.x WinNT
2729Mozilla/1.22 (compatible; MSIE 5.01; PalmOS 3.0) EudoraWeb 2EudoraWeb 2.0 browser (Eudora Internet Suite) for PalmOS
2730Mozilla/2.0Dummy user agent - i.e. used by Namo Web Editor
2731Mozilla/2.0 (compatible; AOL 3.0; Mac_PowerPC)AOL Mac
2732Mozilla/2.0 (Compatible; AOL-IWENG 3.0; Win16)AOL Win 3.x
2733Mozilla/2.0 (compatible; Ask Jeeves)Ask Jeeves /Teoma robot
2734Mozilla/2.0 (compatible; Ask Jeeves/Teoma)Ask Jeeves /Teoma robot
2735Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/webmasters.shtml)Ask Jeeves /Teoma robot
2736Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://sp.ask.com/docs/about/tech_crawling.html)Ask Jeeves /Teoma robot
2737Mozilla/2.0 (compatible; EZResult -- Internet Search Engine)Direct Hit Robot
2738Mozilla/2.0 (compatible; MS FrontPage x.0)MS Frontpage x.x web editor
2739Mozilla/2.0 (compatible; MSIE 2.1; Mac_PowerPC)IE 2.x Mac Power PC
2740Mozilla/2.0 (compatible; MSIE 3.02; Update a; AK; Windows NT)IE 3.x WinNT
2741Mozilla/2.0 (compatible; MSIE 3.02; Update a; AOL 3.0; Windows 95)IE 3.x AOL Win95
2742Mozilla/2.0 (compatible; MSIE 3.0; AK; Windows 95)IE 3.x Win95
2743Mozilla/2.0 (compatible; MSIE 3.0; Windows 3.1)IE 3.x Win 3.1
2744Mozilla/2.0 (compatible; MSIE 3.0B; Win32)IE 3.x WinXP
2745Mozilla/2.0 (compatible; NEWT ActiveX; Win32)Borland Delphi .OCX component used by WebCollector email harverster
2746Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E)Thunderstone's Webinator Web indexing program
2747Mozilla/2.0 compatible; Check&Get 1.1x (Windows 98)Check&Get bookmark and link checking tool
2748Mozilla/2.01 (Win16; I)Netscape 2.x Win3.x International
2749Mozilla/2.02Gold (Win95; I)Netscape 2.x Gold Win95
2750Mozilla/3.0 (compatible)Faked user agent for diff. purposes i.e.: - some download manager - E-mail harvesting
2751Mozilla/3.0 (compatible; AvantGo 3.2)AvantGo PDA browser
2752Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com)Searchhippo robot
2753Mozilla/3.0 (compatible; HP Web PrintSmart 04b0 1.0.1.34)HP Web PrintSmart (discontinued) - web page printing software
2754Mozilla/3.0 (compatible; Indy Library)Internet Direct Library for Borland (often used as e-mail address collector and mass mailing tool)
2755Mozilla/3.0 (compatible; Linkman)Outertechs Linkman bookmark tool
2756Mozilla/3.0 (compatible; MuscatFerret/1.5.4; claude@euroferret.com)Euroferret robot
2757Mozilla/3.0 (compatible; MuscatFerret/1.5; olly@muscat.co.uk)Euroferret robot
2758Mozilla/3.0 (compatible; MuscatFerret/1.6.x; claude@euroferret.com)Euroferret robot
2759Mozilla/3.0 (compatible; netart generator/1.0; libwww-perl/5.64)Netart Generator - script generated random websites
2760Mozilla/3.0 (compatible; NetPositive/2.2)NetPositive BEOS browser
2761Mozilla/3.0 (compatible; Opera/3.0; Windows 3.1) v3.1Opera 3.x Win3.x
2762Mozilla/3.0 (compatible; Opera/3.0; Windows 95/NT4) 3.2Opera 3.x Win95/NT
2763Mozilla/3.0 (compatible; PerMan Surfer 3.0; Win95)Perman Surfer bookmark tool
2764Mozilla/3.0 (compatible; REL Software Web Link Validator 2.x)Web Link Validator link validation software
2765Mozilla/3.0 (compatible; scan4mail (advanced version) http://www.peterspages.net/?scan4mail)Scan4Mail online mail extraction service
2766Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com)WebWobot UK search engine robot (82.43.129.2xx)
2767Mozilla/3.0 (compatible; Web Link Validator 2.x)Web Link Validator http://www.relsoftware.com/ link validation softwareWeb Link Validator link validation software
2768Mozilla/3.0 (compatible; WebCapture x.x; Auto; Windows)Xelios Web Capture (now Wysigot) website downloading tool (Discontinued)
2769Mozilla/3.0 (compatible; Webinator-DEV01.home.iprospect.com/2.56)Iprospect search engine positioning using Thunderstone's Webinator
2770Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56)CyberAlert's Media Monitor using Thunderstone's Webinator
2771Mozilla/3.0 (Compatible;Viking/1.8)Viking server user/client
2772Mozilla/3.0 (DreamPassport/3.0)One of DC-Sakuras download manager user-agent names
2773Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)Ilse Netherlands robot (62.69.178.xx)
2774Mozilla/3.0 (Liberate DTV 1.1)Liberate DTV server suite / TV-emulator
2775Mozilla/3.0 (Planetweb/2.100 JS SSL US; Dreamcast US)Planetweb 2.1 Browser (discontinued) for Dreamcast
2776Mozilla/3.0 (Slurp.so/Goo; slurp@inktomi.com; http://www.inktomi.com/slurp.html)http://www.goo.ne.jp /Inktomi robot
2777Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)Inktomi (Hotbot-Lycos-NBCi) robot - 72.30.61.xx(x)
2778Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)Inktomi (Hotbot-Lycos-NBCi) robot - 72.30.61.xx(x)
2779Mozilla/3.0 (Vagabondo/1.1 MT; webcrawler@NOSPAMwise-guys.nl; http://webagent.wise-guys.nl/)WiseGuys robot Netherland - 82.94.216.2
2780Mozilla/3.0 (Vagabondo/1.x MT; webagent@wise-guys.nl; http://webagent.wise-guys.nl/)WiseGuys robot Netherland - 82.94.216.2
2781Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)Ilse Netherlands robot
2782Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMwise-guys.nl; http://webagent.wise-guys.nl/)WiseGuys robot Netherland - 82.94.216.2
2783Mozilla/3.0 (Win16; I)Netscape 3.x Win3.x
2784Mozilla/3.0 (Win95; I)Netscape 3.x Win95
2785Mozilla/3.0 (WinNT; I)Netscape 3.x WinNT
2786Mozilla/3.0 (WorldGate Gazelle 3.5.1 build 11; FreeBSD2.2.8-STABLE)Netscape 3.x FreeBSD
2787Mozilla/3.0 (X11; I; OSF1 V4.0 alpha)Netscape 3.x OSF1 V4.0 alpha
2788Mozilla/3.0 NAVIO_AOLTV (11; 13; Philips; PH200; 1; R2.0C36_AOL.0110OPTIK; R2.0.0139d_OPTIK)AOL Web TV
2789Mozilla/3.0 WebTV/1.2 (compatible; MSIE 2.0)WebTV
2790Mozilla/3.01 (compatible; AmigaVoyager/2.95; AmigaOS/MC680x0)Amiga Voyager Browser Amiga
2791Mozilla/3.01 (Compatible; Links2Go Similarity Engine)Links2Go robot
2792Mozilla/3.01 (compatible; Netbox/3.5 R92; Linux 2.2)Netgem Netbox cable modem TV Box Linux
2793Mozilla/3.01-C-MACOS8 (Macintosh; I; PPC)Netscape 3.x Mac
2794Mozilla/3.01Gold (X11; I; Linux 2.0.32 i486)Netscape 3.x Linux
2795Mozilla/3.01Gold (X11; I; SunOS 5.5.1 sun4m)Netscape 3.x SunOS
2796Mozilla/3.01SGoldC-SGI (X11; I; IRIX 6.3 IP32)Netscape 3.x Irix
2797Mozilla/3.04 (compatible; ANTFresco/2.13; RISC OS 4.02)ANT Fresco Browser Risc OS
2798Mozilla/3.04 (compatible; NCBrowser/2.35; ANTFresco/2.17; RISC OS-NC 5.13 Laz1UK1309)NCBrowser ANT Fresco Browser Risc OS
2799Mozilla/3.04 (compatible;QNX Voyager 2.03B ;Photon)QNX OS Voyager embedded browser
2800Mozilla/3.x (I-Opener 1.1; Netpliance)I-Opener (was www.netpliance.com/) web PC
2801Mozilla/4.0Yahoo Mindset: Intent-driven Search (66.228.182.1xx)
2802Mozilla/4.0 (agadine3.0) www.agada.deAgada search (Germany) robot
2803Mozilla/4.0 (Compatible); URLBase 6URLBase 6 bookmark manager
2804Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com)Astrafind! adult search robot (66.98.252.xx)
2805Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)WiseGuys robot Netherland - 82.94.216.2
2806Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)WiseGuys robot Netherland - 82.94.216.2
2807Mozilla/4.0 (compatible; ReGet Deluxe 5.1; Windows NT 5.1)ReGet Deluxe! download manager
2808Mozilla/4.0 (compatible; Advanced Email Extractor v2.xx)Advanced Email Extractor e-mail collector (spam bot)
2809Mozilla/4.0 (compatible; Arachmo)Arachmo Spider - web site file extraction tool
2810Mozilla/4.0 (compatible; BorderManager 3.0)Novell Border Manager security suite
2811Mozilla/4.0 (compatible; BOTW Spider; +http://botw.org)Best of the Web directory link checking
2812Mozilla/4.0 (compatible; B_L_I_T_Z_B_O_T)Blitzsuche Germany robot
2813Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)Content Control from Blue Coat
2814Mozilla/4.0 (compatible; Check&Get 3.0; Windows NT)Check&Get bookmark manager, web change monitor and archiver
2815Mozilla/4.0 (compatible; ChristCrawler.com ChristCrawler@ChristCENTRAL.com)Christcentral.com Christcrawler (was www.christcrawler.com)
2816Mozilla/4.0 (compatible; crawlx, crawler@trd.overture.com)Yahoo Search Marketing crawler (68.142.211.1xx)
2817Mozilla/4.0 (compatible; DAUMOA-video; +http://ws.daum.net/aboutkr.html)DAUMOA - Daum search Korea robot (211.115.109.xxx)
2818Mozilla/4.0 (compatible; DepSpid/5.0x; +http://about.depspid.net)DepSpid distributed web crawler for link dependencies
2819Mozilla/4.0 (compatible; DnloadMage 1.0)Download Mage download manager
2820Mozilla/4.0 (compatible; FastCrawler3 support-fastcrawler3@fast.no)Fast/Alltheweb crawler
2821Mozilla/4.0 (compatible; FDSE robot)Fluid Dynamics Search Engine (FDSE) robot used by Abadoor.de
2822Mozilla/4.0 (compatible; GPU p2p crawler http://gpu.sourceforge.net/search_engine.php)GPU Distributed Search Engine crawler
2823Mozilla/4.0 (compatible; grub-client-0.2.x; Crawl your stuff with http://grub.org)Grub open source crawler
2824Mozilla/4.0 (compatible; grub-client-0.3.x; Crawl your own stuff with http://grub.org)Grub open source crawler
2825Mozilla/4.0 (compatible; grub-client-2.x)Grub open source crawler used by Looksmart ( 64.241.242.xx)
2826Mozilla/4.0 (compatible; ibisBrowser)ibisBrowser Japanese mobile browser
2827Mozilla/4.0 (compatible; ICS 1.2.xxx)Novell iChain Cool Solutions caching
2828Mozilla/4.0 (compatible; IE-Favorites-Check-0.5)IE Favorites Check - Bookmark manager
2829Mozilla/4.0 (compatible; Iplexx Spider/1.0 http://www.iplexx.at)Iplexx Austria (webhosting company) logfile spamming bot
2830Mozilla/4.0 (compatible; KeepNI web site monitor)KeepNi Monitors - Web site monitoring / link checking tool
2831Mozilla/4.0 (compatible; Link Utility; http://net-promoter.com)NetPromoter Link Utility link checking tool
2832Mozilla/4.0 (compatible; Lotus-Notes/5.0; Windows-NT)Lotus Notes 5.0 browser
2833Mozilla/4.0 (compatible; MSIE 4.01; AOL 4.0; Windows 98)IE 4.x AOL Win98
2834Mozilla/4.0 (compatible; MSIE 4.01; Mac_PowerPC)IE 4.x Mac Power PC
2835Mozilla/4.0 (compatible; MSIE 4.01; MSIECrawler; Windows 95)Internet Explorer 4.0 URL check
2836Mozilla/4.0 (compatible; MSIE 4.01; Vonna.com b o t)Vonna search robot
2837Mozilla/4.0 (compatible; MSIE 4.01; Windows 95)IE 4.x Win95
2838Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; MSN Companion 2.0; 800x600; Compaq)IE PDA Browser Windows CE
2839Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19.123.2.733) OrangeBot-Mobile 2008.0 (mobilesearch.support@orange-ftgroup.com)Orange France robot for mobiles
2840Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPS; 240x320)IE for Windows CE on a PocketPC (HP iPAQ)
2841Mozilla/4.0 (compatible; MSIE 4.01; Windows NT Windows CE)IE PDA Browser Windows CE
2842Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)IE 4.x WinNT
2843Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot) Microsoftdiff. IPs / services i.e.: - Microsoft server information robot (see link) - Okanagan Internet Junction web filter (robot)
2844Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) ACRUnknown robot from American College of Radiology (ACR) running MS Site Server
2845Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Indonesia InteractiveIndonesia Interactive Web-portal robot on MS Site Server
2846Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) WebQuest DesignsWebquestdesigns hosting
2847Mozilla/4.0 (compatible; MSIE 5.01; Windows 95) via Avirt Gateway Server v4.0Avirt Gateway proxy server
2848Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) (samualt9@bigfoot.com)Metacarta.com (66.28.xx.xxx) robot
2849Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; NetCaptor 6.5.0RC1)NetCaptor IE browser addon
2850Mozilla/4.0 (compatible; MSIE 5.0; AOL 5.0; Windows 95; DigExt; Gateway2000; sureseeker.com)IE 5.x AOL Win95 Sureseeker search plugin
2851Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC; AtHome021)IE 5.x Mac PowerPC AtHome user
2852Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com)www.netnose.com crawler
2853Mozilla/4.0 (compatible; MSIE 5.0; Win32) via proxy gateway CERN-HTTPD/3.0 libwww/2.17WinXP via CERN httpd proxy server
2854Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5Echo.com robot
2855Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/)Voila.fr robot
2856Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6Voila.fr robot
2857Mozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en]WinME Opera 5.x
2858Mozilla/4.0 (compatible; MSIE 5.0; Windows ME; Link Checker 2.x.xx http://www.kyosoft.com)Kyosoft's Link Checker
2859Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS AgentBeijing Express Email Address Extractor via DHCP Data Transport Services (DTS)
2860Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)Girafa (browser plug-in) robot
2861Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; www.psychedelix.com)Galaxy robot (63.121.41.xxx)
2862Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; www.psychedelix.com/; http://www.galaxy.com/info/crawler.html)Galaxy robot (63.121.41.xxx)
2863Mozilla/4.0 (compatible; MSIE 5.0; YANDEX)Yandex Search Russia link checking (213.180.206.2xx)
2864Mozilla/4.0 (compatible; MSIE 5.5; AOL 4.0; Windows 98; GoBeez (www.gobeez.com))Gobeez starting page plugin
2865Mozilla/4.0 (compatible; MSIE 5.5; Windows 95; Transmission Segment; Hotbar 2.0)IE 5.5 Win95 Hotbar plug-in
2866Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Crazy Browser 1.x.x)Crazy Browser - IE based tabbed Browser
2867Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; KITV4.7 Wanadoo)Wanadoo Internet services
2868Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; SAFEXPLORER TL)Safexplorer (safexplorer.com - site is offline) kids browser
2869Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; SYMPA; Katiesoft 7; SimulBrowse 3.0)Katiesoft Scroll (ex www.katiesoft.com now discarded) & SimulBrowse (ex www.simulbrowse.com now dead) IE browser plugins
2870Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; BTinternet V8.1)Windows ME BTOpenworld Internet services
2871Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; MSIECrawler)Windows ME Internet Explorer URL check
2872Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)Cobion Germany Brand Protection Services robot
2873Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; QXW03018)Cobion Germany Brand Protection Services robot
2874Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Active Cache RequestIE 5.5 Win2000 / user agent
2875Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API RequestMaybe: - MS Internet Security & Acceleration Server (ISA) cache refreshing request (see link) or - IE 5.5 Win2000 probably with some (website) API request component (see 2nd link) - suspected as email-harvester / site scanning tool (see http://www.byte.com/documents/s=493/byt20010208s0001/index.htm
2876Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; .NET CLR 1.0.3705)IE 5.5 Win2000 with MS.NET SDK
2877Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; AIRF)IE 5.5 Win2000 / user agent w. AI RoboForm (AIRF) password manager
2878Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; AspTear 1.5)AspTear URL fetching program component / Download32.com spider
2879Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; N_o_k_i_a)Nokia.com network
2880Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461) RPT-HTTPClient/0.3-3EUnknown Object Sciences Corp. robot using the HTTPClient
2881Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.singingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revision: 3.10Singingfish media spider (64.12.186.2xx) via AOL search
2882Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows 98; .NET CLR 1.1.4322; MEGAUPLOAD 2.0)Megaupload Mega Manager - Download manager toolbar for IE
2883Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows NT 5.1; SV1; HbTools 4.7.2)Hotbar IE graphical skin
2884Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Skampy/0.9.x [en]Skaffe.com directory link checker
2885Mozilla/4.0 (compatible; MSIE 6.0; TargetSeek/1.0; +http://www.targetgroups.net/TargetSeek.html)TargetSeek Crawler concerning electronics industry product announcements
2886Mozilla/4.0 (compatible; MSIE 6.0; Win32) WebWasher 3.0IE 6.0 WebWasher ad filter
2887Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) REL Software Web Link Validator 2.x)Web Link Validator link validation software
2888Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) Web Link Validator 2.x)Web Link Validator link validation software
2889Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Net M@nager V3.02 - www.vinn.com.au)IE 6.0 Netmanager IE add-on
2890Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; support@illumit.com; http://www.illumit.com/Products/weblight/)WebLight web analyzer & link checker
2891Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; http://www.Abolimba.de)Abolimba Multibrowser - IE based browser
2892Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; Lunascape 2.1.3)Lunascape IE based browser (Japan)
2893Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Google Wireless Transcoder;)Google wireless transcoder (GWT) proxy for rewriting websites for mobiles
2894Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP entries t_st; http://tuezilla.de/t_st-odp-entries-agent.html)Tüzilla (Germany) - ODP link checking using Robozilla
2895Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP links test; http://tuezilla.de/test-odp-links-agent.html)Tüzilla (Germany) - ODP link checking using Robozilla
2896Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ZoomSpider.net bot; .NET CLR 1.1.4322)ZoomSpider.Net indexing robot for several directorys
2897Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) (dns_admin@c-a-s-h.com)unknown robot from 64.246.44.xx
2898Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Covac UPPS Cathan 1.2.5;)Covac Software UPPS (Universal PHP Proxy Server) - free public proxy server
2899Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Crayon Crawler; snprtz|T04056566514940; (R1 1.5))GetNetWise Crayon Crawler web filter
2900Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Deepnet Explorer)Deepnet Explorer - IE based browser
2901Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; heritrix/1.3.0 http://www.cs.washington.edu/research/networking/websys/)Heritrix Internet Archive's open-source web project used by Analysis Projects at UW
2902Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 3.0)IE 6x WinXP Hotbar plug-in
2903Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; iOpus-I-M)IE 6x WinXP iOpus Internet Macros - Internet-based macro recorder
2904Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; iRider 2.21.1108; FDM)iRider - IE based browser / Free Download Manager (FDM)
2905Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; KKman3.0)KKman http://www.kkman.com/ - Japanese IE based browser
2906Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MathPlayer2.0)IE 6x WinXP MathPlayer mathematical notation plugin
2907Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Maxthon)Maxton (ex MyIE2) - IE based browser
2908Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; PeoplePal 3.0; MSIECrawler)IE 6x WinXP peoplepc online PeoplePal IE toolbar
2909Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461; IOpener Release 1.1.04)IE 6x WinXP / I-Opener (was www.netpliance.com/) web PC
2910Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 qihoobot@qihoo.net)Qihoo search (China) robot
2911Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SIMBAR Enabled; InfoPath.1)SimBar IE toolbar for accessing The Sims sites / Infopath IE form & spreadsheet plugin
2912Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; StumbleUpon.com 1.760; .NET CLR 1.1.4322)IE 6x WinXP Stumble Upon IE toolbar
2913Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Embedded Web Browser from: http://bsalsa.com/; MSIECrawler)Balsa Productions embedded web browser package for Borland Delphi
2914Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )ChangeDetection robot for web page monitoring
2915Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)IE 6x WinXP also used by WebSite Pro HTML editor
2916Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; DX-Browser 5.0.0.0)DX-Browser - German IE based browser
2917Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; ezPeer+ v1.0 Beta (0.4.1.98); ezPeer+ v1.0 (0.5.0.00); .NET CLR 1.1.4322; MSIECrawler)ezPeer+ P2P IE addon
2918Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; IBP; .NET CLR 1.1.4322)Axandra IBP website promotion software ?
2919Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MRA 4.3 (build 01218))MRA = Mail.ru Agent - Instant Messenger / VoIP
2920Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)MSN Search robot (207.46.89.xx)
2921Mozilla/4.0 (compatible; MSIE 7.0; Win32) Link Commander 4.0Link Commander bookmark manager
2922Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; bgft)IE 7.0 - WinXP
2923Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB5; User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://bsalsa.com) ; .NET CLR 2.0.50727)GTB = Google Toolbar Internet Explorer add-on
2924Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0)MSIE 7.0 *and* Trident token used by Internet Explorer 8 in compatibility view mode
2925Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)Internet Explorer 8
2926Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)IE 8.0 (beta) on Win Vista
2927Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; Orange 8.0; GTB6.3; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30618; OfficeLiveConnector.1.3; OfficeLivePatch.1.3)Bsalsa embedded browser
2928Mozilla/4.0 (compatible; MSIE enviable; DAUMOA 2.0; DAUM Web Robot; Daum Communications Corp., Korea; +http://ws.daum.net/aboutkr.html)DAUMOA - Daum search Korea robot (211.115.109.xxx)
2929Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.1; DAUM Web Robot; Daum Communications Corp., Korea)DAUMOA - Daum search Korea robot (211.115.109.xxx)
2930Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/delete_main.asp)Naver Search Korea Naverbot
2931Mozilla/4.0 (compatible; Netcraft Web Server Survey)Netcraft webserver info
2932Mozilla/4.0 (compatible; NetPromoter Spider;http://www.net-promoter.com/)NetPromoter Link Utility link checking tool
2933Mozilla/4.0 (compatible; Opera/3.0; Windows 4.10) 3.51 [en]Opera 3.x WinNT
2934Mozilla/4.0 (compatible; Powermarks/3.5; Windows 95/98/2000/NT)Powermarks bookmark manager
2935Mozilla/4.0 (compatible; RSS Popper)RSS Popper - MS Outlook RSS reader plugin
2936Mozilla/4.0 (compatible; SiteKiosk 4.0; MSIE 5.0; Windows 98; SiteCoach 1.0)SiteKiosk public terminal browser
2937Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)Entireweb Search Speedyspider (62.13.25.xxx)
2938Mozilla/4.0 (compatible; SPENG)SiteProbe - website status checking
2939Mozilla/4.0 (compatible; SuperCleaner 2.xx; Windows 98)Super Cleaner privacy tool (bookmark checking)
2940Mozilla/4.0 (compatible; Synapse)Synapse - Apache web service for processing XML documents
2941Mozilla/4.0 (compatible; WebCapture 3.0; Windows)Web2PDF - Adobe Acrobat plugin for site traversal and other services for the Web Capture feature
2942Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)Windows HTTP Services (WinHTTP / XML-parser)
2943Mozilla/4.0 (compatible; WSN Links)WSN Links PHP directory software
2944Mozilla/4.0 (compatible; www.euro-directory.com; urlchecker1.0)Euro Directory (German / Austrian) directory link checking
2945Mozilla/4.0 (compatible; www.galaxy.com)Galaxy robot
2946Mozilla/4.0 (compatible; www.linkguard.com Linkguard Online 1.0; Windows NT)Linkguard.com link validation (service is offline)
2947Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid)Yahoo Search Japan robot (203.141.52.)
2948Mozilla/4.0 (compatible; Yahoo Japan; for robot study; kasugiya)Yahoo Japan robot (202.93.76.xx)
2949Mozilla/4.0 (compatible;MSIE 6.0; Windows NT 5.0; H010818)Faked IE id string used by DeepTrawl link checking tool
2950Mozilla/4.0 (fantomBrowser)spoofed referer by Fantomaster (Multiblocker) anonymity products
2951Mozilla/4.0 (fantomCrew Browser)spoofed referer by Fantomaster (Multiblocker) anonymity products
2952Mozilla/4.0 (hhjhj@yahoo.com)unknown robot from - 64.57.223.40 - 66.28.233.xxx (cogentco.com)
2953Mozilla/4.0 (JemmaTheTourist;http://www.activtourist.com)Activtourist Jemma spider
2954Mozilla/4.0 (MobilePhone PM-8200/US/1.0) NetFront/3.x MMP/2.0NetFront (v3.x) for Pocket PC (here on Sanyo PM-8200 cell phone)
2955Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)Google robot from 66.249.66.xxx
2956Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible; Googlebot/2.1; http://www.google.com/bot.html)Google robot from 66.249.66.xxx
2957Mozilla/4.0 (Mozilla; http://www.mozilla.org/docs/en/bot.html; master@mozilla.com)Unknown robot from Mozilla.org
2958Mozilla/4.0 (Sleek Spider/1.2)ASI - Any Search Info robot
2959Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn.furlbot@looksmart.net)Furl (Looksmart) online bookmark tool robot
2960Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)Wisenut robot
2961Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)Wisenut robot
2962Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)Wisenut robot
2963Mozilla/4.0 compatible ZyBorg/1.0 for Homepage (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)Wisenut robot
2964Mozilla/4.0 efp@gmx.netUnknown robot from 66.230.140.xx (argon.oxeo.com) maybe an e-mail collector
2965Mozilla/4.0 WebTV/2.6 (compatible; MSIE 4.0)WebTV
2966Mozilla/4.0 [en] (Ask Jeeves Corporate Spider)Ask / Ask Jeeves robot
2967Mozilla/4.0(compatible; Zealbot 1.0)LookSmart spider
2968Mozilla/4.01 (compatible; NORAD National Defence Network)HideMe - Web based anonymous proxy server service
2969Mozilla/4.01 [en](Win95;I)Some download manager spoofing Netscape 4.01
2970Mozilla/4.02 [en] (X11; I; SunOS 5.6 sun4u)Netscape 4.x SunOS 5.6
2971Mozilla/4.04 (compatible; Dulance bot; +http://www.dulance.com/bot.jsp)Dulance Bot - Dulance automated price comparison engine
2972Mozilla/4.04 [en] (X11; I; HP-UX B.10.20 9000/712)Netscape 4.x HP-Unix
2973Mozilla/4.04 [en] (X11; I; IRIX 5.3 IP22)Netscape 4.x IRIX
2974Mozilla/4.05 (Macintosh; I; 68K Nav)Netscape 4.x Macintosh 68k
2975Mozilla/4.05 (Macintosh; I; PPC Nav)Netscape 4.x Macintosh PowerPC
2976Mozilla/4.05 [en] (X11; I; SunOS 4.1.4 sun4m)Netscape 4.x SunOS 4.1.4
2977Mozilla/4.08 [en] (Win98; U ;Nav)Version 4.08 [en]-98306
2978Mozilla/4.08 [en] (WinNT; U)Netscape 4.x WinNT
2979Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_TrueRobot/1.4 libwww/5.2.8Echo.com robot
2980Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_VoilaBot/1.6 libwww/5.3.2Voila.fr robot
2981Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)HTTrack Offline Browser
2982Mozilla/4.5 (compatible; iCab 2.5.3; Macintosh; I; PPC)iCab MAC Web browser MAC Power PC
2983Mozilla/4.5 (compatible; OmniWeb/4.0.5; Mac_PowerPC)OmniWeb 4.x.x Mac browser
2984Mozilla/4.5 (compatible; OmniWeb/4.1-beta-1; Mac_PowerPC)OmniWeb 4.x.x Mac browser
2985Mozilla/4.5 RPT-HTTPClient/0.3-2different IPs using the HTTPClient library (mostly link checking)
2986Mozilla/4.5 [en]C-CCK-MCD {RuralNet} (Win98; I)RuralNet Internet Services
2987Mozilla/4.5b1 [en] (X11; I; Linux 2.0.35 i586)Netscape 4.x Linux
2988Mozilla/4.6 [en] (http://www.cnet.com/)Cnet robot for Search.com (216.239.114.xx)
2989Mozilla/4.61 [de] (OS/2; I)Netscape 4.x OS/2
2990Mozilla/4.61 [en] (X11; U; ) - BrowseX (2.0.0 Windows)BrowseX cross-platform browser
2991Mozilla/4.7Nameprotect (12.148.196.128 - 12.148.196.255) snoopbot
2992Mozilla/4.7 (compatible; http://eidetica.com/spider)Eidetica earch and text mining spider
2993Mozilla/4.7 (compatible; Intelliseek; http://www.intelliseek.com)Intelliseek (64.158.138.xx) robot
2994Mozilla/4.7 (compatible; OffByOne; Windows 98) Webster Pro V3.2OffByOne Browser
2995Mozilla/4.7 (compatible; Whizbang)WhizBang! Labs information extraction robot
2996Mozilla/4.7 (compatible; WhizBang; http://www.whizbang.com/crawler)WhizBang! Labs information extraction robot
2997Mozilla/4.7 [en](BecomeBot@exava.com)BecomeBot - Becomecom shopping search (64.124.85.xx(x))
2998Mozilla/4.7 [en](Exabot@exava.com)Exabot - exava shopping search (64.124.85.xx(x))
2999Mozilla/4.7 [en]C-CCK-MCD {Yahoo;YIP052400} (Win95; I)unknown
3000Mozilla/4.72 [en] (BACS http://www.ba.be)http://www.ba.be robot
3001Mozilla/4.72C-CCK-MCD Caldera Systems OpenLinux [en] (X11; U; Linux 2.2.14 i686)Netscpape 4.7x Caldera Open Linux Pentium III
3002Mozilla/4.75C-ja [ja] (X11; U; OSF1 V5.1 alpha)Netscape 4.7x Japan OSF1 alpha
3003Mozilla/4.76 (Windows 98; U) Opera 5.12 [en]Opera 5.x Win 98
3004Mozilla/4.76 [en] (X11; U; FreeBSD 4.4-STABLE i386)Netscape 4.7x FreeBSD
3005Mozilla/4.76 [en] (X11; U; SunOS 5.7 sun4u)Netscape 4.7x SunOS
3006Mozilla/4.77C-SGI [en] (X11; U; IRIX 6.5 IP32)IRIX 6.5
3007Mozilla/5.0GigaMedia / NTT DoCoMo robot
3008Mozilla/5.0 (+http://www.eurekster.com/mammoth) Mammoth/0.1Eurekster Swicki community search using SLI-Systems site search engine Mammoth
3009Mozilla/5.0 (+http://www.sli-systems.com/) Mammoth/0.1SLI Systems mammoth robot
3010Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/)Clush search robot
3011Mozilla/5.0 (compatible) GM RSS Panel XGreasemonkey RSS panel Firefox plugin
3012Mozilla/5.0 (compatible; +http://www.evri.com/evrinid)Evri search robot
3013Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620008 distributed crawler for 80legs
3014Mozilla/5.0 (compatible; Abonti/0.8 - http://www.abonti.com)Abonti WebSearch beta robot
3015Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/)HitCompanies Aihit crawler
3016Mozilla/5.0 (compatible; AnsearchBot/1.x; +http://www.ansearch.com.au/)Ansearch Australian search robot
3017Mozilla/5.0 (compatible; archive.org_bot/1.10.0 +http://www.loc.gov/minerva/crawl.html)The Library of Congress Minerva crawler
3018Mozilla/5.0 (compatible; archive.org_bot/1.13.1x http://crawler.archive.org)Heritrix - The Internet Archive's open-source crawler (207.241.225.2xx)
3019Mozilla/5.0 (compatible; archive.org_bot/1.5.0-200506132127 http://crawler.archive.org) Hurricane KatrinaHeritrix - The Internet Archive's open-source crawler
3020Mozilla/5.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/webmasters.shtml)Ask Jeeves /Teoma robot
3021Mozilla/5.0 (compatible; BanBots/2.0b; Fetch; +http://www.banbots.com)Project BanBots Perl script robot
3022Mozilla/5.0 (compatible; BecomeBot/1.23; http://www.become.com/webmasters.html)BecomeBot - Become.com shopping search (64.124.85.xx(x))
3023Mozilla/5.0 (compatible; BecomeBot/1.xx; MSIE 6.0 compatible; http://www.become.com/webmasters.html)BecomeBot - Become.com shopping search (64.124.85.xx(x))
3024Mozilla/5.0 (compatible; BecomeBot/2.0beta; http://www.become.com/webmasters.html)BecomeBot - Become.com shopping search (64.124.85.xx(x))
3025Mozilla/5.0 (compatible; BecomeBot/2.x; MSIE 6.0 compatible; http://www.become.com/site_owners.html)BecomeBot - Become.com shopping search (64.124.85.xx(x))
3026Mozilla/5.0 (compatible; BecomeJPBot/2.3; MSIE 6.0 compatible; +http://www.become.co.jp/site_owners.html)BecomeBot - Become.com shopping search (64.124.85.xx(x))
3027Mozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers)BlogRefsBot.com blog robot
3028Mozilla/5.0 (compatible; Bot; +http://pressemitteilung.ws/spamfilterPressemitteilungen Webservice RSS / news crawler (Germany)
3029Mozilla/5.0 (compatible; BuzzRankingBot/1.0; +http://www.buzzrankingbot.com/)BuzzRanking internet content analysis
3030Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@betaspider.com)Charlotte indexing spider for Searchme / Wikiseek
3031Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.searchme.com/support/)Charlotte indexing spider for Searchme / Wikiseek
3032Mozilla/5.0 (compatible; Crawling jpeg; http://www.yama.info.waseda.ac.jp)Unknown graphics crawler or downloading agent from Yamana Laboratory - Waseda Univerity Japan (133.9.238.xx)
3033Mozilla/5.0 (compatible; Custo 3 (Netwu.com); Windows NT 5.1)Custo web site spidering tool (link checking)
3034Mozilla/5.0 (compatible; de/1.13.2 +http://www.de.com)De.com German travel related search via Amazon Web Services
3035Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)Diffbot beta - RSS and news feed crawler
3036Mozilla/5.0 (compatible; DNS-Digger-Explorer/1.0; +http://www.dnsdigger.com)DNS-Digger - DNS server neighbourhood search
3037Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com)DNS-Digger - DNS server neighbourhood search
3038Mozilla/5.0 (compatible; EARTHCOM.info/2.01; http://www.earthcom.info)Earthcom (Czech Republic) search robot (194.108.39.xx)
3039Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)enter4u / Earthcom.info search (Czech Republic)
3040Mozilla/5.0 (compatible; egothor/8.0g; +http://ego.ms.mff.cuni.cz/)Prague Faculty of Mathematics and Physics using Egothor open source crawler
3041Mozilla/5.0 (compatible; Exabot Test/3.0; +http://www.exabot.com/go/robot)Exalead (France) search robot (193.47.80.xx)
3042Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/main/CrawlerFAQs.fhtml)TheFind.com - Shopping search robot
3043Mozilla/5.0 (compatible; Galbot/1.0; +http://www.galbot.com/bot.html)Galbot tagging robot (beta) - Denmark
3044mozilla/5.0 (compatible; genevabot http://www.healthdash.com)Geneva Single-Site Search Engine used by Healthdash health search
3045Mozilla/5.0 (compatible; Google Desktop) Paros/3.2.12Paros - a Java based HTTP/HTTPS proxy
3046Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)Google robot
3047mozilla/5.0 (compatible; heritrix/1.0.4 http://innovationblog.com)Unknown robot using Heritrix
3048Mozilla/5.0 (compatible; heritrix/1.10.2 +http://i.stanford.edu/)The Stanford University InfoLab robot using Heritrix
3049Mozilla/5.0 (compatible; heritrix/1.12.1 +http://newstin.com/)Newstin news feed search using Heritrix
3050Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com)Page-store.com vertical search via Amazon Web Services
3051Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) [email:paul@page-store.com]Page-store.com vertical search via Amazon Web Services
3052mozilla/5.0 (compatible; heritrix/1.3.0 http://archive.crawler.org)Heritrix Internet Archive's open-source web project
3053Mozilla/5.0 (compatible; heritrix/1.4.0 +http://www.chepi.net)Chepi Beta search Spain (194.116.240.1xx) using Heritrix
3054Mozilla/5.0 (compatible; heritrix/1.4t http://www.truveo.com/)Truveo data mining robot using Heritrix
3055Mozilla/5.0 (compatible; heritrix/1.5.0 http://www.l3s.de/~kohlschuetter/projects/crawling/)L3S WebCrawling Project (Germany) using Heritrix
3056Mozilla/5.0 (compatible; heritrix/1.5.0-200506231921 http://pandora.nla.gov.au/crawl.html)Pandora Internet Archive crawler (Australia) using Heritrix
3057Mozilla/5.0 (compatible; heritrix/1.6.0 http://www.worio.com/)WORIO (beta) search for computer scientists and programmers using Heritrix open-source crawler
3058Mozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/)greatarea.com website collection project using Heritrix
3059Mozilla/5.0 (compatible; Heritrix/1.8.0 http://www.hanzoarchives.com)hanzo:web social web archiving service
3060Mozilla/5.0 (compatible; heritrix/1.x.x +http://www.accelobot.com)Accelobot - Accelovation Market Discovery software robot
3061Mozilla/5.0 (compatible; heritrix/2.0.0-RC1 +http://www.aol.com)Unknown AOL robot using Heritrix
3062Mozilla/5.0 (compatible; Hermit Search. Com; +http://www.hermitsearch.com)Hermits Search.com - Products and service search robot
3063Mozilla/5.0 (compatible; http://www.IsMySiteUp.Net/bot/ )IsMySiteUp? - Online website monitoring service
3064Mozilla/5.0 (compatible; http://www.UptimeAuditor.com/bot/ )UptimeAuditor - real time web monitoring
3065Mozilla/5.0 (compatible; HyperixScoop/1.3; +http://www.hyperix.com)Hyperix vertical search crawler
3066Mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0)Unknown robot (reads robots.txt) from chinatelecom (219.142.78.xx)
3067Mozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html)ID-Search.org - Russian search project
3068Mozilla/5.0 (compatible; InterseekWeb/3.x)Najdi.si (Slovenia) search using Interseek/Web Interseek/API Search Engine
3069Mozilla/5.0 (compatible; Jim +http://www.hanzoarchives.com)hanzo:web social web archiving service
3070Mozilla/5.0 (compatible; Konqueror/2.0.1; X11); Supports MD5-Digest; Supports gzip encodingKonqueror 2.0.x X11
3071Mozilla/5.0 (compatible; Konqueror/2.1.1; X11)Konqueror 2.1.x X11
3072Mozilla/5.0 (compatible; Konqueror/2.2.2)Konqueror 2.2.x
3073Mozilla/5.0 (compatible; Konqueror/2.2.2; Linux 2.4.14-xfs; X11; i686)Konqueror 2.2.x Linux
3074Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)Exalead (France) search robot (193.47.80.xx)
3075Mozilla/5.0 (compatible; LemSpider 0.1)Lemur Consulting LemIR spider
3076Mozilla/5.0 (compatible; LinksManager.com_bot http://linksmanager.com/linkchecker.html)Linksmanager.com online link checking service
3077Mozilla/5.0 (compatible; LinkStash Bookmark Manager; http://www.xrayz.co.uk/)LinkStash Bookmark Manager
3078Mozilla/5.0 (compatible; MojeekBot/2.0; http://www.mojeek.com/bot.html)Mojeek Search Preview robot (217.155.205.xx)
3079Mozilla/5.0 (compatible; MOSBookmarks/v2.6-Plus; Link Checker)Joomla!/Mambo component - MosBookmarks (bot) link checking
3080Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; crawler_admin@podtech.net)PodTech entertainment and video network crawler
3081Mozilla/5.0 (compatible; OnetSzukaj/5.0; http://szukaj.onet.pl)onet.pl Szukaj (Search) robot (213.180.128.1xx)
3082Mozilla/5.0 (compatible; PagestackerBot; http://www.pagestacker.com)Pagestacker online bookmark service
3083Mozilla/5.0 (compatible; PalmeraBot; http://www.links24h.com/help/palmera) Version 0.001PalmeraBot - Links24h.com search engine robot
3084Mozilla/5.0 (compatible; PEAR HTTP_Request class; http://feed.moo.jp/)FeedMo feed search (Japan) using Pear HTTP
3085Mozilla/5.0 (compatible; Phonifier; +http://www.phonifier.com)PHONifier mobile access to web content
3086Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)pmoz.info ODP link checking bot
3087Mozilla/5.0 (compatible; pogodak.ba/3.x)Pogodak search (Slovenia) robot via Interseek
3088Mozilla/5.0 (compatible; Pogodak.hr/3.1)Pogodak search (Slovenia) robot via Interseek
3089Mozilla/5.0 (compatible; Proximic crawler; +http://www.proximic.com/en/about-us/contact-us.html)Proximic Publisher Widget - RSS and news content generator
3090Mozilla/5.0 (compatible; PWeBot/3.1; http://www.programacionweb.net/robot.php)ProgramacionWeb.net PWeBot crawler (Argentina)
3091Mozilla/5.0 (compatible; Quantcastbot/1.0; www.quantcast.com)Quantcast - Open Internet Ratings Service
3092Mozilla/5.0 (compatible; robtexbot/1.0; http://www.robtex.com/ )robtex - Multi-RBL check and AS-numbercheck
3093Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)ScoutJet (Blekko) search web crawler
3094Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)Scrub the web robot (66.93.156.xx)
3095Mozilla/5.0 (compatible; ShunixBot/1.x.x +http://www.shunix.com/robot.htm)Shunixbot (France) beta / test semantic web indexing robot
3096Mozilla/5.0 (compatible; ShunixBot/1.x; http://www.shunix.com/bot.htm)Shunixbot (France) beta / test semantic web indexing robot
3097Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com)Skreemr - Audio search engine
3098Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9Snap Firefox Search Plugin
3099Mozilla/5.0 (compatible; SpurlBot/0.2)Spurl.net bookmark service & search engine (84.40.30.xxx)
3100Mozilla/5.0 (compatible; SummizeBot +http://www.summize.com)Summize - Opinion and review search robot
3101Mozilla/5.0 (compatible; SYCLIKControl/LinkChecker;)Syclik Control web content management system
3102Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.html)Synoo web directory robot
3103Mozilla/5.0 (compatible; Theophrastus/x.x; http://users.cs.cf.ac.uk/N.A.Smith/theophrastus.php)Theophrastus Internet Spider for a basic search engine project
3104Mozilla/5.0 (compatible; TridentSpider/3.1)Interseek - Java search engine technology used for Pogodak search
3105Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)WiseGuys robot
3106Mozilla/5.0 (compatible; Webduniabot/1.0; +http://search.webdunia.com/bot.aspx)Webdunia search (India) robot
3107Mozilla/5.0 (compatible; Windows NT 5.0; phpwebbrainBot/0.1 - http://www.monsterli.ch/phpwebbrain/)phpwebbrain online bookmark service (Germany)
3108Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com)WORIO (beta) search for computer scientists and programmers using Heritrix open-source crawler
3109Mozilla/5.0 (compatible; WoW Lemmings Kathune/2.0;http://www.wowlemmings.com/kathune.html)Kathune spider for World of Warcraft guild data. Used to power WoW Lemmings
3110Mozilla/5.0 (compatible; XTbot/1.0v; +http://www.externaltest.com)eXternalTest - Server and online services monitoring
3111Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/slurp)Yahoo / Inktomi search robot
3112Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)Inktomi robot (202.160.180.xxx) for Yahoo China
3113Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)Inktomi robot for Yahoo (via 66.196.xx.xxx)
3114Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html)Yesup Seo - Toronto SEO Service
3115Mozilla/5.0 (compatible; Yoono; http://www.yoono.com/)Yoono - community based search (193.110.140.xxx / 194.0.179.[x]xx)
3116Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )Youdao search (China) robot
3117Mozilla/5.0 (compatible; Zenbot/1.3; +http://zen.co.za/webmasters/)Zenbot robot for the Southern African Zen search service
3118Mozilla/5.0 (compatible; zermelo +http://www.powerset.com) [email:paul@page-store.com,crawl@powerset.com]Powerset Natural Language Search crawler (under development) using Heritrix via Amazon Web Services
3119Mozilla/5.0 (compatible;archive.org_bot/1.7.1; collectionId=316; Archive-It; +http://www.archive-it.org)Heritrix - The Internet Archive's open-source crawler (207.241.225.2xx)
3120Mozilla/5.0 (compatible;archive.org_bot/heritrix-1.9.0-200608171144 +http://pandora.nla.gov.au/crawl.html)Wayback Machine Internet Archive crawler
3121Mozilla/5.0 (compatible;FindITAnswersbot/1.0;+http://search.it-influentials.com/bot.htm)FindITAnswers - Search engine for software developers
3122Mozilla/5.0 (compatible;MAINSEEK_BOT)Mainseek search (Poland) robot
3123Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; http://mozshot.nemui.org/)MozShot - Technical demo to take screenshot of any URL
3124Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b5) Gecko/2008032619 Firefox/3.0b5Mozilla Firefox 3.0 beta (Gran Paradiso) for MacOS
3125Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.0.1) Gecko/20021219 Chimera/0.6Chimera browser (Mozilla/Gecko engine) - now Camino Mac PowerPC
3126Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.0.1) Gecko/20030306 Camino/0.7Camino browser (Mozilla/Gecko engine) - ex Chimera Mac PowerPC
3127Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US) AppleWebKit/xx (KHTML like Gecko) OmniWeb/v5xx.xxOmniWeb 5.x.x Mac OS X browser
3128Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) AppleWebKit/xxx.x (KHTML like Gecko) Safari/12x.xSafari 1.2x browser (Mozilla/Gecko engine) MAC OS X
3129Mozilla/5.0 (Macintosh; U; PPC; en-US; rv:0.9.2) Gecko/20010726 Netscape6/6.1Netscape 6.x Mac PowerPC
3130Mozilla/5.0 (research@mediatrec.com)unknown robot from gw.ocg-corp.com
3131Mozilla/5.0 (Sage)Sage - RSS and Atom feed reader extension for Mozilla Firefox
3132Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)Inktomi (Hotbot-Lycos - NBCi etc.) robot
3133Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)Inktomi (Hotbot-Lycos - NBCi etc.) robot
3134Mozilla/5.0 (SunOS 5.8 sun4u; U) Opera 5.0 [en]Opera 5.x SunOS
3135Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)Twiceler experimental web crawler
3136Mozilla/5.0 (Version: xxxx Type:xx)Some spambot from Romania (82.208.139.1xx & 86.123.65.xx) - Maybe email harvesting
3137Mozilla/5.0 (wgao@genieknows.com)GenieKnows.com search robot (64.5.245.xx / 64.5.220.xxx)
3138Mozilla/5.0 (Windows; U; Win98; en-US; rv:0.9.2) Gecko/20010726 Netscape6/6.1Netscape 6.x Win98
3139Mozilla/5.0 (Windows; U; Win98; en-US; rv:x.xx) Gecko/20030423 Firebird Browser/0.6Firebird browser (Mozilla/Gecko engine) - ex Phoenix Win98
3140Mozilla/5.0 (Windows; U; Win9x; en; Stable) Gecko/20020911 Beonex/0.8.1-stableBeonex Communicator browser (Mozilla/Gecko engine)
3141Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.7) NimbleCrawler 1.11 obeys UserAgent NimbleCrawler For problems contact: crawler_at_dataalchemy.comHealthline health related search robot (72.5.115.xx)
3142Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/0.2.153.1 Safari/525.19Google Chrome browser based on WebKit (Safari)
3143Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.5) Gecko/20060731 Firefox/1.5.0.5 Flock/0.7.4.1Flock web browser built on Mozilla technologies
3144Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.4/Megaupload x.0Megaupload Mega Manager - Download manager plugin for Firefox
3145Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008092215 Firefox/3.0.1 Orca/1.1 beta 3Orca browser - based on Gecko
3146Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.x.x) Gecko/20041107 Firefox/x.xFirefox browser (Mozilla/Gecko engine) - ex Firebird WinXP
3147Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.xx) Gecko/20030504 Mozilla Firebird/0.6Firebird browser (Mozilla/Gecko engine) - ex Phoenix WinXP
3148Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.xxx) Gecko/20041027 Mnenhy/0.6.0.104Mnenhy - enhanced mail & news Mozilla based browser
3149Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)Voila.fr robot
3150Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5Mozilla Firefox 3.0 beta (Gran Paradiso) for Win
3151Mozilla/5.0 (Windows; U;XMPP Tiscali Communicator v.10.0.1; Windows NT 5.1; it; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3Tiscali Communicator - Online services suite
3152Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For problems contact: crawler@healthHealthline health related search robot (72.5.115.xx)
3153Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.comHealthline health related search robot (72.5.115.xx)
3154Mozilla/5.0 (X11; Linux i686; U;rv: 1.7.13) Gecko/20070322 Kazehakase/0.4.4.1Kazehakase - Gecko based browser (Japan)
3155Mozilla/5.0 (X11; U; Linux 2.4.2-2 i586; en-US; m18) Gecko/20010131 Netscape6/6.01Netscape 6.x Linux
3156Mozilla/5.0 (X11; U; Linux i686; de-AT; rv:1.8.0.2) Gecko/20060309 SeaMonkey/1.0SeaMonkey browser suite (ex Mozilla) on Linux
3157Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.7.6) Gecko/20050405 Epiphany/1.6.1 (Ubuntu) (Ubuntu package 1.0.2)Epiphany (Mozilla/Gecko engine) browser Linux
3158Mozilla/5.0 (X11; U; Linux i686; en-US; Nautilus/1.0Final) Gecko/20020408Nautilus (developed by Eazel.com) 1.x Browser Linux
3159Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.3) Gecko/20010801Mozilla (Gecko) 0.9x browser Linux
3160Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/20021130Spinn3r social network crawler
3161Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2b) Gecko/20021007 Phoenix/0.3Phoenix 0.3 browser (Mozilla/Gecko engine) - now Firebird Linux
3162Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040413 Epiphany/1.2.1Epiphany (Mozilla/Gecko engine) browser Linux
3163Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBotSnap Firefox Search Plugin
3164Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1) Gecko/20061129 BonEcho/2.0Bon Echo Alpha - developer preview of future Firefox browser
3165Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.1) Gecko/20061205 Iceweasel/2.0.0.1 (Debian-2.0.0.1+dfsg-2)IceWeasel - the GNU version of the Firefox browser
3166Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9a8) Gecko/2007100619 GranParadiso/3.0a8Mozilla Firefox 3.0 beta (Gran Paradiso) for Linux
3167Mozilla/5.0 Galeon/1.0.2 (X11; Linux i686; U;) Gecko/20011224Galeon 1.x Browser Linux
3168Mozilla/5.0 gURLChecker/0.x.x (Linux)gURLChecker - GNOME link checking tool
3169Mozilla/5.0 URL-SpiderURL Spider - used by usww.net
3170Mozilla/5.0 usww.com-Spider-for-w8.netW8net spider
3171Mozilla/5.0 wgao@genieknows.comGenieKnows.com search robot (64.5.245.xx / 64.5.220.xxx)
3172Mozilla/5.0 whoiam [http://www.axxus.de/]axxus.de German business directory
3173Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)Yuntis : Collaborative Web Resource Categorization and Ranking Project robot
3174MQbot metaquerier.cs.uiuc.edu/crawlerMetaExplorer project's MetaQuerier robot
3175MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu; mqbot@cs.uiuc.edu)MetaExplorer project's MetaQuerier robot
3176MSFrontPage/4.0MS Frontpage 4.x
3177MSIE 4.0 (Win95)Some faked UA - maybe for a download manager
3178MSIE-5.13 (larbin@unspecified.mail)unknown robot from gw.ocg-corp.com (209.126.176.x)
3179msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)MSN media search robot
3180msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)Windows Live product search (Beta) robot
3181MSNBOT/0.xx (http://search.msn.com/msnbot.htm)MSN Search robot - 131.107.xxx.xxx 204.95.96.xxx - 204.95.111.xxx 207.46.xxx.xxx
3182msnbot/x.xx ( http://search.msn.com/msnbot.htm)MSN Search robot - 131.107.xxx.xxx 204.95.96.xxx - 204.95.111.xxx 207.46.xxx.xxx
3183MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default)Microsoft search for mobiles
3184MSNPTC/1.0MSN Search robot - 131.107.xxx.xxx 204.95.96.xxx - 204.95.111.xxx 207.46.xxx.xxx
3185MSProxy/2.0Microsoft proxy server
3186MSRBOTMacEdition CodeBitch link checking
3187MSRBOT (http://research.microsoft.com/research/sv/msrbot)Microsoft MSRBot
3188Mulder, VCR-1.0StreamBox VCR user agent
3189multiBlocker browserMultiblocker (Fantomaster) anonymity software user
3190multicrawler ( http://sw.deri.org/2006/04/multicrawler/robots.html)MultiCrawler for DERI Galway's Semantic Web Search Engine cluster
3191MultiText/0.1Virginia Tech Digital Library Research Laboratory robot
3192MusicWalker2.0 ( http://www.somusical.com)SoMusical! musical directory link checking
3193MVAClientUnknown bad bot from diff. Taiwanese IPs
3194My WinHTTP ConnectionWindows HTTP Services (WinHTTP)
3195myDaemonunknown user robot (24.124.34.42)
3196MyGetRight/1.0.0GetRight download manager
3197MyGetRight/1.0bGetRight download manager
3198Mylinea.com Crawler 2.0Mylinea France web catalogue crawler
3199mylinkcheck/1.02VDOG - SEO webdirecory (Germany) link checking
3200Naamah 1.0.1/Blogbot (http://blogbot.de/)Blogbot (Germany) robot
3201Naamah 1.0a/Blogbot (http://blogbot.de/)Blogbot (Germany) robot
3202NABOT/5.0Naver Japan / Korea robot
3203nabot_1.0Naver Japan / Korea robot
3204NameOfAgent (CMS Spider)Badbot searching for Wordpress wp-login.php
3205naoFavicon4IE/1.xxnaoFavicon4IE
3206NASA Search 1.0Unknown spambot / harvester from diff. IPs
3207NationalDirectory-WebSpider/1.3Nationaldirectory spider
3208NationalDirectoryAddURL/1.0Nationaldirectory spider
3209NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)Naver Japan / Korea robot
3210NaverBot_dloader/1.5Naver Japan / Korea robot
3211NavissoBotNavisso closed beta robot (69.41.162.1xx)
3212NavissoBot/1.7 (+http://navisso.com/)Navisso closed beta robot (69.41.162.1xx)
3213NCSA Beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html)Vias Information Archival robot
3214Nebullabot/2.2 (http://bot.nebulla.info)Nebulla.info distributed crawler (Germany)
3215NEC Research Agent -- compuman at research.nj.nec.comNEC Researchindex robot - now CiteSeer.IST scientific document index
3216NEC-Hayek/1.0rcn.com user agent ? NEC Researchindex robot ?
3217Net-Seekr Bot/Net-Seekr Bot V1 (http://www.net-seekr.com)Net Seekr search robot
3218NetAnts/1.2xNetAnts download manager
3219NETCOMplete/x.xxNetComplete IE browser package
3220NetinfoBot/1.0 (http://netinfo.bg/netinfobot.html)Netinfo.bg search (Bulgaria) robot
3221NetLookout/2.24Netlookout internet notifier
3222Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don't___spam_me_@netluchs.de)Netluchs (Germany) search (193.164.8.xx)
3223NetMechanic Vx.0NetMechanic link checker
3224NetNewsWire/2.x (Mac OS X; http://ranchero.com/netnewswire/)NewsGator NetNewsWire - Mac RSS feed reader
3225NetNoseCrawler/v1.0unknown InCom (216.0.107.xx) robot
3226Netprospector JavaCrawlerNetprospector metasearch software
3227NetPumper/x.xxNetpumper download manager
3228NetResearchServer(http://www.look.com)Look.com robot (209.87.232.x)
3229NetResearchServer/x.x(loopimprovements.com/robot.html)IncyWincy search engine using DMOZ database
3230NetSeer/Nutch-0.9 (NetSeer Crawler; http://www.netseer.com; crawler@netseer.com)NetSeer search (beta) crawler via Amazon Web Services - see also Teemer
3231NetSprint -- 2.0Wirtualna Polska / Netsprint search (Poland) robot
3232NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com; support@netwhat.com)NetWhat Search crawler (69.9.167.1xx)
3233NetZippyNetzippy robot
3234NeuralBot/0.2unknown
3235newsearchengine (ThisUser@unspecified.mail)Unknown (12.238.4.xxx) attbi.com client robot
3236NewsGator FetchLinks extension/0.2.0 (http://graemef.com)FetchLinks plugin for NewsGator RSS reader
3237NewsGatorOnline/2.0 (http://www.newsgator.com; 1 subscribers)NewsGator online RSS reader
3238NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot)Eliyon Crawler for Business People Search
3239NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v0.xNextopia crawler
3240NG-Search/0.90 (NG-SearchBot; http://www.ng-search.com; )find your keywords - semantic search (Germany) robot
3241NG/1.0Exalead (France) search robot (193.47.80.xx)
3242NG/4.0.1229Exalead Websearch image crawler (193.47.80.xx)
3243nicebotUnknown UA from PlanetLab distributed network
3244NICO/1.0NicoZone childsafe search robot
3245Nikita the Spider (http://NikitaTheSpider.com/)Nikita the Spider - Online HTML validation , link checking
3246NITLE Blog Spider/0.01Experimental LSI (?) robot from 140.233.69.xx (Middlebury.edu)
3247Nitro Downloader 1.x (www.klsofttools.com)Download manager
3248Noago SpiderNoago spider
3249Nocilla/1.0telefonica.es user robot
3250Nokia-WAPToolkit/1.2 googlebot(at)googlebot.comGoogle WAP robot
3251Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi http://find.mobi/bot.html abuse@mtld.mobi)Botmobi crawler for Find.mobi mobile search
3252Nokia6610/1.0 (3.09) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)YahooSeeker/M1A1-R2D2 - Yahoo mobile web crawling robot
3253Nokia7110/1.0 (05.01) (Google WAP Proxy/1.0)Google WAP proxy
3254NokodoBot/1.x (+http://nokodo.com/bot.htm)Nokodo public beta search robot (67.18.222.xx)
3255Norbert the Spider(Burf.com)Burf.com UK Search Engine robot
3256noxtrumbot/1.0 (crawler@noxtrum.com)noXtrum search robot (Spain)
3257noyona_0_1Noyona job search (preview)
3258NP/0.1 (NP; http://www.nameprotect.com; npbot@nameprotect.com)Nameprotect copyright search robot (24.177.134.x)
3259NPBot (http://www.nameprotect.com/botinfo.html)Nameprotect copyright search robot (24.177.134.x)
3260NPBot-1/2.0Nameprotect copyright search robot (24.177.134.x)
3261Nsauditor/1.xNsauditor Network Security Auditor
3262NSPlayer/10.0.0.xxxx WMFSDK/10.0NetShow Media Player = Windows Media Player 10
3263nsyght.com/Nutch-1.0-dev (nsyght.com; Nsyght.com)Nsyght social search application
3264nsyght.com/Nutch-x.x (nsyght.com; search.nsyght.com)Nsyght social search application
3265nttdirectory_robot/0.9 (super-robot@super.navi.ocn.ne.jp)NTT Directory robot
3266Nucleus SiteList LinkChecker/1.1Nucleus CMS SiteList link managing plugin
3267nuSearch Spider www.nusearch.com (compatible; MSIE 4.01)nuSearch spider (84.9.136.xxx)
3268NuSearch Spider (compatible; MSIE 6.0)nuSearch spider (84.9.136.xxx)
3269NuSearch Spider www.nusearch.comnuSearch spider (84.9.136.xxx)
3270NutchNutch open source robot
3271Nutch crawler/Nutch-0.9 (picapage.com; admin@picapage.com)Picapage search for handheld devices using Nutch
3272Nutch/Nutch-0.9 (Eurobot; http://www.ayell.eu )Ayell Euronet business directory robot using Nutch
3273NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)Netsweeper content filtering engine (66.207.120.2xx) powered by Nutch
3274NutchCVS/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)Nutch open source robot
3275NutchCVS/0.7.1 (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu)Robot from University of Washington Computer Science & Engineering (128.208.6.2xx)
3276NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.org/nutch/bot.html; ec2test at lucene.com)Amazon Elastic Compute Cloud (Amazon EC2) robot
3277NutchOrg/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)Nutch open source robot
3278nutchsearch/Nutch-0.9 (Nutch Search 1.0; herceg_novi at yahoo dot com)Unknown robot using Nutch (maybe private crawling) via Cox network (70.187.130.25x)
3279NutchVinegarCrawl/Nutch-0.8.1 (Vinegar; http://www.cs.washington.edu; eytanadar at gmail dot com)Unknown crawler from University of Washington - Computer science
3280obidos-bot (just looking for books.)Weblog bookwatch robot
3281ObjectsSearch/0.01-dev (ObjectsSearch;http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com)Objects Search robot
3282ObjectsSearch/0.0x (ObjectsSearch; http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com)Objects Search robot
3283oBot ((compatible;Win32))Cobion Germany Brand Protection Services robot
3284Ocelli/1.x (http://www.globalspec.com/Ocelli)GlobalSpec Engineering Search robot (66.194.55.xxx)
3285OctopusOctopus download manager
3286Octora Beta - www.octora.comOctora blog or RSS information crawler - beta (66.228.114.xx)
3287Octora Beta Bot - www.octora.comOctora RSS feed search
3288Offline Explorer 1.*Meta Products Offlinebrowser
3289OliverPerryClaymont robot / Internetseer Web Site Monitoring
3290OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Internet CategorizerOmniExplorer http://www.omni-explorer.com/ car & shopping search (64.62.175.xxx)OmniExplorer car & shopping search (64.62.175.xxx)
3291OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Job CrawlerOmniExplorer car & shopping search (64.62.175.xxx)
3292OmniExplorer_Bot/1.1x (+http://www.omni-explorer.com) Torrent CrawlerOmniExplorer car & shopping search (64.62.175.xxx)
3293OmniExplorer_Bot/x.xx (+http://www.omni-explorer.com) WorldIndexerOmniExplorer car & shopping search (64.62.175.xxx)
3294onCHECK Linkchecker von www.scientec.de fuer www.onsinn.deonsearch.de German web directory link checking
3295onCHECK-Robot, www.onsearch.deonsearch.de German web directory link checking
3296Onet.pl SA- http://szukaj.onet.plonet.pl Szukaj (Search) robot (213.180.128.1xx)
3297online link validator (http://www.dead-links.com/)Dead-Links.com link validation spider
3298Online24-Bot (Version: 1.0x, powered by www.online24.de)Online24 shopping portal (Germany) link checking
3299OntoSpider/1.0 libwww-perl/5.65OntoSpider - Dutch robot for a research project. (195.11.244.xx)
3300OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )SeetooZ search crawler
3301OpenAcoon v4.0.x (www.openacoon.de)OpenAcoon open source search engine (used by Acoon search)
3302Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)Openfind.com.tw robot
3303Openfind data gatherer- Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)Openfind.com.tw robot
3304Openfind Robot/1.1A2Openfind.com.tw robot
3305OpenISearch/1.x (www.openisearch.com)open i search robot - search engine in development
3306OpenTaggerBot (http://www.opentagger.com/opentaggerbot.htm)Opentagger social bookmarking system
3307OpenTextSiteCrawler/2.9.2OpenText crawler
3308OpenWebSpider/0.x.x (http://www.openwebspider.org)OpenWebSpider - Open Source web search engine
3309OpenWebSpider/xOpenWebSpider - Open Source web search engine
3310Opera/5.0 (Linux 2.0.38 i386; U) [en]Opera 5.0 Linux
3311Opera/5.11 (Windows ME; U) [ru]Opera 5.11 faked WinME referer
3312Opera/5.12 (Windows 98; U) [en]Opera 5.12 Win98
3313Opera/6.01 (larbin@unspecified.mail)unknown robot from gw.ocg-corp.com (209.126.176.x)
3314Opera/6.x (Linux 2.4.8-26mdk i686; U) [en]Opera 6.x- Mandrake Linux
3315Opera/6.x (Windows NT 4.0; U) [de]Opera 6.x WinNT
3316Opera/7.x (Windows NT 5.1; U) [en]Opera 7.x WinXP
3317Opera/8.xx (Windows NT 5.1; U; en)Opera 8.x (Beta) WinXP
3318Opera/9.0 (Windows NT 5.1; U; en)Opera 9 (Beta) Browser
3319Opera/9.00 (Windows NT 5.1; U; de)Opera 9 (final)
3320Opera/9.60 (Windows NT 5.1; U; de) Presto/2.1.1Opera browser 9.6x on WinXP (Presto = Operas rendering engine)
3321OpidooBOT (larbin2.6.3@unspecified.mail)Opidoo Search Belgium robot (62.4.83.xxx)
3322OPWV-SDK UP.Browser/7.0.2.3.119 (GUI) MMP/2.0 Push/POOpen Wave Phone Simulator SDK
3323Oracle Application Server Web Cache 10gOracle Application Server cache
3324Oracle iMTCrawlerOracle interMedia Text - Text and web documents indexing
3325Oracle Ultra SearchOracle Search
3326OrangeSpiderOrangeslicer semantic search (Beta) Germany (193.201.52.1xx)
3327Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm)Orbiter - DailyOrbit search spider
3328Orca Browser (http://www.orcabrowser.com)Orca browser - based on Gecko
3329OSSProxy 1.3.305.321 (Build 305.321 Win32 en-us)(Dec 21 2005 16:30:54)Marketscore (was Netsetter) internet accelerator
3330OutfoxBot/0.x (For internet experiments; http://; outfox.agent@gmail.com)Unknown robot from Chinanet (220.181.8.xxx)
3331OutfoxMelonBot/0.5 (for internet experiments; http://; outfoxbot@gmail.com)Unknown robot from Chinanet (60.191.80.1)
3332Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)Overture/Fast/Alltheweb crawler (66.77.73.xxx)
3333OWR_Crawler 0.1Unknown robot from 198.169.127.xx (innovationplace.com)
3334ozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; ozelot@flying-cat.de)Ozelot - Flying Cat's search engine robot (Germany)
3335PADLibrary SpiderPADLibrary.com - PAD file software robot for FindFiles.com
3336PageBitesHyperBot/600 (http://www.pagebites.com/)Pagebites job search crawler
3337Pagebull http://www.pagebull.com/Pagebull visual search engine
3338Pagestacker BotPagestacker online bookmark service
3339page_verifier (http://www.securecomputing.com/goto/pv)Secure Computing SmartFilter Tools - malware crawler
3340PagmIEDownloadDownloadmanager ?
3341parallelContextFocusCrawler1.1parallelContextFocusCrawler1.1CFC crawler used by Italian academic and research network (GARR)
3342ParaSite/1.0b (http://www.ianett.com/parasite/)http://www.ianett.com robot
3343Patwebbot (http://www.herz-power.de/technik.html)Patsearch (Germany) robot
3344pavuk/0.9pl29b i686-pc-linux-gnuPavuk web downloading program for Unix
3345PBrowse 1.4bSome site scanning tool via diff. IPs- i.e.: - cox.net (68.4.xxx.xxx)
3346pd02_1.0.0 pd02_1.0.0@dzimi@post.skPost.sk / Eurotel.sk robot
3347PEAR HTTP_Request class ( http://pear.php.net/ )Pear HTTP_Request PHP extension package
3348PEERbot www.peerbot.comPeerbot - favicon search robot
3349PeopleChat/Search_EngineUnknown robot from 64.5.48.xxx (Plethoric.net)
3350PEval 1.4bSome site scanning tool via diff. IPs
3351PHP/3.x.xxdiff. IPs / services
3352PHP/4.0.4pl1diff. IPs / services
3353PHP/4.0.6diff. IPs / services- i.e.: -NTT/Verio Inc. link checker
3354PHP/4.1.1diff. IPs / services- i.e.: - Phenominet.com link checking
3355PHP/4.1.2diff. IPs / services- i.e.: - 209.114.200.xx = MyNetCrawler link checking - 216.139.207.xxx = Mixcat crawler
3356PicoSearch/1.0Pico Search robot
3357Piffany_Web_Scraper_v0.xPiffany targeted search web spider
3358Piffany_Web_Spider_v0.xPiffany targeted search web spider
3359PigeonBot1.0 BETAWhois Source domain name information robot (66.249.26.xx)- s.also: - SurveyBot
3360PingALink Monitoring Services 1.0PingALink website monitoring
3361PingALink Monitoring Services 1.0 (http://www.pingalink.com)PingALink website monitoring
3362Pingdom GIGRIB (http://www.pingdom.com)Pingdom web site monitoring
3363pipeLiner/0.3a (PipeLine Spider;http://www.pipeline-search.com/webmaster.html; webmaster'at'pipeline-search.com)pipeline search (DMOZ based) search robot (24.106.39. xxx)
3364pipeLiner/0.xx (PipeLine Spider; http://www.pipeline-search.com/webmaster.html)pipeline search (DMOZ based) search robot (24.106.39. xxx)
3365PitaPita crawler
3366Pizilla++ ver 2.45Private user-agent via Hurricane Electric Internet Services
3367PJspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com)Portaljuice spider
3368Plagger/0.x.xx (http://plagger.org/)Plagger - pluggable RSS/Atom feed aggregator written in Perl
3369PlagiarBot/1.0unknown ucsd.edu robot
3370PlantyNet_WebRobot_V1.9 dhkang@plantynet.comPlantynet web filtering services - Blacklist DB robot
3371plinki/0.1 (you got plinked! (thats a good thing..); http://www.plinki.com; crawl@plinki.com)Unknown UA from 66.220.23.2xx
3372PluckFeedCrawler/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com; 1 subscribers)Pluck RSS feed crawler
3373Pluggd/Nutch-0.9 (automated crawler http://www.pluggd.com;support at pluggd dot com)Pluggd Podcast search engine
3374Pockey-GetHTML/4.12.0 (Win32; GUI; ix86)Yutaka Endo's Pockey / GetHTML / GetHTMLW - some downloading tool from Japan
3375Pockey-GetHTML/x.xxYutaka Endo's Pockey / GetHTML / GetHTMLW - some downloading tool from Japan
3376Pockey/x.x.xYutaka Endo's Pockey / GetHTML / GetHTMLW - some downloading tool from Japan
3377Pockey7.x.x(WIN32GUI)Yutaka Endo's Pockey / GetHTML / GetHTMLW - some downloading tool from Japan
3378POE-Component-Client-HTTP/0.64 (perl; N; POE; en; rv:0.640000)HTTP user-agent for POE (portable networking framework for Perl )
3379PoirotThePlanet/jaja-jak-globusy.com Google Adsense refferer spam bot from 70.85.116.* / 70.84.128.xxx / 70.85.193.xxx
3380polybot 1.0 (http://cis.poly.edu/polybot/)Polybot webcrawler
3381Pompos/1.x http://dir.com/pompos.htmlDir.com / Iliad French recherche robot
3382Pompos/1.x pompos@iliad.frIliad / Free French recherche robot
3383Popdexter/1.0Popdex - web site popularity crawler
3384Port Huron LabsUnknown spam bot / harvester (63.223.10.***)
3385PortalBSpider/2.0 (spider@portalb.com)PortalB (now Alacra search) spider
3386portalmmm/2.0 S500i(c20;TB)portalmmm IMode mobile browser
3387PostFavoritesYahoo (66.94.237.1xx / 216.109.121.xx) favorites tracking robot
3388potbot 1.0Potbot : A simple IRC bot written in Perl
3389PRCrawler/Nutch-0.9 (data mining development project; crawler@projectrialto.com)Project Rialto - data mining development project
3390PrivacyFinder Cache Bot v1.0CUPS robot for AT&T Privacy Bird Privacy Preferences (P3P) enhancements
3391PrivacyFinder/1.1CUPS robot for AT&T Privacy Bird Privacy Preferences (P3P) enhancements
3392Privoxy/3.0 (Anonymous)Privoxy web proxy
3393Production Bot 0116BSome site scanning tool from diff. IPs- i.e.: - 67.99.33.x (lightningcon.broadwing.net)
3394Production Bot 2016BSome site scanning tool from diff. IPs- i.e.: - 216.232.64.xx (telus.net)
3395Production Bot DOT 3016BSome site scanning tool from diff. IPs- i.e.: - 141.154.181.xxx (east.verizon.net)
3396Program Shareware 1.0.2Some spam bot
3397Progressive Downloadunknown
3398Progressive Download HTTP checkunknown
3399Project XP5 [2.03.07-111203]XP5 robot
3400PROve AnswerBot 4.0Answerchase PROve Answerbot
3401ProWebGuide Link Checker (http://www.prowebguide.com)ProWebguide robot
3402psbot/0.1 (+http://www.picsearch.com/bot.html)Picsearch robot (62.119.21.13x)
3403PSurf15a 11Some site scanning tool via diff. IPs- i.e.: QWest Net
3404PSurf15a 51Some site scanning tool via diff. IPs- i.e.: Optonline net (24.191.xxx.xxx)
3405PSurf15a VASome site scanning tool via diff. IPs- i.e.: - choiceone.net (216.153.xxx.xxx) - attbi.com (12.250.xxx.xxx) - optonline.net (24.191.xxx.xxx)
3406psychecloneUnknown website grabbing / ripping for unknown purposes from 208.66.195.x - Digitalinfinity.org Russia
3407PubCrawl (pubcrawl.stanford.edu)Some robot from Stanford University (171.64.75.xxx = PubCrawl.Stanford.EDU)
3408puf/0.91beta6a (Linux 2.2.18; i686)Parallel URL Fetcher downloading tool
3409puf/0.93.2a (Linux 2.4.18; i686)Parallel URL Fetcher downloading tool
3410pulseBot (pulse Web Miner)WebarooBot - Webaroo web site search / theme based downloading tool (64.124.122.2xx)
3411PureSightPureSight Internet content filter
3412PuxaRapido v1.0Puxa Rapido download manager
3413PWeBot/1.2 Inspector (http://www.programacionweb.net/robot.php)ProgramacionWeb.net PWeBot link checking (Argentina)
3414PycURLFast Search robot (using PycURL Python component- s. below)
3415PycURL/7.xx.xPycURL - Python interface to libcurl
3416Python-urllib/1.1xPython URL fetcher - robot used by Naver Japan/Korea
3417Python-urllib/2.0a1Python URL fetcher - robot used by Google
3418Qango.com Web Directory (http://www.qango.com/)Qango.com Web Directory robot
3419QEAVis Agent/Nutch-0.9 (Quantitative Evaluation of Academic Websites Visibility; http://nlp.uned.es/qeavisQEAVis: Quantitative Evaluation of Academic Websites Visibility using Nutch
3420QPCreep Test Rig ( We are not indexing- just testing )Quepasa!com (Latin American search) robot
3421QuepasaCreep ( crawler@quepasacorp.com )Quepasa!com (Latin American search) robot
3422QuepasaCreep v0.9.1xQuepasa!com (Latin American search) robot
3423QueryN MetasearchQueryN Metasearch robot
3424Quicksilver (Blacktree,MacOSX)Blacktrees Quicksilver helper application for Mac
3425QuickTime\xaa.7.0.4 (qtver=7.0.4;cpu=PPC;os=Mac 10.3.9)Quicktime for Macintosh
3426QweeryBot/3.01 ( http://qweerybot.qweery.nl)Qweerybot for the Qweery search engine (in development) - Netherland
3427Qweery_robot.txt_CheckBot/3.01 (http://qweerybot.qweery.com)Qweerybot for the Qweery search engine (in development) - Netherland
3428R6_CommentReader_(www.radian6.com/crawler)Radian6 RSS feed comment crawler
3429R6_FeedFetcher_(www.radian6.com/crawler)Radian6 Rss feed crawler
3430rabaz (rabaz at gigabaz dot com)gigaBaz - the brainbot (Germany) robot
3431RaBot/1.0 Agent-admin/phortse@hanmail.netDAUMOA - Daum search Korea robot (211.115.109.xxx)
3432Rainbot1.1Bot Provider for the All Womans Bot Service?
3433ramBot xtreme x.xIntersearch.de (was www.intersearch.de) robot (Germany)
3434RAMPyBot - www.giveRAMP.com/0.1 (RAMPyBot - www.giveRAMP.com; http://www.giveramp.com/bot.html; support@giveRAMP.com)giveRAMP Search Engine robot (64.69.43.1xx)
3435RAMPyBot/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)giveRAMP Search Engine robot (64.69.43.1xx)
3436Rank Exec (rankexec.com) Reciprocal Link Manager 1.x/botRank Exec reciprocal link checking
3437Rankivabot/3.2 (www.rankiva.com; 3.2; vzmxikn)Rankiva website popularity robot
3438Rational SiteCheck (Windows NT)Innova/IBM Rational SiteCheck - Rational robot
3439RAYSPIDER/Nutch-0.9Unknown spider from Raytheon Company - maybe Raytheon High Speed Guard proxy
3440ReadABlog Spider (compatible; 1.1; feed update; www.readablog.com)Read A Blog - RSS feed and blog search engine
3441RealDownload/4.0.0.4xRealDownload download manager
3442REAP-crawler Nutch/Nutch-1.0-dev (Reap Project; http://reap.cs.cmu.edu/REAP-crawler/; Reap Project)The REAP Web Crawler for the REAP project
3443Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/)Reaper robot for SiteSearch
3444Reaper/2.0x (+http://www.sitesearch.ca/reaper)Reaper robot for SiteSearch
3445REBOL Core 2.x.x.x.xREBOL messaging language for distributed Internet apps
3446REBOL View 1.x.x.x.xREBOL/View - machine independent internet client application
3447RebusnetBot (+http://www.rebusnet.biz)Rebusnet software site - link / submission checking
3448RebusnetPADBot/1.5x (+http://www.rebusnet.biz)Rebusnet software site - link / submission checking
3449reciprocal links checker (http://www.recip-links.com/)Online reciprocal link checker
3450RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com)rediff.com search link checking
3451RedCarpet/1.2 (http://www.redcarpet-inc.com/robots.html)RedCarpet crawler for Pronto price comparison search(66.179.107.1xx)
3452RedCell/0.1 (InfoSec Search Bot (Coming Soon); http://www.telegenetic.net/bot.html; lhall@telegenetic.net)Der Bot for telegenetic.net's security related search (65.220.67.2xx)
3453RedCell/0.1 (RedCell; telegenetic.net/bot.html; lhall_at_telegenetic.net)Der Bot for telegenetic.net's security related search (65.220.67.2xx)
3454RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/)RedKernel Softwares robot
3455REL Link Checker Lite x.xREL Link Checker Lite free version of Web Link Validator
3456RepoMonkey Bait & Tackle/v1.01unknown
3457Rewebber/1.2 libwww-perl/5.41Rewebber proxy service
3458rico/0.1Applied Semantics Auto-Categorizer for QWestDex Direct
3459RixBot (http://babelserver.org/rix)RixBot Rebol Indexer for the RIX - Rebol related search (195.204.121.xx)
3460RMA/1.0 (compatible; RealMedia)StreamBox VCR user agent
3461RMA/1.0 (compatible; RealMedia)Real Media server acting as client
3462RoboCrawl (http://www.canadiancontent.net)Canadian Content search crawler
3463RoboCrawl (www.canadiancontent.net)Canadian Content Search (207.44.220.xx) robot
3464RoboPal (http://www.findpal.com/)FindPal Australia metasearch robot (61.68.139.xx)
3465Robot/www.pj-search.comPopJapanSearch robot
3466Robot: NutchCrawler- Owner: wdavies@acm.orgExperimental robot using Wget via attbi.net
3467Robot@SuperSnooper.ComSupersnooper robot
3468Robozilla/1.0Netscape Directory / DMOZ Open Directory link crawler
3469Rome Client (http://tinyurl.com/64t5n) Ver: 0.9ROME - Open source Java tools for RSS and Atom feeds
3470Rotondo/3.1 libwww/5.3.1Qualigo.de robot
3471RPT-HTTPClient/0.3-xdifferent IPs using the HTTPClient library (mostly link checking)
3472RRC (crawler_admin@bigfoot.com)Metacarta.com (66.28.xx.xxx) robot
3473RssBandit/1.5.0.10 (.NET CLR 1.1.4322.2407; WinNT 5.1.2600.0; http://www.rssbandit.org) (.NET CLR 1.1.4322.2407; WinNT 5.1.2600.0; )RSS Bandit RSS/Atom reader for .NET framework
3474RSSMicro.com RSS/Atom Feed RobotRSS Micro Search - RSS feed search engine
3475RSSOwl/1.2.3 2006-11-26 (Windows; U; zhtw)RSSOwl embedded RSS feed reader
3476RSSOwl/1.2.4 Preview Release 2007-04-15 (Windows; U; zhtw)RSSOwl embedded RSS feed reader
3477RssReader/1.0.xx.x (http://www.rssreader.com) Microsoft Windows NT 5.1.2600.0Ykoon RssReader news feed reader
3478RSurf15a 41Some site scanning tool via diff. IPs- i.e.: - dslx.net (208.35.1x.xxx) - Home.com
3479RSurf15a 51Some site scanning tool via diff. IPs- i.e.: - dslx.net (208.35.1x.xxx) - Home.com
3480RSurf15a 81Some site scanning tool via diff. IPs- i.e.: - dslx.net (208.35.1x.xxx) - Home.com
3481Rubbot/1.0 (+http://rubhub.com/)rubhub blog spider
3482RufusBot (Rufus Web Miner; http://64.124.122.252/feedback.html)WebarooBot - Webaroo web site search / theme based downloading tool (64.124.122.2xx)
3483RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html)WebarooBot - Webaroo web site search / theme based downloading tool (64.124.122.2xx)
3484Rumours-Agentunknown robot from rumours.jp (202.214.69.xxx)
3485RX BarRX (Reflexive Search) Bar for IE
3486S&L Spider (http://search.hirners.com/)Search & Links directory spider
3487S.T.A.L.K.E.R. (http://www.seo-tools.net/en/bot.aspx)SEO-Tools.net link checking ?
3488SafariBookmarkChecker (+http://www.coriolis.ch/)SafariBookmarkChecker for Mac OS X
3489sait/Nutch-0.9 (SAIT Research; http://www.samsung.com)sait robot - unknown robot from Samsung International Korea
3490SandCrawler - Compatibility TestingSandcrawler robot from Microsoft (131.107.0.xx)
3491SapphireWebCrawler/1.0 (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu)Sapphire Web Crawler from Carnegie Mellon University's Language Technologies Institute
3492SapphireWebCrawler/Nutch-1.0-dev (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu)Sapphire Web Crawler from Carnegie Mellon University's Language Technologies Institute
3493savvybot/0.2WebSavvy Directory robot
3494SBIder/0.7 (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html)SiteSell SBIder Nutch based crawler
3495SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html)SiteSell SBIder Nutch based crawler
3496SBL-BOT (http://sbl.net)Softbyte Labs Black Widow web site ripper
3497ScanWebScanWeb - regular expression based web page searching tool
3498ScholarUniverse/0.8 (Nutch;+http://scholaruniverse.com/bot.jsp; fetch-agent@scholaruniverse.com)ScholarUniverse - Scholarly experts search robot
3499schwarzmann.biz-Spider_for_paddel.org+(http://www.innerprise.net/usp-spider.asp)URL Spider Pro (USP) used by German Schwarzmann GmbH
3500Science Traveller International 1X/1.01X Web Browser
3501ScollSpider/2.0 (+http://www.webwobot.com/ScollSpider.php)WebWobot UK search engine robot (82.43.129.2xx)
3502Scooter-3.0.EUAltavista robot
3503Scooter-3.0.FSAltavista robot
3504Scooter-3.0.HDAltavista robot
3505Scooter-3.0.VNSAltavista robot
3506Scooter-3.0QIAltavista robot
3507Scooter-3.2Altavista robot
3508Scooter-3.2.BTAltavista robot
3509Scooter-3.2.DILAltavista robot
3510Scooter-3.2.EXAltavista robot
3511Scooter-3.2.JTAltavista robot
3512Scooter-3.2.NIVAltavista robot
3513Scooter-3.2.SF0Altavista robot
3514Scooter-3.2.snippetAltavista robot
3515Scooter-3.3devAltavista robot
3516Scooter-ARS-1.1Altavista robot
3517Scooter-ARS-1.1-ihAltavista robot
3518scooter-venus-3.0.vnsAltavista robot
3519Scooter-W3-1.0Altavista robot
3520Scooter-W3.1.2Altavista robot
3521Scooter/1.0Altavista robot
3522Scooter/1.0 scooter@pa.dec.comAltavista robot
3523Scooter/1.1 (custom)Altavista robot
3524Scooter/2.0 G.R.A.B. V1.1.0Altavista robot
3525Scooter/2.0 G.R.A.B. X2.0Altavista robot
3526Scooter/3.3Altavista robot
3527Scooter/3.3.QA.pczukorAltavista robot
3528Scooter/3.3.vscooterAltavista robot
3529Scooter/3.3_SFAltavista robot
3530Scooter2_Mercator_x-x.0Altavista using Mercator robot
3531Scooter_bh0-3.0.3Altavista robot
3532Scooter_trk3-3.0.3Altavista robot
3533Scope (Mars+)Scope Navigator mobile browser (Japan)
3534ScoutAboutSome nec.com robot using Research Republic ScoutAbout Research Tool
3535ScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/Ant.com search robot
3536scoutmasterScoutMaster information retrieval software
3537Scrubby/2.x (http://www.scrubtheweb.com/)Scrub the web robot (66.93.156.xx)
3538Scrubby/3.0 (+http://www.scrubtheweb.com/help/technology.html)Scrub the web robot (66.93.156.xx)
3539Search+URL Search+ search software
3540Search-Engine-StudioXtreem Search Engine Studio - SE software
3541search.ch V1.4Search.ch robot
3542search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch)Search.ch robot
3543Search/1.0 (http://www.innerprise.net/es-spider.asp)Enterprise Search web indexing / site searching tool
3544searchbot admin@google.comUnknown robot / website grabber from Chinatelecom (219.142.78.xxx)
3545SearchByUsa/2 (SearchByUsa; http://www.SearchByUsa.com/bot.html; info@SearchByUsa.com)SearchByUSA robot (69.150.7.xxx)
3546SearchdayBotSearchday (Germany) search robot
3547SearchExpress Spider0.99Searchexpress spider
3548SearchGuild/DMOZ/Experiment (searchguild@gmail.com)Searchguild forum & directory robot (81.3.75.xxx)
3549SearchGuild_DMOZ_Experiment (chris@searchguild.com)Searchguild forum & directory robot (81.3.75.xxx)
3550Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk)Searchit robot (69.93.107.xx)
3551Searchmee! Spider v0.98aSearchmee! Search Engine (prototype) robot by findanisp.com
3552SearchSight/2.0 (http://SearchSight.com/)SearchSight search robot
3553SearchSpider.com/1.1SearchSpider robot
3554Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspider.com)SearchSpider robot
3555SearchTone2.0 - IDEAREJanas (Ideare.com / Tiscali.it) robot
3556Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3seekport. beta search (Germany) robot
3557Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF)seekport. beta search (Germany) robot
3558Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2seekport. beta search (Germany) robot
3559Seeker.lookseek.comLookseek search robot / link checking
3560semaforo.netsemaforo.net web filtering software
3561Semager/1.1 (http://www.semager.de/blog/semager-bots/)Semager.de (was NG-Search) semantic search - Germany
3562Semager/1.x (http://www.semager.de)Semager.de (was NG-Search) semantic search - Germany
3563semanticdiscovery/0.xSemantic Discovery domain checking tool
3564Sensis Web Crawler (search_comments\at\sensis\dot\com\dot\au)Sensis Australia search robot
3565Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au)Sensis Australia search robot
3566SeznamBot/1.0Seznam Search (Czech Republic) robot
3567SeznamBot/1.0 (+http://fulltext.seznam.cz/)Seznam Search (Czech Republic) robot
3568SeznamBot/2.0-test (+http://fulltext.sblog.cz/)Seznam Search (Czech Republic) robot
3569ShablastBot 1.0Unknown robot from Shablast.com - Website has no content - Ignores robots.txt
3570Shareaza v1.x.x.xxShareaza P2P peer-to-peer download client
3571SharewarePlazaFileCheckBot/1.0+(+http://www.SharewarePlaza.com)SharewarePlaza File Check Bot - link checking
3572Shim CrawlerChikayama-Taura Lab Shim-Crawler used for The Kototoi Project (Japan) - (133.11.36.xx)
3573Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; crawl@logos.ic.i.u-tokyo.ac.jp)Chikayama-Taura Lab Shim-Crawler used for The Kototoi Project (Japan) - (133.11.36.xx)
3574ShopWiki/1.0 ( +http://www.shopwiki.com/)ShopWiki shopping search based on LittleWiki search
3575ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)ShopWiki shopping search based on LittleWiki search
3576Shoula.com Crawler 2.0Shoula Search Engine crawler
3577SietsCrawler/1.1 (+http://www.siets.biz)Siets Crawler - Web based site crawling application
3578Sigram/Nutch-1.0-dev (Test agent for Nutch development; http://www.sigram.com/bot.html; bot at sigram dot com)Sigram's Nutch robot - crawler testing
3579Siigle Orumcex v.001 Turkey (http://www.siigle.com)Siigle search (Turkey) robot
3580silk/1.0Slider Search directory robot (194.213.194.2xx)
3581silk/1.0 (+http://www.slider.com/silk.htm)/3.7Slider Search directory robot (194.213.194.2xx)
3582SimpleFavPanel/1.2SimpleFavPanel - IE newsfeed panel plugin
3583Simpy 1.x; http://www.simpy.com/Simpy bookmarking and personal search engine
3584Simpy/1.x (Simpy; http://www.simpy.com/?ref=bot; feedback at simpy dot com)Simpy bookmarking and personal search engine
3585Sirketcebot/v.01 (http://www.sirketce.com/bot.html)Sirketçe search - Turkey
3586SiteBar/3.x.x (Bookmark Server; http://sitebar.org/)SiteBar online bookmark manager
3587SiteBar/x.xSiteBar bookmark server
3588SiteBar/x.x.x (Bookmark Server; http://sitebar.org/)SiteBar bookmark server
3589sitecheck.internetseer.comInternetseer Web Site Monitoring / Claymont robot
3590sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer.com)Internetseer Web Site Monitoring
3591SiteRecon+(xx)SiteRecon website monitoring spider at xx minute intervals
3592SiteSnaggerPC Magazin web site downloadmanager
3593SiteSpider +(http://www.SiteSpider.com/)Site Spider robot (66.249.17.xx)
3594SiteSucker/1.x.xSiteSucker Mac website downloading tool
3595SiteTaggerBot (http://www.sitetagger.com/bot.htm)SiteTagger.com bookmark organizer
3596SiteTruth.com site rating systemSiteTruth - Automatic site legitimacy rating system
3597SiteWinderWebwasher.com (217.146.159.xx) internet filter
3598SiteXpertXtreeme SiteXpert sitemap & search engine builder
3599Skampy/0.9.x (http://www.skaffe.com/skampy-info.html)Skaffe.com directory link checker
3600Skimpy/0.x (http://www.skaffe.com/skampy-info.html)Skaffe.com directory link checker
3601Skywalker/0.1 (Skywalker; anonymous; anonymous)Visvo distributed website crawler based on Nutch
3602Slarp/0.1Only.com robot
3603SleipnirSleipnir - Japanese Explorer based browser & search bar
3604Sleipnir Version 1.xxSleipnir - Japanese Explorer based browser & search bar
3605Sleipnir Version2.xSleipnir - Japanese Explorer based browser & search bar
3606Sleipnir/2.xxSleipnir - Japanese Explorer based browser & search bar
3607Slider_Search_v1-deSlider Search directory robot (194.213.194.2xx)
3608SlimBrowserSlim Browser (IE based browser) - uses this user agent for favicon.ico only
3609Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html)Inktomi (Hotbot-Lycos-NBCi) robot
3610Slurp/2.0-KiteWeekly (slurp@inktomi.com; http://www.inktomi.com/slurp.html)Inktomi (Hotbot-Lycos-NBCi) robot
3611Slurp/si (slurp@inktomi.com; http://www.inktomi.com/slurp.html)Inktomi (Hotbot-Lycos-NBCi) robot
3612Slurpy Verifier/1.0Inktomi (Hotbot-Lycos-NBCi) robot - 72.30.61.xx(x)
3613SlySearch (slysearch@slysearch.com)Slysearch robot (now Turnitin robot)
3614SlySearch/1.0 http://www.plagiarism.org/crawler/robotinfo.htmlSlysearch robot (now Turnitin robot)
3615SlySearch/1.x http://www.slysearch.comSlysearch robot (now Turnitin robot)
3616SmartDownload/1.2.67 (Win32; Jan 12 1999)Netzip/Smartdownload download manager
3617SmartDownload/1.2.77 (Win32; Feb 1 2000)Netzip/Smartdownload download manager
3618SmartDownload/1.2.77 (Win32; Jun 19 2001)Netzip/Smartdownload download manager
3619smartwit.comLoop Improvements NRS Enterprise search (69.44.155.xx[x])
3620SmiffyDCMetaSpider/1.0SmiffyDCMetaSpider - Robot to check the retro-adding of Dublin Core metadata
3621sna-0.0.1 (mikemuzio@msn.com)Snoopy PHP-client
3622sna-0.0.1 mikeelliott@hotmail.comSnoopy PHP-client
3623snap.com beta crawler v0Unknown bot from bb2.net (66.234.139.xxx) also as Snapbot/1.0
3624Snapbot/1.0Unknown bot from bb2.net (66.234.139.xxx) - also as snap.com
3625Snapbot/1.0 (Snap Shots, +http://www.snap.com)Unknown bot from Psinet / Cogentco - not from Snap.com
3626Snappy/1.1 ( http://www.urltrends.com/ )My UrlTrends online web ranking service
3627Snarfer/0.x.x (http://www.snarfware.com/)Snarfer RSS reader
3628SnoopRob/x.xUnknown robot from 217.229.156.xx (T-Online Germany)
3629Snoopy v1.xxSnoopy PHP-client
3630Snoopy v1.xx- : User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MyIE2)Snoopy PHP-client
3631Snoopy_v0.xxSnoopy PHP-client
3632SnykeBot/0.6 (http://www.snyke.com)Snyke.com France robot
3633SocSciBot ()Link crawler for the social sciences
3634SoftBank/1.0/812SH/SHJ001 Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1NetFront browser on Softbank mobile phone
3635SoftHypermarketFileCheckBot/1.0+(+http://www.softhypermaket.com)Soft Hypermarket link checking
3636Softizerbot (http://www.softizer.com)Softizer.com software directory link checking
3637sogou develop spiderUnknown UA from Chinanet (220.181.26.1xx) faking Sogou search robot
3638Sogou Orion spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)Unknown UA from Chinanet (220.181.18.xx) faking Sogou search robot
3639sogou spiderUnknown UA from Chinanet (220.181.26.1xx) faking Sogou search robot
3640Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)Unknown UA from Chinanet (220.181.26.1xx) faking Sogou search robot
3641sohu agentUnknown UA from Chinanet (220.181.26.1xx) faking Sogou search robot
3642sohu-searchSohu (Search Fox) search robot China (61.135.131.xxx)
3643Sosospider+(+http://help.soso.com/webspider.htm)SOSO search (China) spider
3644Space Bison/0.02 [fu] (Win67; X; SK)Default Proxomitron (discontinued) filtering proxy user agent identifier
3645SpeedDownload/1.xSpeed Download (Mac) download manager
3646speedfind ramBot xtreme 8.1Speedfind.de robot
3647Speedy Spider (Beta/x.x; speedy@entireweb.com)Entireweb search robot
3648Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/speedyspider/)Entireweb search spider
3649Speedy_Spider (http://www.entireweb.com)Entireweb search robot
3650Sphere Scout&v4.0 - scout at sphere dot comSphere blog and news search robot
3651SphiderSphider - a lightweight search engine in PHP
3652Spida/0.1Only.com robot
3653Spider-Sleek/2.0 (+http://search-info.com/linktous.html)Search-Info ODP/DMOZ spider
3654spider.batsch.comBatsch robot
3655Spider.TerraNautic.net - v:1.04TerraNautic spider for Schnellsuchen touristic search (Germany)
3656spider.yellopet.com - www.yellopet.comYellopet spider
3657Spider/maxbot.com admin@maxbot.comMaxbot .gov .mil .edu indexing robot
3658SpiderKU/0.xUnknown robot from CPE at Kasetsart University (158.108.35.xxx)
3659SpiderManYahoo Search user agent or spider (202.165.102.xxx)
3660SpiderMonkey/7.0x (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml)SpiderMonkey Canada robot
3661Spinne/2.0Spider.de robot
3662Spinne/2.0 medMedkatalog (medical catalogue) Austria robot
3663Spinne/2.0 med_AHMedkatalog (medical catalogue) Austria robot
3664Spock Crawler (http://www.spock.com/crawler)Spock - people search application - via Amazon web services
3665sportsuchmaschine.de-Robot (Version: 1.02- powered by www.sportsuchmaschine.de)Sportsuchmaschine (German sports related search) link checking / robot
3666sproose/0.1-alpha (sproose crawler; http://www.sproose.com/bot.html; crawler@sproose.com)Sproose personalized search (38.100.225.xx)
3667SQ WebscannerSQ Webscanner Mac download manager
3668Squid-PrefetchSimple page-prefetch for Squid web proxy
3669squidclamSquidclam is a replacement for SquidClamAV-Redirector
3670SquidClamAV_Redirector 1.x.xSCAVR - Squid helper script for scanning download URLs for viruses
3671Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu)AOL Search / Pacific Internet Exchange robot
3672Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)diff. IPs / services i.e.: - Inria.fr robot - Websense (Internet filtering) robot
3673Sqworm/2.9.89-BETA (beta_release; 20020130-839; i686-pc-linux-gnu)Time Warner Telecom user robot ?
3674SSurf15a 11Some site scanning tool via diff. IPs i.e.: - choiceone.net (216.153.xxx.xxx) - epix.net (216.108.198.xx)
3675StackRambler/x.x Rambler search (Russia) robot (81.19.6x.xx)
3676Stamina/1.4Stamina download manager
3677Star DownloaderStar Downloader download manager
3678StarDownloader/1.xxStar Downloader download manager
3679stat statcrawler@gmail.comExperimental search engine spider from 66.92.186.xxx
3680Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)Steeler crawler
3681Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)Steeler - University of Tokyo web crawler
3682Strategic Board Bot (+http://www.strategicboard.com)Strategic Board blog & news search robot
3683Strategic Board Bot (+http://www.strategicboard.com)Strategic Board blogs and news aggregator robot
3684Submission Spider at surfsafely.comSurfsafely submission verifier
3685suchbaer.deSuchbaer.de (Germany) search robot
3686suchbaer.de (CrawlerAgent v0.103)Suchbaer.de (Germany) search robot
3687suchbotSuchbot Germany robot
3688Suchknecht.at-RobotSuchknecht Austria robot
3689suchpadbot/1.0 (+http://www.suchpad.de)suchpad search Germany robot (213.239.194.xx)
3690Sunrise XP/2.xSunrise XP handheld news / website reader and converter
3691Sunrise/0.42g (Windows XP)Sunrise XP web sites and newsfeeds converter and handheld reader
3692SuperBot/x.x (Win32)SuperBot website copier
3693SuperBot/x.x.x.xx (Windows XP)SuperBot website copier
3694Superdownloads SpidermanUbbi Superdownloads (Brazil) link checking
3695SURF SurfControl Web Filtering
3696SurferF3 1/0Wanadoo Rechereche robot
3697SurfMasterMaskbit Surfmaster bookmark tool
3698SurveyBot/2.2 Whois SourceWhois Source domain name information robot (66.249.26.xx)
3699SurveyBot/2.3 (Whois Source)Whois Source domain name information robot (66.249.26.xx)
3700suzuranYokogao Search Engine robot (Kanazawa University)
3701SWB/V1.4 (HP)HP Secure Web Browser for OpenVMS
3702swbot/0.9c libwww/5.3.1unknown
3703Swooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.htm)Swooglebot Swoogle's semantic web crawler
3704SWSBot-Images/1.2 http://www.smartwaresoft.com/swsbot12.htmlSWSBot - SmartWareSoft (85.186.255.xx) software search engine created for Playfuls.com
3705SygolBot http://www.sygol.netSygol Search (Italy) robot
3706Sylera/1.2.xSylera browser (Japan)
3707SyncBotMindspring.com user robot
3708SyncIT/x.xSyncIT link validation
3709Syndirella/0.91preSyndirella desktop information aggregator (beta)
3710SynoBotSynomia (France) robot
3711Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawlerSyntryx Solution Suite - domain / keyword crawler (216.7.179.xx)
3712Szukacz/1.xSzukacz.pl (Polish search) robot
3713Szukacz/1.x (robot; www.szukacz.pl/jakdzialarobot.html; szukacz@proszynski.pl)Szukacz.pl (Polish search) robot
3714T-Online BrowserGerman T-Online browser & internet suite
3715tags2dir.com/0.8 (+http://tags2dir.com/directory/)tags2dir.com directory index
3716Tagword (http://tagword.com/dmoz_survey.php)TAGword DMOZ survey - ODP link checking robot
3717Tagyu Agent/1.0Tagyu - del.icio.us bookmark collection online tag generator
3718Talkro Web-Shot/1.0 (E-mail: webshot@daumsoft.com- Home: http://222.122.15.190/webshot)Daumsoft Talkro IR robot
3719TAMU_CS_IRL_CRAWLER/1.0Texas A&M University - Dept. of Computer Science crawler (server or link checking ?)
3720targetblaster.com/0.9kTargetblaster user link validation ?
3721TargetYourNews.com botTarget Your News - user submitted links
3722TCDBOT/Nutch-0.8 (PhD student research;http://www.tcd.ie; mcgettrs at t c d dot IE)Trinity College Dublin (Ireland) TCDBOT
3723TEHTTP header for transfer encoding used as user agent name ?
3724TeamSoft WinInet ComponentWinInet Internet client app.
3725TECOMAC-Crawler/0.xTecomac Gmbh (Germany) crawler software - now Arexera Information Technologies
3726Tecomi Bot (http://www.tecomi.com/bot.htm)Tecomi (Germany) beta / test robot (84.201.65.xxx)
3727Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://www.netseer.com/crawler.html; crawler@netseer.com)Teemer crawler for NetSeer search (beta) via Amazon Web Services - see also NetSeer/Nutch
3728Teleport Pro/1.2x(.1xxx)Teleport (website) downloading tool
3729Teoma MPTeoma crawler (65.214.36.xx[x])
3730teomaagent crawler-admin@teoma.comTeoma crawler (65.214.36.xx[x])
3731teomaagent1 [crawler-admin@teoma.com]Teoma crawler (65.214.36.xx[x])
3732teoma_agent1Teoma crawler (65.214.36.xx[x])
3733Teradex Mapper; mapper@teradex.com; http://www.teradex.comTeradex Directory robot
3734TeragramCrawlerTeragram multilingual text & data processing software
3735terraminds-bot/1.0 (support@terraminds.de)Terraminds blog search (Germany)
3736TerrawizBot/1.0 (+http://www.terrawiz.com/bot.html)Terrawiz Indian Search Engine robot
3737Test spiderNoceans Information Portfolio Manager (66.35.69.x)
3738TestCrawler/Nutch-0.9 (Testing Crawler for Research ; http://balihoo.com/index.aspx; tgautier at balihoo dot com)Balihoo - Search Engine for Advertising Media
3739The Expert HTML Source Viewer (http://www.expert-html.com)Expert HTML online source viewer
3740TheRarestParser/0.2a (http://therarestwords.com/)The Rarest Words - Linguistic experiment crawler via Amazon Web Services
3741TheSuBot/0.1 (www.thesubot.de)TheSuBot robot (Germany) for an unknown theme based search engine
3742thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de)ThumbShots website thumbnail service (Germany) robot
3743thumbshots-de-Bot (Version: 1.02- powered by www.thumbshots.de)ThumbShots.de (Germany) robot
3744timboBot/0.9 http://www.breakingblogs.com/timbo_bot.htmlBreaking Blogs timbo bot blog robot
3745http://www.timelyweb.com/TimelyWeb web page monitoring tool
3746TinEye/1.1 (http://tineye.com/crawler.html)TinEye crawler for an open image search project
3747tivraSpider/1.0 (crawler@tivra.com)Tivra spider from AT&T Labs Research
3748TJG/SpiderTjgroup spider
3749TJvMultiHttpGrabber ComponentTJvHttpGrabber (JEDI Visual Component Library)
3750Tkensaku/x.x(http://www.tkensaku.com/q.html)Tkensaku Search (Japan) robot from 210.239.46.xxx (www.tken.com)
3751toCrawl/UrlDispatcherUnknown robot from 195.68.98.xx (coltfrance.com)
3752Topodia/1.2-dev (Topodia - Crawler for HTTP content indexing; http://www.topodia.com/; support@topodia.com)Topodia search engine and personal information assistant (in development)
3753TOPOS robot/1.1 (http://www.topos.com.ua/)Topos search (Russia) robot
3754traazibot/testengine (+http://www.traazi.de)Traazi! search (Germany) robot
3755Trailfire-bot/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)Trailfire web collection and annotating system
3756Trailfire-bot/0.7.1 (Trailfire page content analyzer; http://trailfire.com; info@trailfire.com)Trailfire web collection and annotating system
3757Trailfire/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)Trailfire web collection and annotating system
3758Trampelpfad-SpiderTrampelpfad Webkatalog spider
3759Trampelpfad-Spider-v0.1Trampelpfad Webkatalog spider
3760tricosMetaCheck 1.2216-08-1999 (http://www.tricos.com/metacheck)Tricos meta tag validation
3761TSurf15a 11some bad user agent
3762TulipChain/5.x (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://java.sun.com/) Linux/2.4.17Tulip Chain browser / link checker for Dmoz.org directory
3763TulipChain/5.xx (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://apple.com/) Mac_OS_X/10.2.8Tulip Chain browser / link checker for Dmoz.org directory
3764Tumblr/1.0 RSS syndication (+http://www.tumblr.com/) (support@tumblr.com)Tumblr Tumblelogs RSS and news syndication crawler
3765TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html)Turnitin (ex SlySearch) robot for helping educational institutions prevent plagiarism
3766Turnpike Emporium LinkChecker/0.1TurnPike Emporium Directory (207.67.198.x) link checking
3767TutorGig/1.5 (+http://www.tutorgig.com/crawler)TutorGig tutorial search robot
3768Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler)TutorGig tutorial search robot
3769Twiceler www.cuill.com/robots.htmlTwiceler experimental web crawler
3770Twiceler-0.9 http://www.cuill.com/twiceler/robot.htmlTwiceler experimental web crawler
3771Twisted PageGetterFile downloading component from Twisted Python
3772Twitturly / v0.xTwitt(url)y URL tracking service for Twitter via Amazon Web Services
3773Twotrees Reactive Filter V2.0Twotrees content filter
3774Tycoon Agent/Nutch-1.0-devTycoon - Hewlett-Packards distributed cluster solution robot
3775TygoBotTygo Search robot
3776TygoProwlerTygo Search robot
3777UCmoreUCMore - IE navigation and search plugin
3778UCMore Crawler AppUCMore - IE navigation and search plugin
3779UCWEB5.1Ucweb mobile browser
3780UDMuser agent - maybe UdmSearch (see UdmSearch) ?
3781UdmSearch/3.1.xUdmSearch / MySearch (now mnoGoSeach) offline browser/search client
3782UIowaCrawler/1.0University of Iowa Crawler- possibly MySpiders
3783UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; http://www.ukwizz.com/)UKWizz search robot
3784UltraseekInfoseek robot
3785Under the Rainbow 2.2Unknown mail harvester/spambot from 80.58.13.xxx (proxycache.rima-tde.net)
3786unknownght.com Web Server IIS vs Apache Survey. See Results at www.DNSRight.comDNS Right - Online DNS tools
3787UofTDB_experiment (leehyun@cs.toronto.edu)Unknown robot from University of Toronto (128.100.5.1xx)
3788UP.Browser/3.01-IG01 UP.Link/3.2.3.4Mobile phone browser
3789updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)Updated! search robot
3790updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om)Updated! search robot
3791UPG1 UP/4.0 (compatible; Blazer 1.0)Handspring (PalmOS powered cellphone) Treo Blazer browser
3792UptimebotUptimeBot.com online link popularity check
3793UptimeBot(www.uptimebot.com)UptimeBot.com online link popularity check
3794URI::Fetch/0.06URI::Fetch - client for fetching HTTP pages and syndication feeds (RSS Atom)
3795URL Spider Pro/x.xx (innerprise.net)Innerprise URL Spider Pro (now ES.NET) web indexing / site searching tool
3796URLBase/6.xURLBase - Internet shortcut manager
3797URLBlazeURLBlaze file sharing link toolkit
3798urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html://URLFAN news crawler
3799URLGetFileURLGetFile downloading tool
3800URL_Spider_Pro/x.xInnerprise URL Spider Pro (now ES.NET) web indexing / site searching tool
3801URL_Spider_Pro/x.x+(http://www.innerprise.net/usp-spider.asp)Innerprise URL Spider Pro (now ES.NET) web indexing / site searching tool
3802User-Agent: BoardReader Favicon Fetcher /1.0 info@boardreader.comBoardReader search favicon fetcher
3803User-Agent: BoardReader Image Fetcher /1.0 info@boardreader.comBoardReader search image fetcher
3804User-Agent: LjSEEK Picture-Bot /1.0 contact@ljseek.comljpic.com - LiveJournal picture feed search
3805User-Agent: FileHeap! file downloader (http://www.fileheap.com)FileHeap download manager
3806User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)Malformed UA header from some guestbook/forum spammer
3807User-Agent: Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com)Skizzle search robot
3808user-agent=Mozilla/3.01Goldunknown robot (reads robots.txt) or sitegrabber. From different IPs- ie.: 62.98.8.xx (wind.it)
3809USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html)University of Sydney NLP Spider for research in Natural Language Processing
3810UtilMind HTTPGetWeb Thief Site Grabber
3811Utopia WebWasher 3.0WebWasher ad filter
3812uTorrent/1500uTorrent BitTorrent client
3813VadixBotUnknown bad behaving bot via Road Runner - see link
3814Vagabondo-WAP/2.0 (webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)/1.0 ProfileWiseGuys WAP pages robot
3815Vagabondo/1.x MT (webagent@wise-guys.nl)WiseGuys robot Netherland - 82.94.216.2
3816Vagabondo/2.0 MTWiseGuys robot Netherland - 82.94.216.2
3817Vagabondo/2.0 MT (webagent at wise-guys dot nl)WiseGuys robot Netherland - 82.94.216.2
3818Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl)WiseGuys robot Netherland - 82.94.216.2
3819Vagabondo/3.0 (webagent at wise-guys dot nl)WiseGuys robot Netherland - 82.94.216.2
3820Vakes/0.01 (Vakes; http://www.vakes.com/; search@vakes.com)Open Directory link checking from Vakes
3821VayalaCreep-v0.0.1 (haploid@haploid.com)unknown level3.net (63.214.172.xxx) robot
3822Vayala|Creep-v0.0.1 (codepoet@wildties.com)unknown level3.net (63.214.172.xxx) robot
3823vb wininetiNet Grabber - Internet content grabber
3824versus 0.2 (+http://versus.integis.ch)Versus Project robot - Comparing methods for near-uniform URL sampling
3825versus crawler eda.baykan@epfl.chUnknown robot from EPFL University Switzerland (128.178.155.xxx)
3826VerticrawlbotVerticrawl - Semantic search engine solution (French)
3827VeryGoodSearch.com.DaddyLongLegsVeryGoodSearch.com link submission checking
3828verzamelgids.nl - Networking4all Bot/x.xVerzamelgids NL link checking robot
3829Verzamelgids/2.2 (http://www.verzamelgids.nl)Verzamelgids NL link checking robot
3830Vespa CrawlerUnknown robot from Yahoo Norway
3831virus_detector (virus_harvester@securecomputing.com)Sidewinder G2 anti-virus and anti-spyware protection
3832VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com)Visbot crawler for a search software under development
3833Visicom ToolbarSome IE toolbar made with Visicom Media Dynamic Toolbar software
3834Vision Research Lab image spider at vision.ece.ucsb.eduVision research lab's Cortina - content based image retrieval (128.111.60.xx)
3835VLC media player - version 0.8.5 Janus - (c) 1996-2006 the VideoLAN teamVLC - Cross-platform media player and streaming server
3836VMBot/0.x.x (VMBot; http://www.VerticalMatch.com/; vmbot@tradedot.com)VM - Vertical Search Engine (China)
3837Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/)Vortex Web Indexing Robot for a study on internet link distribution
3838voyager-hc/1.0Kosmix health, auto and travel search crawler (204.14.48.x / 38.113.234.xxx)
3839voyager/1.0Kosmix health, auto and travel search crawler (204.14.48.x / 38.113.234.xxx)
3840voyager/2.0 (http://www.kosmix.com/html/crawler.html)Kosmix health, auto and travel search crawler (204.14.48.x / 38.113.234.xxx)
3841VSE/1.0 (testcrawler@hotmail.com)Vivisimo search crawler (206.210.89.xxx)
3842VSE/1.0 (testcrawler@vivisimo.com)Vivisimo search crawler (206.210.89.xxx)
3843vspiderVerity vspider indexing software
3844vspider/3.xVerity vspider indexing software
3845VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;+vwbot@cs.uiuc.eduVWBot - MetaQuerier Crawler for the MetaQuerier project at the University of Illinois
3846W3C-checklink/3.x.x.x libwww-perl/5.xxW3C Link Checker
3847W3C-checklink/4.x [4.xx] libwww-perl/5.xxxW3C Link Checker
3848W3C-WebCon/5.x.x libwww/5.x.xWebCon - the Libwww command line tool
3849W3CLineMode/5.4.0 libwww/5.x.xW3C Line Mode (character based Web browser)
3850W3CRobot/5.4.0 libwww/5.4.0Unknown link checking using Libwww via Korea Telecom (221.148.44.xxx)
3851W3C_Validator/1.xxx libwww-perl/5.xxW3C HTML-Code Validator
3852w3m/0.x.xxw3m Linux pager / text-based browser
3853W3SiteSearch Crawler_v1.1 http://www.w3sitesearch.deW3 Site Search (Germany) search engine solution
3854wadaino.jp-crawler 0.2 (http://wadaino.jp/)Wadain (Japan) Blog / RSS search crawler
3855WannaBe (Macintosh; PPC)Wanna-Be text mode browser
3856WapOnWindows 1.0WapOnWindows WAP browser for PCs
3857Watchfire WebXM 1.0Watchfire WebXM intranet solution
3858WAVcheck 1.0.x (http://www.webbanalys.se/apps/WAVcheck/)WAVcheck - Simple Vendor Discovery Tool for detecting client-side tags from web analytics vendors
3859Wavefire/0.8-dev (Wavefire; http://www.wavefire.com; info@wavefire.com)Wavefire local search community engine (64.141.15.1xx)
3860Waypath development crawler - info at waypath dot comWaypath blog discovery engine robot
3861Waypath Scout v2.x - info at waypath dot comWaypath blog discovery engine robot
3862WDG_Validator/1.1WDG HTML-code validator
3863Web Image CollectorDatafire.com's Web Image Collector (graphics downloading tool)
3864Web Link Validator 1.5Relsoft link checking software
3865Web SnooperRankMeter ranking software
3866web-bekannt (Version: 1.02, powered by www.internetservice-franken.de)Web-bekannt German web directory link checking
3867web-bekannt (Version: 1.02, powered by www.web-bekannt.de)Web-bekannt German web directory link checking
3868Web-Bot V1.03Unkown link or server checking from Würzburg University Germany (132.187.10.xx)
3869Web-Robot/5.0 (en-US; web-robot.com/policy.html) Web-Robot Crawler/2.0.3Unknown robot from 69.50.233.x (nectartech.com)
3870web2express.org/Nutch-0.9-dev (leveled playing field; http://web2express.org/; info at web2express.org)Web2Express / Web2x - Open data searching tool
3871WebAlta Crawler/1.2.1 (http://www.webalta.ru/bot.html)WebAlta search Russia crawler (85.21.201.xx)
3872WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html)WebarooBot / RufusBot from webaroo offline search service
3873WebarooBot (Webaroo Bot; http://www.webaroo.com/rooSiteOwners.html)WebarooBot / RufusBot from webaroo offline search service
3874WebAuto/3.4xxx (WinNT; I)Yanasoft WebAuto website copier / downloading tool
3875webbandit/4.xx.0Web Bandit personal search software
3876WebBug/5.xAmansoft WebBug web server protocol test
3877Webclipping.comWebClipping.com - online news monitoring service
3878webcollage/1.xxWebCollage Syndicator graphics crawler/collector
3879WebCompass 2.0Quarterdecks WebCompass search tool
3880WebCopier vx.xWebCopier offline browser
3881WebCopier vx.xaWebCopier offline browser
3882WebCorp/1.0WebCorp linguistic search engine (UK)
3883webcrawl.netWebcrawl Search robot (64.40.105.xxx)
3884WebDownloader for X x.xxUnix/Linux Web Downloader
3885Webdup/0.9Unknown robot from china-netcom.com
3886WebFetchWingFlyer WebFetch website downloading tool
3887webfetch/5.x.xwebfetch - command line tool to fetch files via HTTP
3888WebFilter Robot 1.0Verso NetSpective WebFilter
3889WebFilter Robot 1.xTelemate.net NetSpective WebFilter
3890WebFindBot(http://www.web-find.com)Webfind search robot
3891Webglimpse 2.xx.x (http://webglimpse.net)Webglimpse search engine software
3892webGobbler/1.x.xwebGobbler - Online random image generator
3893webhackfake ?
3894WebImages 0.3 ( http://herbert.groot.jebbink.nl/?app=WebImages )herbert.groot.jebbink.nl Web Images collage generator
3895WebLight/4.x.x (support@illumit.com; http://www.illumit.com/Products/weblight/)WebLight web analyzer & link checker
3896Weblink's checker/WebLink's link management system for HTTP- FTP and Mail hyperlinks
3897Weblog Attitude Diffusion 1.0Los Alamos National Laboratoy weblog research project
3898webmeasurement-bot, http://rvs.informatik.uni-leipzig.deUnknown robot from Leipzig University (Germany) faculty for computer science
3899WebMiner/x.x [en] (Win98; I)WebMiner bulk file downloader
3900WeBoX/0.xxWeBoX (Japan) - Browser and web collector
3901WebPix 1.0 (www.netwu.com)WebPix - picture downloading tool
3902WebQLCaesius WebQL - Custom robot/agent generator / web extraction software
3903WebRACE/1.1 (University of Cyprus- Distributed Crawler)WebRACE - HTTP retrieval- annotation and caching engine
3904WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/)WebRankSpider experimental web crawler
3905WebReaper vx.x - www.webreaper.netWebreaper download manager
3906WebReaper [info@webreaper.net]Webreaper download manager
3907WebReaper [webreaper@webreaper.net]Webreaper download manager
3908WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; Search@WebSearch.COM.AU)Websearch Australia robot
3909WebSearchBench WebCrawler v0.1(Experimental)Dortmund University WebSearchBench - Open source search software
3910WebSearchBench WebCrawler V1.0 (Beta)- Prof. Dr.-Ing. Christoph Lindemann- Universität Dortmund- cl@cs.uni-dortmund.de- http://websearchbench.cs.uni-dortmund.de/Dortmund University WebSearchBench - Open source search software
3911Website Explorer/0.9.x.xWeb site downloading tool and offline browser (Japan)
3912Website eXtractorWebsite eXtractor web site downloading tool
3913WebsiteWorth v1.0Sootle web directory Website Worth ranking tool
3914Webspinne/1.0 webmaster@webspinne.deWebspinne.de robot
3915Websquash.com (Add url robot)Websquash.com Search Engine robot / link checking
3916WebStat/1.0 (Unix; beta; 20040314)WebStat - Java statistical computing environment for the web
3917Webster v0.3 ( http://webster.healeys.net/ )Webster - Rev. Healeys web crawler
3918webster-internet.de pad browserWebsters Webmaster Archive (Germany) submission / pad checking
3919WebStripper/2.xxWebStripper download manager
3920WebTrafficExpress/x.0WebTrafficExpress IBM server software
3921WebTrends/3.0 (WinNT)Web Trends link analyzer
3922WebVac (webmaster@pita.stanford.edu)The Stanford WebBase Project crawler
3923WebVal/1.0webval - Python link checking tool
3924Webverzeichnis.de - Telefon: 01908 / 26005Webverzeichnis.de (Germany) directory robot
3925WebVulnCrawl.unknown/1.0 libwww-perl/5.803Web Vulnerability Crawler
3926WebWatcherMonitor/2.01Studio Net.Idea's Web Watcher Monitor robot
3927WebZIP/x.x (http://www.spidersoft.com)WebZip offline browser
3928Wells Search IIUnknown spam bot / harvester (62.163.**.** / 62.194.**.*)
3929WEP Search 00Some spam bot- see link
3930West Wind Internet Protocols 4.xxwwIPStuff - Internet client tools for Visual FoxPro
3931WFARCIBM's Almaden Research robot (Clever search project)
3932Wget/1.x(.x)GNU wget http://www.gnu.org/software/wget/wget.html - file downloaderGNU wget - file downloader
3933Wget/1.x+cvs-stable (Red Hat modified)GNU wget - file downloader
3934Wget/1.x.x+cvsGNU wget - file downloader
3935Whatsup/x.xWhatsup Gold network monitor
3936whatUseek_winona/3.0WhatUSeek / Chubba robot
3937WhizBang! LabWhizBang! Labs (closed since May 2002) information extraction robot
3938Wildsoft Surfersome download agent
3939Willow Internet Crawler by Twotrees V2.1Twotrees crawler
3940WinampMPEG/2.00 (larbin@unspecified.mail)unknown robot from gw.ocg-corp.com (209.126.176.x)
3941WincerSong Agent v1.0Super Affiliate Tracker agent by Wincer Song
3942Windows-Media-Player/10.00.00.xxxxWindows Media Player 10
3943WinGet 1.1Nicksoft WinGet download manager
3944WinHTTP Example/1.0Example code for a WinHTTP C++ library crawler
3945WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkBot; winkbot@wink.com)Wink beta search robot (64.13.136.x)
3946WinPodder (http://winpodder.com)WinPodder - Podcast player and RSS reader
3947WinWAP/3.x (3.x.x.xx; Win32) (Google WAP Proxy/1.0)WinWap - Windows PC WAP browser
3948Wir sind die Borg (Version: 1.03, Sie wurden Assimiliert +http://www.yammba.com/suchmaschine/bot.html)Yammba web directory (Germany) link checking
3949WIRE/0.11 (Linux; i686; Bot,Robot,Spider,Crawler,aromano@cli.di.unipi.it)WIRE crawler used by the University of Pisa - Italy
3950WIRE/0.x (Linux; i686; Bot,Robot,Spider,Crawler)WIRE - Web information retrieval environment crawler
3951WISEbot/1.0 (WISEbot@koreawisenut.com; http://wisebot.koreawisenut.com)Korea Wisenut robot
3952WiseWire-Spider2Wisewire domain checker (Discontinued)
3953wish-project (http://wish.slis.tsukuba.ac.jp/)WISH academic research project for link checking
3954WordChampBotWordchamp web page vocabulary / translation robot
3955WordPress/x.x.x.x PHP/4.x.xxWordPress personal Blog publishing platform
3956worio heritrix bot (+http://worio.com/)WORIO (beta) search for computer scientists and programmers using Heritrix open-source crawler
3957woriobot ( http://www.worio.com/)WORIO (beta) search for computer scientists and programmers via Amazon Web Services
3958WorldLightEntireweb Search robot (62.13.25.xxx)
3959WorQmada/1.0unknown link checking (from 4.18.57.126) ?
3960Wotbox/alpha0.6 (bot@wotbox.com; http://www.wotbox.com)Wotbox spider
3961Wotbox/alpha0.x.x (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02Wotbox spider
3962WSB WebCrawler V1.0 (Beta)- cl@cs.uni-dortmund.deWebSearchBench crawler from Dortmund University- Germany
3963WSB, http://websearchbench.cs.uni-dortmund.deWebSearchBench crawler from Dortmund University- Germany
3964wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/)WUME Lab's web crawler (128.180.121.xxx)
3965Wusage/x.0@boutell.comWusage log-file analysis
3966Wwlib/LinuxWWLib - Wolverhampton Univerity Web Library for classifying web documents
3967WWSBOT 1.x [--- http://www.analyzer.nu ---]WWSBOT web server version checker
3968WWW-Mechanize/1.1xPerl web page fetching module
3969www.arianna.itArianna robot
3970www.business-socket.com registry verify/1.xBusiness-Socket.com link checking ?
3971www.doweb.co.uk crawlerThe DoWeb UK Business directory link checking
3972www4mail/2.x libwww-FM/2.14 (Unix; I)www4mail - web navigation & database search by e-mail
3973WWWC/1.0xWWWC Updating check of Web pages. (Japanese only)
3974WWWeasel Robot v1.00 (http://wwweasel.de)World Wide Weasel Germany robot
3975WWWOFFLE/2.xWWWoffle download manager
3976wwwster/1.x (Beta- mailto:gue@cis.uni-muenchen.de)Unknown robot from CIS at Munich University
3977wxDownload FastwxDownload Fast (wxDFast) open source download manager
3978X-Crawler Arexera (Germany) crawler software
3979Xaldon WebSpiderXaldon WebSpider offline browser
3980Xenu Link Sleuth 1.xxXenu link checker
3981Xenu's Link Sleuth 1.x[a-z]Xenu link checker
3982Xerka WebBot v1.0.0 [UPVOpenDir]XerKa text mining and information retrieval software
3983xine/1.0xine - free Linux / OS/2 multimedia player
3984xirq/0.1-beta (xirq; http://www.xirq.com; xirq@xirq.com)XIRQ search (beta) robot (70.86.206.1xx)
3985XMLSlurp/0.1 libwww-perl/5.805GPath / XMLSlurp - Expression language for tree structured data
3986XRL/2.00b1 (Linux; i686; en-us) (+http://metamark.net/about)Metamark URL Shorten Service
3987XylixXylix Retrieval System software
3988xyro_(xcrawler@cosmos.inria.fr)Inria Crawler
3989Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)Yahoo Search Japan robot (211.14.8.2xx)
3990Y!J-SRD/1.0Yahoo Search Japan robot (203.216.197.xxx)
3991Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)Yahoo Search Japan robot (211.14.8.2xx)
3992Y!OASIS/TEST no-ad Mozilla/4.08 [en] (X11; I; FreeBSD 2.2.8-STABLE i386)Yahoo picture service for mobiles
3993Y!TunnelProY!TunnelPro - Yahoo! Messenger companion user agent
3994yacy (www.yacy.net; v20040602; i386 Linux 2.4.26-gentoo-r13; java 1.4.2_06; MET/en)Yacy distributed P2P web search engine robot
3995yacybot (x86 Windows XP 5.1; java 1.5.0_06; Europe/de) yacy.netYacy distributed P2P web search engine robot
3996Yahoo Pipes 1.0(Yahoo) Pipes interactive data aggregator robot
3997Yahoo! MindsetYahoo Mindset: Intent-driven Search (66.228.182.1xx)
3998Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html )Yahoo blog indexing robot (209.191.83.1xx)
3999Yahoo-MMAudVid/1.0 (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com)Yahoo multimedia crawler (206.190.43.xx)
4000Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;Mozilla 4.0 compatible; MSIE 7.0;Windows NT 5.0; .NET CLR 2.0)Yahoo multimedia crawler
4001Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com)Yahoo multimedia crawler via Fastsearch.net (66.77.73.xx)
4002Yahoo-Test/4.0Yahoo Search robot
4003Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawlerYahoo crawler via Overture (66.77.73.3x)
4004YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide)Yahoo Publisher Network RSS crawler
4005YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)Yahoo Product Search crawler ( 68.142.195..x)
4006YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/)Yahoo Product Search crawler ( 66.196.93.x)
4007YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/yahooseeker.html)Yahoo Product Search crawler ( 66.196.93.x)
4008YahooSeeker/1.1 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/)Yahoo Product Search crawler ( 66.196.93.x)
4009YahooSeeker/bsv3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html )Yahoo Product Search crawler ( 68.142.195..x)
4010YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;cafekelsa-dev-webmaster@yahoo-inc.com )Yahoo robot
4011Yandex/1.01.001 (compatible; Win16; I)Yandex Search Russia link checking (213.180.206.2xx)
4012Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)Yanga search robot by Gigabase (Russian Federation)
4013yarienavoir.net/0.2Yarienavoir search (Belgium) robot
4014Yeti1noon.com search Korea robot (222.231.21.xxx)
4015Yeti/0.01 (nhn/1noon, yetibot@naver.com, check robots.txt daily and follows it)1noon.com search Korea robot (222.231.21.xxx)
4016Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)Naver search (Korea) robot
4017yggdrasil/Nutch-0.9 (yggdrasil biorelated search engine; www dot biotec dot tu minus dresden do de slash schroeder; heiko dot dietze at biotec dot tu minus dresden dot de)yggdrasil spider for GoPubMed biorelated search engine
4018YodaoBot/1.0 (http://www.yodao.com/help/webmaster/spider/; )Yodao search (China)
4019yoofind/yoofind-0.1-dev (yoono webcrawler; http://www.yoono.com ; MyEmail)Yoono - community based search (193.110.140.xxx / 194.0.179.[x]xx)
4020yoogliFetchAgent/0.1Yoogli search (under development) agent
4021yoono/1.0 web-crawler/1.0Yoono - community based search (193.110.140.xxx / 194.0.179.[x]xx)
4022YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search EngineYottaCars bot - YottaCar car search engine ( 64.62.175.xxx)
4023YottaShopping_Bot/4.12 (+http://www.yottashopping.com) Shopping Search EngineYottaShopping bot - YottaShopping search engine ( 64.62.175.xxx)
4024YTunnelProY!TunnelPro - Yahoo! Messenger companion user agent
4025Z-Add Link Checker (http://w3.z-add.co.uk/linkcheck/)Z-Add online link checker
4026Zao-CrawlerZao crawler for Kototoi Project
4027Zao-Crawler 0.2bZao crawler for Kototoi Project
4028Zao/0.1 (http://www.kototoi.org/zao/)Zao crawler for Kototoi Project
4029ZBot/1.00 (icaulfield@zeus.com)Zeus Internet Marketing Robot based on Webster Pro component
4030ZearchitZearchit German search / directory
4031ZeBot_lseek.net (bot@ze.bz)Ze.bz Moteur de Recherche robot
4032ZeBot_www.ze.bz (ze.bz@hotmail.com)Ze.bz Moteur de Recherche robot
4033zedzo.digest/0.1 (http://www.zedzo.com/)ZedZo Search (beta) robot
4034Zend_Http_ClientZend PHP frameworks Zend_Http_Client component
4035zermelo Mozilla/5.0 compatible; heritrix/1.12.1 (+http://www.powerset.com) [email:crawl@powerset.com,email:paul@page-store.com]Powerset Natural Language Search crawler (under development) using Heritrix via Amazon Web Services
4036zerxbot/Version 0.6 libwww-perl/5.79Zerx search robot ?
4037Zeus ThemeSite Viewer Webster Pro V2.9 Win32Zeus Internet Marketing Robot (based on Webster Pro)
4038Zeus xxxxx Webster Pro V2.9 Win32Zeus Internet Marketing Robot (based on Webster Pro)
4039Zeusbot/0.07 (Ulysseek's web-crawling robot; http://www.zeusbot.com; agent@zeusbot.com)Zeusbot robot for building the Ulsysseek.com index
4040Ziggy -- The Clown From Hell!!Unknown agent (server- or link checking ?) from 198.173.158.xx
4041ZipppBot/0.xx (ZipppBot; http://www.zippp.net; webmaster@zippp.net)Zipp.net web search robot
4042ZIPPPCVS/0.xx (ZipppBot/.xx;http://www.zippp.net; webmaster@zippp.net)Zipp.net web search robot
4043Zippy v2.0 - Zippyfinder.comZippyfinder robot
4044Zoo Tycoon 2 Client -- http://www.zootycoon.comMicrosoft Zoo Tycoon 2 game client
4045ZoomSpider - wrensoft.comZoom Search Engine software spider
4046Zscho.de Crawler/Nutch-1.0-Zscho.de-semantic_patch (Zscho.de Crawler collecting for machine learning; http://zscho.de/ )
4047zspider/0.9-dev http://feedback.redkolibri.com/zspider robot for a new search engine
4048ZyBorg/1.0 (ZyBorg@WISEnut.com; http://www.WISEnut.com)Wisenut robot
4049Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (FM Scene 4.6.1)FM Scene soccer fan pages plugin
4050Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (.NET CLR 3.5.30729) (Prevx 3.0.5)PrevX security plugin
Sabalico Logo
Sabali Mail Logo
Sabali Analytics Logo
SEO Guide Logo
Keyword Tool Logo
Domain Search Logo
What is my IP Logo
My Device Logo
My Browser Logo
My Location Logo
My Weather Logo
Test Speed Logo
Website On Logo
Page Load Logo
Code Editor Logo
Format Code Logo
Encode File Logo
Generator Password Logo
QR Code Generator Logo
Barcode Generator Logo
Online Sign Logo
Counter Word Logo
Lorem Ipsum Generator Logo
Sprite Sheet Logo
Resize Image Logo
Image Compress Logo
Image Color Logo
Color Picker Logo
Color Convert Logo
rStatistics Logo
Free Data Logo
Unit Convert Logo
Byte Convert Logo
Coordinate Converter Logo
Temperature Convert Logo
Text Convert Logo
Currency Convert Logo
Phone Country Logo
To-Do List Logo
Finance Calculator Logo
Sabalico Icon Powered by Sabalico™ 2012-2020 ©
Made in USA Sabalico Made in USA

Warning: file_get_contents(https://sabalico.dev/scripts/datatables.core.js): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in /home/sabalico/public_html/addons/domains/my-device.org/robots.php on line 164

Warning: file_get_contents(https://sabalico.dev/scripts/datatables.buttons.js): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in /home/sabalico/public_html/addons/domains/my-device.org/robots.php on line 165

Warning: file_get_contents(https://sabalico.dev/scripts/datatables.jszip.js): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in /home/sabalico/public_html/addons/domains/my-device.org/robots.php on line 166

Warning: file_get_contents(https://sabalico.dev/scripts/datatables.pdfs.js): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in /home/sabalico/public_html/addons/domains/my-device.org/robots.php on line 167

Warning: file_get_contents(https://sabalico.dev/scripts/datatables.fonts.js): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in /home/sabalico/public_html/addons/domains/my-device.org/robots.php on line 168

Warning: file_get_contents(https://sabalico.dev/scripts/datatables.html5.js): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in /home/sabalico/public_html/addons/domains/my-device.org/robots.php on line 169

Warning: file_get_contents(https://sabalico.dev/scripts/datatables.start.js): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found in /home/sabalico/public_html/addons/domains/my-device.org/robots.php on line 170