Spiders, Crawlers and web robots - Forum Posters Union

		Forum Posters Union > Search Engine Intelligence & Research
Spiders, Crawlers and web robots

Spiders, Crawlers and web robots Intelligence on search engine spider bots and identification, bad bots from spam botnets, content scrapers, tools to identify web robots, blocking malicious bots.

Page 1 of 6

Threads in Forum : Spiders, Crawlers and web robots

Forum Tools

Search this Forum

Views: 221 Announcement: ALL POSTS are moderated due to spambots 01-10-2009 AnthonyCea (Publisher)
	Rating Thread / Thread Starter	Last Post	Replies	Views
	Sticky: Ban Spam Bots and Content Scraper Crawlers AnthonyCea	11-18-2009 10:06 PM by AnthonyCea	2	4,594
	Sticky: Block Bad Bots and malicious spam botnets AnthonyCea	11-17-2009 02:34 PM by AnthonyCea	6	3,320
	Sticky: Great Spider and Crawler Identification pages AnthonyCea	08-28-2009 12:07 PM by AnthonyCea	6	2,932
	Sticky: Web Robot and Content Scraper Bot Identification tools AnthonyCea	11-10-2007 09:14 PM by AnthonyCea	0	1,492
	atraxbot/0.3 AnthonyCea	Yesterday 02:55 PM by AnthonyCea	0	10
	spbot/2.0.4 AnthonyCea	09-01-2010 07:09 PM by AnthonyCea	12	323
	QunarBot/1.0 AnthonyCea	08-30-2010 09:59 AM by AnthonyCea	13	374
	Moved: PANTECH G200 spambot user agent on IP 60.217.249.146 wingnut1	-	-	-
	DoCoMo/1.0/D209i user agent? wingnut1	08-29-2010 08:00 AM by wingnut1	0	23
	CyberPatrol SiteCat Webbot AnthonyCea	08-25-2010 07:37 AM by AnthonyCea	6	1,382
	Hailoobot/1.2 AnthonyCea	08-20-2010 03:22 PM by AnthonyCea	0	42
	eCairn-Grabber/1.0 curl/7.15 AnthonyCea	08-19-2010 04:02 AM by lfarcy	8	1,047
	HuaweiSymantec spider AnthonyCea	08-18-2010 06:49 PM by AnthonyCea	3	196
	goblox Bot Search AnthonyCea	08-17-2010 09:09 PM by AnthonyCea	0	70
	Spinn3r 3.1 AnthonyCea	08-14-2010 01:48 PM by AnthonyCea	3	685
	woriobot AnthonyCea	08-14-2010 09:59 AM by AnthonyCea	2	378
	TwitterBot attack AnthonyCea	08-12-2010 12:20 PM by AnthonyCea	4	177
	Yeti/1.0 AnthonyCea	08-09-2010 12:37 PM by AnthonyCea	9	1,623
	WWW-Mechanize/1.0.0 AnthonyCea	08-07-2010 06:03 PM by AnthonyCea	1	114
	EasyDL/3.04 AnthonyCea	08-07-2010 11:05 AM by AnthonyCea	0	74
	StatoolsBot AnthonyCea	08-06-2010 07:10 PM by AnthonyCea	0	62
	Yandex/1.01.001 ( 1 2) AnthonyCea	07-16-2010 01:44 PM by SkydivingForums	40	9,311
	Plukkie/1.3 AnthonyCea	07-13-2010 02:44 PM by AnthonyCea	0	121
	agbot/1.0 AnthonyCea	07-13-2010 12:06 PM by AnthonyCea	0	82
	MSR-ISRCCrawler AnthonyCea	07-13-2010 12:04 PM by AnthonyCea	6	1,333
	COMODOspider/Nutch-1.0 AnthonyCea	07-09-2010 07:54 AM by AnthonyCea	0	141
	linkfluence miqrogroove	07-05-2010 11:14 AM by miqrogroove	0	128
	Labrador/0.2 AnthonyCea	07-02-2010 12:41 PM by AnthonyCea	0	93
	Qualidator.com Bot 1.0 AnthonyCea	07-01-2010 08:34 PM by AnthonyCea	0	97
	Sycara Spider Bot AnthonyCea	07-01-2010 07:02 AM by AnthonyCea	0	91
	YandexBot/3.0 AnthonyCea	06-29-2010 06:40 PM by AnthonyCea	1	416
	Fake GoogleBot spoofed crawler AnthonyCea	06-28-2010 03:32 AM by miqrogroove	15	3,891
	LoneStarBot/1.06 AnthonyCea	06-27-2010 11:39 AM by WebshoppeSolutions	1	189
	TestBot/Nutch-1.1 on IP 184.73.84.188 AnthonyCea	06-19-2010 02:00 PM by AnthonyCea	0	164
	SemtoBot/0.1 AnthonyCea	06-17-2010 05:52 PM by AnthonyCea	0	172
	LeapTag Bot AnthonyCea	06-16-2010 11:30 AM by AnthonyCea	2	478
	Sogou Pic Spider/3.0 miqrogroove	06-16-2010 04:51 AM by miqrogroove	2	290
	Unidentified bot 64.233.172.18 is a Google IP AnthonyCea	06-15-2010 05:08 PM by AnthonyCea	9	1,111
	searchdnabot/Nutch-1.0 AnthonyCea	06-14-2010 02:30 PM by AnthonyCea	4	412
	Nuhk/3.0 miqrogroove	06-12-2010 05:04 AM by miqrogroove	2	140
	206.183.1.74 copilot.thunderstone.com bot AnthonyCea	06-10-2010 04:27 PM by AnthonyCea	2	1,190
	Twiceler-0.9 AnthonyCea	06-09-2010 02:01 PM by AnthonyCea	20	3,799
	IScraperBot/0.1 miqrogroove	06-08-2010 08:55 PM by AnthonyCea	3	546
	findfiles.net/0.96 miqrogroove	06-05-2010 08:15 AM by findfiles.net	8	341
	Sogou web spider/4.0 AnthonyCea	06-02-2010 07:17 AM by AnthonyCea	0	189
	spbot/2.0.3 AnthonyCea	05-31-2010 07:59 AM by AnthonyCea	0	205
	httpunit/1.5 AnthonyCea	05-30-2010 07:58 PM by AnthonyCea	1	223
	S2Bot/1.0 AnthonyCea	05-29-2010 04:17 PM by AnthonyCea	0	135
	pirst 110.45.148.160 AnthonyCea	05-29-2010 01:43 PM by AnthonyCea	0	247
	psbot/0.1 AnthonyCea	05-29-2010 11:02 AM by AnthonyCea	0	140
	Infoaxe./Nutch-0.9 AnthonyCea	05-24-2010 03:18 PM by AnthonyCea	0	153
	Y!J-BSC/1.0 AnthonyCea	05-22-2010 05:39 PM by AnthonyCea	0	139
	Y!J-BRO/YFSJ crawler AnthonyCea	05-22-2010 05:38 PM by AnthonyCea	0	144
	Yandex/2.01.000 AnthonyCea	05-20-2010 05:39 AM by AnthonyCea	1	351
	SSL Labs Bot (alpha.ssllabs.com) AnthonyCea	05-18-2010 06:38 AM by AnthonyCea	0	243
	DotBot/1.0.1 AnthonyCea	05-17-2010 06:03 AM by AnthonyCea	6	1,097
	spbot/2.0.2 AnthonyCea	05-14-2010 09:27 AM by AnthonyCea	17	793
	Search17Bot/1.1 AnthonyCea	05-12-2010 03:44 PM by AnthonyCea	0	255
	Jomjaibot/1.0 Crawl AnthonyCea	05-12-2010 03:17 PM by AnthonyCea	0	228
	production01.observer.at AnthonyCea	05-10-2010 08:56 AM by AnthonyCea	0	168
	Yandex/1.02.000 AnthonyCea	05-09-2010 05:24 PM by AnthonyCea	4	521
	sSearch Crawler on IP 85.10.204.138 AnthonyCea	05-08-2010 04:56 PM by AnthonyCea	0	169
	Survey Robot AnthonyCea	05-08-2010 11:03 AM by AnthonyCea	0	167
	suchmaschinenoptimierung.de SEO tools bot AnthonyCea	05-05-2010 01:20 PM by AnthonyCea	0	206
	bixolabs/1.0 AnthonyCea	05-05-2010 11:05 AM by AnthonyCea	4	260
	Unidentified bot running from VeriSign.com data center AnthonyCea	05-05-2010 09:23 AM by AnthonyCea	4	1,347
	adgooroo.com content scanner bot on IP 65.47.180.46 AnthonyCea	05-03-2010 05:48 PM by AnthonyCea	0	337
	Baiduspider China Search Engine Bot AnthonyCea	05-03-2010 06:05 AM by AnthonyCea	25	2,377
	Willow Internet Crawler by Twotrees V2.1 AnthonyCea	04-30-2010 10:55 AM by AnthonyCea	0	208
	bitlybot miqrogroove	04-28-2010 07:53 AM by AnthonyCea	2	529
	NaverBot/1.0 Korean Search Engine Naver AnthonyCea	04-27-2010 07:40 AM by AnthonyCea	5	752
	How to Block the Amazon AWS EC2 miqrogroove	04-23-2010 07:47 PM by kabo0m	9	892
	Daumoa/2.0 miqrogroove	04-16-2010 05:47 AM by miqrogroove	2	362
	Websense bot IP range AnthonyCea	04-12-2010 08:55 AM by orasis	2	543
	Semager/1.4 miqrogroove	04-10-2010 04:07 PM by AnthonyCea	1	259
	Purebot/1.1 - Malicious Content Scraper and Spam Agent miqrogroove	04-08-2010 12:49 PM by miqrogroove	4	1,821
	New msnbot IP range and user agents AnthonyCea	04-07-2010 06:14 PM by AnthonyCea	5	532
	SBIder/Nutch-1.0-dev AnthonyCea	04-05-2010 09:50 PM by AnthonyCea	0	203
	seoprofiler.com/bot/ is real trouble in more ways than one AnthonyCea	04-05-2010 09:57 AM by AnthonyCea	0	535
	Links4Later/1.0 AnthonyCea	04-03-2010 01:59 PM by AnthonyCea	0	197
	yacybot P2P search engine spider AnthonyCea	04-01-2010 10:02 AM by AnthonyCea	4	731
	69.36.233.15 Robot Operated by StumbleUpon miqrogroove	03-30-2010 08:14 AM by AnthonyCea	10	678
	TurnitinBot/2.1 AnthonyCea	03-29-2010 10:10 PM by miqrogroove	1	318
	Boardreader spider crawling forums ( 1 2) AnthonyCea	03-28-2010 06:27 PM by AnthonyCea	43	2,624
	Stealth bot from Yahoo on IP 209.191.87.216 AnthonyCea	03-26-2010 06:22 AM by AnthonyCea	0	272
	SimilarPages/Nutch-1.0-dev AnthonyCea	03-26-2010 03:37 AM by orasis	24	1,999
	mxbot/1.0 AnthonyCea	03-25-2010 06:48 PM by AnthonyCea	0	385
	abby/1.0 ellerdale.com/crawler AnthonyCea	03-25-2010 06:35 PM by AnthonyCea	0	343
	Netcraft Web Server Survey AnthonyCea	03-24-2010 04:01 PM by AnthonyCea	1	944
	Moreoverbot/5.00 AnthonyCea	03-22-2010 06:15 AM by AnthonyCea	4	539
	Cityreview Robot AnthonyCea	03-17-2010 09:35 AM by AnthonyCea	0	221
	DoubleVerify Crawler AnthonyCea	03-14-2010 05:53 AM by AnthonyCea	1	349
	CatchBot/1.0 AnthonyCea	03-13-2010 03:45 PM by AnthonyCea	0	310
	NetcraftSurveyAgent/1.0 AnthonyCea	03-11-2010 12:55 PM by AnthonyCea	0	363
	spbot/2.0.1 AnthonyCea	03-11-2010 05:26 AM by AnthonyCea	2	317
	msnbot-UDiscovery miqrogroove	03-10-2010 07:15 AM by AnthonyCea	2	320
	Wasabot miqrogroove	03-04-2010 07:00 AM by miqrogroove	4	502
	Speedy Spider AnthonyCea	03-03-2010 05:26 AM by AnthonyCea	6	1,063
	spbot/2.0 AnthonyCea	03-02-2010 09:40 AM by AnthonyCea	6	603
	Jigsaw/2.2.5 miqrogroove	03-01-2010 04:45 AM by AnthonyCea	1	226
	L.webis/0.47 AnthonyCea	02-25-2010 04:33 PM by AnthonyCea	0	277
	Fake MSNBot Spider out of China AnthonyCea	02-24-2010 10:17 AM by AnthonyCea	4	1,107
	FairShare-http://fairshare.cc bot AnthonyCea	02-22-2010 08:46 PM by AnthonyCea	18	1,431
	R6_FeedFetcher radian6.com/crawler AnthonyCea	02-22-2010 07:31 PM by AnthonyCea	16	2,521

Page 1 of 6

Forum Tools	Search this Forum
Mark This Forum Read View Parent Forum	Search this Forum : Advanced Search

	New posts		Hot thread with new posts
	No new posts		Hot thread with no new posts
	Thread is closed

Posting Rules
You may not post new threads You may not post replies You may not post attachments You may not edit your posts vB code is On Smilies are On [IMG] code is On HTML code is Off

Forum Jump

All times are GMT -7. The time now is 04:58 AM.

ForumPostersUnion.com