Spiders, Crawlers and web robots - Forum Posters Union

		Forum Posters Union > Search Engine Intelligence & Research
Spiders, Crawlers and web robots

Spiders, Crawlers and web robots Intelligence on search engine spider bots and identification, bad bots from spam botnets, content scrapers, tools to identify web robots, blocking malicious bots.

Page 1 of 6

Threads in Forum : Spiders, Crawlers and web robots

Forum Tools

Search this Forum

Views: 221 Announcement: ALL POSTS are moderated due to spambots 01-10-2009 AnthonyCea (Publisher)
	Rating Thread / Thread Starter	Last Post	Replies	Views
	Sticky: Ban Spam Bots and Content Scraper Crawlers AnthonyCea	11-18-2009 10:06 PM by AnthonyCea	2	4,632
	Sticky: Block Bad Bots and malicious spam botnets AnthonyCea	11-17-2009 02:34 PM by AnthonyCea	6	3,361
	Sticky: Great Spider and Crawler Identification pages AnthonyCea	08-28-2009 12:07 PM by AnthonyCea	6	2,963
	Sticky: Web Robot and Content Scraper Bot Identification tools AnthonyCea	11-10-2007 09:14 PM by AnthonyCea	0	1,503
	QunarBot/1.0 AnthonyCea	09-08-2010 07:56 PM by AnthonyCea	18	489
	spbot/2.0.4 AnthonyCea	09-04-2010 07:13 PM by AnthonyCea	13	360
	X4d Backlinkchecker web robot AnthonyCea	09-04-2010 02:36 PM by AnthonyCea	0	28
	atraxbot/0.3 AnthonyCea	09-02-2010 02:55 PM by AnthonyCea	0	51
	DoCoMo/1.0/D209i user agent? wingnut1	08-29-2010 08:00 AM by wingnut1	0	40
	CyberPatrol SiteCat Webbot AnthonyCea	08-25-2010 07:37 AM by AnthonyCea	6	1,405
	Hailoobot/1.2 AnthonyCea	08-20-2010 03:22 PM by AnthonyCea	0	61
	eCairn-Grabber/1.0 curl/7.15 AnthonyCea	08-19-2010 04:02 AM by lfarcy	8	1,060
	HuaweiSymantec spider AnthonyCea	08-18-2010 06:49 PM by AnthonyCea	3	224
	goblox Bot Search AnthonyCea	08-17-2010 09:09 PM by AnthonyCea	0	89
	Spinn3r 3.1 AnthonyCea	08-14-2010 01:48 PM by AnthonyCea	3	698
	woriobot AnthonyCea	08-14-2010 09:59 AM by AnthonyCea	2	388
	TwitterBot attack AnthonyCea	08-12-2010 12:20 PM by AnthonyCea	4	203
	Yeti/1.0 AnthonyCea	08-09-2010 12:37 PM by AnthonyCea	9	1,642
	WWW-Mechanize/1.0.0 AnthonyCea	08-07-2010 06:03 PM by AnthonyCea	1	137
	EasyDL/3.04 AnthonyCea	08-07-2010 11:05 AM by AnthonyCea	0	85
	StatoolsBot AnthonyCea	08-06-2010 07:10 PM by AnthonyCea	0	74
	Yandex/1.01.001 ( 1 2) AnthonyCea	07-16-2010 01:44 PM by SkydivingForums	40	9,451
	Plukkie/1.3 AnthonyCea	07-13-2010 02:44 PM by AnthonyCea	0	145
	agbot/1.0 AnthonyCea	07-13-2010 12:06 PM by AnthonyCea	0	92
	MSR-ISRCCrawler AnthonyCea	07-13-2010 12:04 PM by AnthonyCea	6	1,353
	COMODOspider/Nutch-1.0 AnthonyCea	07-09-2010 07:54 AM by AnthonyCea	0	153
	linkfluence miqrogroove	07-05-2010 11:14 AM by miqrogroove	0	136
	Labrador/0.2 AnthonyCea	07-02-2010 12:41 PM by AnthonyCea	0	101
	Qualidator.com Bot 1.0 AnthonyCea	07-01-2010 08:34 PM by AnthonyCea	0	105
	Sycara Spider Bot AnthonyCea	07-01-2010 07:02 AM by AnthonyCea	0	99
	YandexBot/3.0 AnthonyCea	06-29-2010 06:40 PM by AnthonyCea	1	443
	Fake GoogleBot spoofed crawler AnthonyCea	06-28-2010 03:32 AM by miqrogroove	15	3,908
	LoneStarBot/1.06 AnthonyCea	06-27-2010 11:39 AM by WebshoppeSolutions	1	199
	TestBot/Nutch-1.1 on IP 184.73.84.188 AnthonyCea	06-19-2010 02:00 PM by AnthonyCea	0	175
	SemtoBot/0.1 AnthonyCea	06-17-2010 05:52 PM by AnthonyCea	0	185
	LeapTag Bot AnthonyCea	06-16-2010 11:30 AM by AnthonyCea	2	488
	Sogou Pic Spider/3.0 miqrogroove	06-16-2010 04:51 AM by miqrogroove	2	301
	Unidentified bot 64.233.172.18 is a Google IP AnthonyCea	06-15-2010 05:08 PM by AnthonyCea	9	1,129
	searchdnabot/Nutch-1.0 AnthonyCea	06-14-2010 02:30 PM by AnthonyCea	4	430
	Nuhk/3.0 miqrogroove	06-12-2010 05:04 AM by miqrogroove	2	155
	206.183.1.74 copilot.thunderstone.com bot AnthonyCea	06-10-2010 04:27 PM by AnthonyCea	2	1,210
	Twiceler-0.9 AnthonyCea	06-09-2010 02:01 PM by AnthonyCea	20	3,812
	IScraperBot/0.1 miqrogroove	06-08-2010 08:55 PM by AnthonyCea	3	564
	findfiles.net/0.96 miqrogroove	06-05-2010 08:15 AM by findfiles.net	8	363
	Sogou web spider/4.0 AnthonyCea	06-02-2010 07:17 AM by AnthonyCea	0	200
	spbot/2.0.3 AnthonyCea	05-31-2010 07:59 AM by AnthonyCea	0	222
	httpunit/1.5 AnthonyCea	05-30-2010 07:58 PM by AnthonyCea	1	231
	S2Bot/1.0 AnthonyCea	05-29-2010 04:17 PM by AnthonyCea	0	147
	pirst 110.45.148.160 AnthonyCea	05-29-2010 01:43 PM by AnthonyCea	0	273
	psbot/0.1 AnthonyCea	05-29-2010 11:02 AM by AnthonyCea	0	148
	Infoaxe./Nutch-0.9 AnthonyCea	05-24-2010 03:18 PM by AnthonyCea	0	161
	Y!J-BSC/1.0 AnthonyCea	05-22-2010 05:39 PM by AnthonyCea	0	149
	Y!J-BRO/YFSJ crawler AnthonyCea	05-22-2010 05:38 PM by AnthonyCea	0	155
	Yandex/2.01.000 AnthonyCea	05-20-2010 05:39 AM by AnthonyCea	1	376
	SSL Labs Bot (alpha.ssllabs.com) AnthonyCea	05-18-2010 06:38 AM by AnthonyCea	0	258
	DotBot/1.0.1 AnthonyCea	05-17-2010 06:03 AM by AnthonyCea	6	1,113
	spbot/2.0.2 AnthonyCea	05-14-2010 09:27 AM by AnthonyCea	17	814
	Search17Bot/1.1 AnthonyCea	05-12-2010 03:44 PM by AnthonyCea	0	269
	Jomjaibot/1.0 Crawl AnthonyCea	05-12-2010 03:17 PM by AnthonyCea	0	247
	production01.observer.at AnthonyCea	05-10-2010 08:56 AM by AnthonyCea	0	176
	Yandex/1.02.000 AnthonyCea	05-09-2010 05:24 PM by AnthonyCea	4	531
	sSearch Crawler on IP 85.10.204.138 AnthonyCea	05-08-2010 04:56 PM by AnthonyCea	0	183
	Survey Robot AnthonyCea	05-08-2010 11:03 AM by AnthonyCea	0	176
	suchmaschinenoptimierung.de SEO tools bot AnthonyCea	05-05-2010 01:20 PM by AnthonyCea	0	217
	bixolabs/1.0 AnthonyCea	05-05-2010 11:05 AM by AnthonyCea	4	277
	Unidentified bot running from VeriSign.com data center AnthonyCea	05-05-2010 09:23 AM by AnthonyCea	4	1,370
	adgooroo.com content scanner bot on IP 65.47.180.46 AnthonyCea	05-03-2010 05:48 PM by AnthonyCea	0	356
	Baiduspider China Search Engine Bot AnthonyCea	05-03-2010 06:05 AM by AnthonyCea	25	2,398
	Willow Internet Crawler by Twotrees V2.1 AnthonyCea	04-30-2010 10:55 AM by AnthonyCea	0	218
	bitlybot miqrogroove	04-28-2010 07:53 AM by AnthonyCea	2	547
	NaverBot/1.0 Korean Search Engine Naver AnthonyCea	04-27-2010 07:40 AM by AnthonyCea	5	765
	How to Block the Amazon AWS EC2 miqrogroove	04-23-2010 07:47 PM by kabo0m	9	926
	Daumoa/2.0 miqrogroove	04-16-2010 05:47 AM by miqrogroove	2	371
	Websense bot IP range AnthonyCea	04-12-2010 08:55 AM by orasis	2	562
	Semager/1.4 miqrogroove	04-10-2010 04:07 PM by AnthonyCea	1	272
	Purebot/1.1 - Malicious Content Scraper and Spam Agent miqrogroove	04-08-2010 12:49 PM by miqrogroove	4	1,863
	New msnbot IP range and user agents AnthonyCea	04-07-2010 06:14 PM by AnthonyCea	5	543
	SBIder/Nutch-1.0-dev AnthonyCea	04-05-2010 09:50 PM by AnthonyCea	0	214
	seoprofiler.com/bot/ is real trouble in more ways than one AnthonyCea	04-05-2010 09:57 AM by AnthonyCea	0	555
	Links4Later/1.0 AnthonyCea	04-03-2010 01:59 PM by AnthonyCea	0	206
	yacybot P2P search engine spider AnthonyCea	04-01-2010 10:02 AM by AnthonyCea	4	741
	69.36.233.15 Robot Operated by StumbleUpon miqrogroove	03-30-2010 08:14 AM by AnthonyCea	10	694
	TurnitinBot/2.1 AnthonyCea	03-29-2010 10:10 PM by miqrogroove	1	332
	Boardreader spider crawling forums ( 1 2) AnthonyCea	03-28-2010 06:27 PM by AnthonyCea	43	2,653
	Stealth bot from Yahoo on IP 209.191.87.216 AnthonyCea	03-26-2010 06:22 AM by AnthonyCea	0	286
	SimilarPages/Nutch-1.0-dev AnthonyCea	03-26-2010 03:37 AM by orasis	24	2,019
	mxbot/1.0 AnthonyCea	03-25-2010 06:48 PM by AnthonyCea	0	405
	abby/1.0 ellerdale.com/crawler AnthonyCea	03-25-2010 06:35 PM by AnthonyCea	0	362
	Netcraft Web Server Survey AnthonyCea	03-24-2010 04:01 PM by AnthonyCea	1	966
	Moreoverbot/5.00 AnthonyCea	03-22-2010 06:15 AM by AnthonyCea	4	561
	Cityreview Robot AnthonyCea	03-17-2010 09:35 AM by AnthonyCea	0	231
	DoubleVerify Crawler AnthonyCea	03-14-2010 05:53 AM by AnthonyCea	1	357
	CatchBot/1.0 AnthonyCea	03-13-2010 03:45 PM by AnthonyCea	0	327
	NetcraftSurveyAgent/1.0 AnthonyCea	03-11-2010 12:55 PM by AnthonyCea	0	388
	spbot/2.0.1 AnthonyCea	03-11-2010 05:26 AM by AnthonyCea	2	327
	msnbot-UDiscovery miqrogroove	03-10-2010 07:15 AM by AnthonyCea	2	328
	Wasabot miqrogroove	03-04-2010 07:00 AM by miqrogroove	4	513
	Speedy Spider AnthonyCea	03-03-2010 05:26 AM by AnthonyCea	6	1,083
	spbot/2.0 AnthonyCea	03-02-2010 09:40 AM by AnthonyCea	6	612
	Jigsaw/2.2.5 miqrogroove	03-01-2010 04:45 AM by AnthonyCea	1	235
	L.webis/0.47 AnthonyCea	02-25-2010 04:33 PM by AnthonyCea	0	287
	Fake MSNBot Spider out of China AnthonyCea	02-24-2010 10:17 AM by AnthonyCea	4	1,132
	FairShare-http://fairshare.cc bot AnthonyCea	02-22-2010 08:46 PM by AnthonyCea	18	1,453
	R6_FeedFetcher radian6.com/crawler AnthonyCea	02-22-2010 07:31 PM by AnthonyCea	16	2,543

Page 1 of 6

Forum Tools	Search this Forum
Mark This Forum Read View Parent Forum	Search this Forum : Advanced Search

	New posts		Hot thread with new posts
	No new posts		Hot thread with no new posts
	Thread is closed

Posting Rules
You may not post new threads You may not post replies You may not post attachments You may not edit your posts vB code is On Smilies are On [IMG] code is On HTML code is Off

Forum Jump

All times are GMT -7. The time now is 12:22 PM.

ForumPostersUnion.com