ForumPostersUnion.com


   

Go Back   Forum Posters Union > Search Engine Intelligence & Research > Spiders, Crawlers and web robots
Reload this Page Yandex/1.01.001
Register FAQ Members List Calendar Search Today's Posts Mark Forums Read

Spiders, Crawlers and web robots Intelligence on search engine spider bots and identification, bad bots from spam botnets, content scrapers, tools to identify web robots, blocking malicious bots.

Reply
Page 1 of 2 1 2 >
 
Thread Tools
  #1  
Old 08-08-2008, 07:13 AM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
Yandex/1.01.001
77.88.22.111 walrus036.yandex.ru
Yandex/1.01.001 (compatible; Win16; H)

The above IP, host name and user agent is the signature from YandexBot.

Russia based Yandex.com is a legitimate Russian language search engine, Yandex owns the largest market share of any pure Russian based search engine.

It seems that Yandex is going to crawl websites worldwide to build a giant database enabling them to compete with Google in the marketplace.

Yandex.com needs to add a link to a spider identification page listing the IP's they crawl from in their user agent so webmasters do not ban the bot.
Reply With Quote
  #2  
Old 09-29-2008, 06:46 PM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
The bot from yandex.ru was back tonight with a new IP, this is a legitimate crawler that should be allowed to crawl your websites.

08:46 PM Guest Viewing Forum
Forum Software 77.88.22.159 walrus117.yandex.ru
Yandex/1.01.001 (compatible; Win16; H)

Information related to '77.88.22.0 - 77.88.23.255'

inetnum: 77.88.22.0 - 77.88.23.255
netname: YANDEX-22-0
descr: Yandex enterprise network
country: RU
admin-c: YNDX1-RIPE
tech-c: YNDX1-RIPE
remarks: INFRA-AW
status: ASSIGNED PA
mnt-by: YANDEX-MNT
source: RIPE # Filtered

role: Yandex LLC Network Operations
address: Yandex LLC
address: 1 bld. 21 Samokatnaya St.
address: 111033
address: Moscow
address: Russian Federation
phone: +7 495 739 7000
fax-no: +7 495 739 7070
remarks: trouble: ------------------------------------------------------
remarks: trouble: Points of contact for Yandex LLC Network Operations
remarks: trouble: ------------------------------------------------------
remarks: trouble: Routing and peering issues: noc@yandex.net
remarks: trouble: SPAM issues: abuse@yandex.ru
remarks: trouble: Network security issues: abuse@yandex.ru
remarks: trouble: Mail issues: postmaster@yandex.ru
remarks: trouble: General information: info@yandex.ru
remarks: trouble: ------------------------------------------------------
admin-c: VLI1-RIPE
admin-c: TVB11-RIPE
tech-c: VLI1-RIPE
nic-hdl: YNDX1-RIPE
mnt-by: YANDEX-MNT
source: RIPE # Filtered
abuse-mailbox: abuse@yandex.ru

% Information related to '77.88.0.0/18AS13238'

route: 77.88.0.0/18
descr: Yandex enterprise network
origin: AS13238
mnt-by: YANDEX-MNT
source: RIPE # Filtered

% Information related to '77.88.22.0/24AS13238'

route: 77.88.22.0/24
descr: Yandex enterprise network
origin: AS13238
mnt-by: YANDEX-MNT
source: RIPE # Filtered

% Information related to '77.88.22.0/23AS13238'

route: 77.88.22.0/23
descr: Yandex enterprise network
origin: AS13238
mnt-by: YANDEX-MNT
source: RIPE # Filtered
Reply With Quote
  #3  
Old 12-09-2008, 03:39 PM
miqrogroove's Avatar
miqrogroove miqrogroove is offline
Senior Member
  
Join Date: Dec 2008
Posts: 306
FYI this bot also commonly identifies as


213.180.214.175
YandexSomething/1.0

213.180.214.175
YandexBlog/0.99.101 (compatible; DOS3.30; Mozilla/5.0; B; robot)
Reply With Quote
  #4  
Old 12-09-2008, 03:51 PM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
I have not researched those IP's.

Do Whois results come back to Yandex ??

Many hackers and spammers spoof legitimate search bots to fool webmasters.
Reply With Quote
  #5  
Old 12-09-2008, 03:59 PM
miqrogroove's Avatar
miqrogroove miqrogroove is offline
Senior Member
  
Join Date: Dec 2008
Posts: 306
Code:
C:\>nslookup 213.180.214.175
Server:  miqrogroove.info-svc.com
Address:  192.168.4.8

Name:    m8b.feeds.yandex.net
Address:  213.180.214.175


C:\>nslookup m8b.feeds.yandex.net
Server:  miqrogroove.info-svc.com
Address:  192.168.4.8

Name:    m8b.feeds.yandex.net
Address:  213.180.214.175
Reply With Quote
  #6  
Old 12-09-2008, 04:04 PM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
Great information, thanks for posting this, it helps other webmasters who really are worried now days due to bad bots and professional spambot operations.
Reply With Quote
  #7  
Old 12-14-2008, 02:42 PM
miqrogroove's Avatar
miqrogroove miqrogroove is offline
Senior Member
  
Join Date: Dec 2008
Posts: 306
This time,
Host: 213.180.214.178

So it's definitely not a single agent.
Reply With Quote
  #8  
Old 12-14-2008, 02:49 PM
miqrogroove's Avatar
miqrogroove miqrogroove is offline
Senior Member
  
Join Date: Dec 2008
Posts: 306
Also,
Host: 93.158.130.160

Code:
C:\>nslookup 93.158.130.160
Server:  miqrogroove.info-svc.com
Address:  192.168.4.8

Name:    robot03d.feeds.yandex.net
Address:  93.158.130.160


C:\>nslookup robot03d.feeds.yandex.net
Server:  miqrogroove.info-svc.com
Address:  192.168.4.8

Name:    robot03d.feeds.yandex.net
Address:  93.158.130.160
Reply With Quote
  #9  
Old 12-14-2008, 03:24 PM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
This company is legitimate, but being from Russia they will have a hard row to hoe with a lot of webmasters thinking they are running bad bots.
Reply With Quote
  #10  
Old 12-14-2008, 03:27 PM
miqrogroove's Avatar
miqrogroove miqrogroove is offline
Senior Member
  
Join Date: Dec 2008
Posts: 306
It's also hard to pin down having 3+ subnets and 2+ TLDs. Do you want me to post additional subnets if I find them?
Reply With Quote
  #11  
Old 12-14-2008, 03:48 PM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
Sure, we never discourage members when it comes to posting, as long as it is relevant to Yandex.

They are free to post also if they have a problem with what you post and they may do just that !
Reply With Quote
  #12  
Old 12-15-2008, 12:19 PM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
93.158.150.21 spider64.yandex.ru
Yandex/1.01.001 (compatible; Win16; I)
Reply With Quote
  #13  
Old 12-16-2008, 01:31 AM
miqrogroove's Avatar
miqrogroove miqrogroove is offline
Senior Member
  
Join Date: Dec 2008
Posts: 306
Code:
C:\>nslookup 87.250.243.199
Server:  miqrogroove.info-svc.com
Address:  192.168.4.8

Name:    m11b.feeds.yandex.net
Address:  87.250.243.199


C:\>nslookup m11b.feeds.yandex.net
Server:  miqrogroove.info-svc.com
Address:  192.168.4.8

Name:    m11b.feeds.yandex.net
Address:  87.250.243.199
Reply With Quote
  #14  
Old 12-30-2008, 06:47 PM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
93.158.148.30 spider10.yandex.ru
Yandex/1.01.001 (compatible; Win16; I)
Reply With Quote
  #15  
Old 12-30-2008, 10:50 PM
miqrogroove's Avatar
miqrogroove miqrogroove is offline
Senior Member
  
Join Date: Dec 2008
Posts: 306
Host: 77.88.26.26 /
Http Code: 406 Date: Dec 30 22:43:49 Http Version: HTTP/1.1 Size in Bytes: 52807
Agent: Yandex/1.01.001 (compatible; Win16; I)

It's causing 406's now, very naughty!
Reply With Quote
  #16  
Old 12-31-2008, 06:39 PM
miqrogroove's Avatar
miqrogroove miqrogroove is offline
Senior Member
  
Join Date: Dec 2008
Posts: 306
Host: 77.88.17.137
Agent: YandexBlogs/0.99.101 (compatible; Mozilla/5.0; robot)
Reply With Quote
  #17  
Old 01-23-2009, 04:45 PM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
06:41 PM Guest Viewing Forum MAC OS Software

77.88.41.220 dech091.yandex.ru
Yandex/1.01.001 (compatible; Win16; H)
Reply With Quote
  #18  
Old 05-23-2009, 05:34 PM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
77.88.42.25 spider52.yandex.ru
Yandex/1.01.001 (compatible; Win16; I)
Reply With Quote
  #19  
Old 06-04-2009, 05:17 AM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
07:11 AM Guest Viewing Index
77.88.42.25 spider52.yandex.ru
Yandex/1.01.001 (compatible; Win16; I)
Reply With Quote
  #20  
Old 08-28-2009, 07:03 PM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
08:56 PM Guest Viewing Index
95.108.142.150 htest01.yandex.ru
Yandex/1.01.001 (compatible; Win16; H)
Reply With Quote
  #21  
Old 09-01-2009, 09:14 AM
Mur Mur is offline
Member
  
Join Date: Sep 2009
Posts: 4
Quote:
Originally Posted by AnthonyCea View Post
08:56 PM Guest Viewing Index
95.108.142.150 htest01.yandex.ru
Yandex/1.01.001 (compatible; Win16; H)
IP address: 95.108.196.252
Host name: spider07.yandex.ru
Tuesday, September 01, 2009 9:19 AM

Good reading about this bot.

I've always classified bots that skip the robots.txt file as bad bots and add them to my redirect list.

This bot ignores my robots.txt disallow and offers no information from their site on how to disallow it. (From what I can see)

It's just my Opinion but.. If standard rules aren't followed I don't see how they will compete as a search engine.

Anyway, if they are a pure Russian search the bot should see My Language settings Western 1033 in the header and just pass me up.
Reply With Quote
  #22  
Old 09-01-2009, 09:29 AM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
Welcome to the forum Mur !!

Every webmaster must be in control of the bots or crawlers they grant permission to crawl their web content or to have free access to their web servers, we simply try to ban all bad bots here.

At this point we do not classify YandexBot as a bad bot simply because they seem to have a legitimate search engine, but you should be suspect of all Russian search engines since many of them like WebAlta are infested by and controlled by spam botnet operators and may be used as a front for spam harvesting operations.

Remember that Yandex.com also has a lot of English users and Yandex.ru presents search results in English also.
Reply With Quote
  #23  
Old 09-09-2009, 05:56 PM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
95.108.150.235 sticker00.yandex.ru
Yandex/1.01.001 (compatible; Win16; H)

Information related to '95.108.128.0 - 95.108.255.255'

inetnum: 95.108.128.0 - 95.108.255.255
org: ORG-YA1-RIPE
netname: RU-YANDEX-20081209
descr: YANDEX LLC
country: RU
admin-c: YNDX1-RIPE
tech-c: YNDX1-RIPE
status: ALLOCATED PA
mnt-by: RIPE-NCC-HM-MNT
mnt-lower: YANDEX-MNT
mnt-routes: YANDEX-MNT
source: RIPE # Filtered

organisation: ORG-YA1-RIPE
org-name: YANDEX LLC
org-type: LIR
address: Yandex LLC
Vladimir Ivanov
1 bld. 21 Samokatnaya St.
111033 Moscow
RUSSIAN FEDERATION
phone: +7 495 739 7000
fax-no: +7 495 739 7070
admin-c: GB90-RIPE
admin-c: TVB11-RIPE
admin-c: VLI1-RIPE
admin-c: AUR2-RIPE
mnt-ref: RIPE-NCC-HM-MNT
mnt-ref: YANDEX-MNT
mnt-by: RIPE-NCC-HM-MNT
source: RIPE # Filtered

role: Yandex LLC Network Operations
address: Yandex LLC
address: 1 bld. 21 Samokatnaya St.
address: 111033
address: Moscow
address: Russian Federation
phone: +7 495 739 7000
fax-no: +7 495 739 7070
remarks: trouble: ------------------------------------------------------
remarks: trouble: Points of contact for Yandex LLC Network Operations
remarks: trouble: ------------------------------------------------------
remarks: trouble: Routing and peering issues: noc@yandex.net
remarks: trouble: SPAM issues: abuse@yandex.ru
remarks: trouble: Network security issues: abuse@yandex.ru
remarks: trouble: Mail issues: postmaster@yandex.ru
remarks: trouble: General information: info@yandex.ru
remarks: trouble: ------------------------------------------------------
admin-c: VLI1-RIPE
admin-c: TVB11-RIPE
tech-c: VLI1-RIPE
nic-hdl: YNDX1-RIPE
mnt-by: YANDEX-MNT
source: RIPE # Filtered
abuse-mailbox: abuse@yandex.ru

% Information related to '95.108.128.0/17AS13238'

route: 95.108.128.0/17
descr: Yandex enterprise network
origin: AS13238
mnt-by: YANDEX-MNT
source: RIPE # Filtered
Reply With Quote
  #24  
Old 09-16-2009, 07:35 AM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
95.108.142.150 htest01.yandex.ru
Yandex/1.01.001 (compatible; Win16; H)
Reply With Quote
  #25  
Old 09-24-2009, 11:47 AM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
Yandex enterprise network IP range
95.108.142.0 - 95.108.142.255
95.108.142.134 ghad.yandex.ru
95.108.142.135 unresolved
95.108.142.136 unresolved
95.108.142.137 unresolved
95.108.142.138 unresolved
95.108.142.139 unresolved
95.108.142.140 grade.yandex.ru
95.108.142.141 grade01.yandex.ru
95.108.142.142 picus.yandex.ru
95.108.142.143 arachnid00.yandex.ru
95.108.142.144 arachnid02.yandex.ru
95.108.142.145 arachnid01.yandex.ru
95.108.142.146 seal000.yandex.ru
95.108.142.147 seal001.yandex.ru
95.108.142.148 seal002.yandex.ru
95.108.142.149 seal003.yandex.ru
95.108.142.150 htest01.yandex.ru
95.108.142.151 waltest01.yandex.ru
95.108.142.152 waltest02.yandex.ru
95.108.142.153 arachnid03.yandex.ru
95.108.142.154 quicktest00.yandex.ru
95.108.142.155 ws-int000.yandex.ru
95.108.142.156 unresolved
95.108.142.157 unresolved
95.108.142.158 unresolved
95.108.142.159 unresolved
95.108.142.160 wstest00.yandex.ru
95.108.142.161 wstest01.yandex.ru
95.108.142.162 wstest02.yandex.ru
95.108.142.163 wstest03.yandex.ru
95.108.142.164 wstest04.yandex.ru
95.108.142.165 wstest05.yandex.ru
95.108.142.166 wstest06.yandex.ru

Nameservers
ns1.yandex.net 213.180.193.1 RIPE Network Coordination Centre
ns4.yandex.net 77.88.19.60 RIPE NCC

WHOIS Record
Data retrieved from whois.ripe.net at 2009-09-24 18:45 GMT

% Information related to '95.108.142.0 - 95.108.142.255'
inetnum: 95.108.142.0 - 95.108.142.255
netname: YANDEX-95-108-142-0
descr: Yandex enterprise network
country: RU
admin-c: YNDX1-RIPE
tech-c: YNDX1-RIPE
remarks: INFRA-AW
status: ASSIGNED PA
mnt-by: YANDEX-MNT
source: RIPE # Filtered

role: Yandex LLC Network Operations
address: Yandex LLC
address: 1 bld. 21 Samokatnaya St.
address: 111033
address: Moscow
address: Russian Federation
phone: +7 495 739 7000
fax-no: +7 495 739 7070
remarks: trouble: ------------------------------------------------------
remarks: trouble: Points of contact for Yandex LLC Network Operations
remarks: trouble: ------------------------------------------------------
remarks: trouble: Routing and peering issues: noc@yandex.net
remarks: trouble: SPAM issues: abuse@yandex.ru
remarks: trouble: Network security issues: abuse@yandex.ru
remarks: trouble: Mail issues: postmaster@yandex.ru
remarks: trouble: General information: info@yandex.ru
remarks: trouble: ------------------------------------------------------
admin-c: VLI1-RIPE
admin-c: TVB11-RIPE
tech-c: VLI1-RIPE
nic-hdl: YNDX1-RIPE
mnt-by: YANDEX-MNT
source: RIPE # Filtered
abuse-mailbox: abuse@yandex.ru

% Information related to '95.108.128.0/17AS13238'

route: 95.108.128.0/17
descr: Yandex enterprise network
origin: AS13238
mnt-by: YANDEX-MNT
source: RIPE # Filtered
Reply With Quote
  #26  
Old 10-13-2009, 06:24 AM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
08:14 AM Guest Viewing Forum

77.88.30.247 spider42.yandex.ru
Yandex/1.01.001 (compatible; Win16; I)

Information related to '77.88.28.0 - 77.88.31.255'

inetnum: 77.88.28.0 - 77.88.31.255
netname: YANDEX-28
descr: Yandex enterprise network
country: RU
admin-c: YNDX1-RIPE
tech-c: YNDX1-RIPE
remarks: INFRA-AW
status: ASSIGNED PA
mnt-by: YANDEX-MNT
source: RIPE # Filtered

role: Yandex LLC Network Operations
address: Yandex LLC
address: 1 bld. 21 Samokatnaya St.
address: 111033
address: Moscow
address: Russian Federation
phone: +7 495 739 7000
fax-no: +7 495 739 7070
remarks: trouble: ------------------------------------------------------
remarks: trouble: Points of contact for Yandex LLC Network Operations
remarks: trouble: ------------------------------------------------------
remarks: trouble: Routing and peering issues: noc@yandex.net
remarks: trouble: SPAM issues: abuse@yandex.ru
remarks: trouble: Network security issues: abuse@yandex.ru
remarks: trouble: Mail issues: postmaster@yandex.ru
remarks: trouble: General information: info@yandex.ru
remarks: trouble: ------------------------------------------------------
admin-c: VLI1-RIPE
admin-c: TVB11-RIPE
tech-c: VLI1-RIPE
nic-hdl: YNDX1-RIPE
mnt-by: YANDEX-MNT
source: RIPE # Filtered
abuse-mailbox: abuse@yandex.ru

% Information related to '77.88.0.0/18AS13238'

route: 77.88.0.0/18
descr: Yandex enterprise network
origin: AS13238
mnt-by: YANDEX-MNT
source: RIPE # Filtered

% Information related to '77.88.30.0/24AS13238'

route: 77.88.30.0/24
descr: Yandex enterprise network
origin: AS13238
mnt-by: YANDEX-MNT
source: RIPE # Filtered

% Information related to '77.88.28.0/22AS13238'

route: 77.88.28.0/22
descr: Yandex enterprise network
origin: AS13238
mnt-by: YANDEX-MNT
source: RIPE # Filtered

% Information related to '77.88.24.0/21AS13238'

route: 77.88.24.0/21
descr: Yandex enterprise network
origin: AS13238
mnt-by: YANDEX-MNT
source: RIPE # Filtered
Reply With Quote
  #27  
Old 10-24-2009, 12:46 PM
miqrogroove's Avatar
miqrogroove miqrogroove is offline
Senior Member
  
Join Date: Dec 2008
Posts: 306
Since I happen to be surfing here anyway, figured I'll do some updates for ya

77.88.17.195 - - [22/Oct/2009:23:44:12 -0600] "GET /robots.txt HTTP/1.1" 200 526 "-" "YandexSomething/1.0"

93.158.130.185 - - [23/Oct/2009:11:09:08 -0600] "GET /robots.txt HTTP/1.1" 200 526 "-" "YandexSomething/1.0"

93.158.149.31 - - [22/Oct/2009:21:00:05 -0600] "GET /de/category/style/eyewear/ HTTP/1.1" 200 66771 "-" "Yandex/1.01.001 (compatible; Win16; I)"

95.108.147.183 - - [23/Oct/2009:09:11:08 -0600] "GET /feed/ HTTP/1.1" 304 334 "-" "YandexBlog/0.99.101 (compatible; DOS3.30; Mozilla/5.0; B; robot) 1 readers"

95.108.147.185 - - [23/Oct/2009:00:39:14 -0600] "GET /robots.txt HTTP/1.1" 200 526 "-" "YandexSomething/1.0"

95.108.147.237 - - [22/Oct/2009:22:41:08 -0600] "GET /feed/ HTTP/1.1" 200 53537 "-" "YandexBlog/0.99.101 (compatible; DOS3.30; Mozilla/5.0; B; robot) 1 readers"

95.108.147.238 - - [23/Oct/2009:02:39:11 -0600] "GET /robots.txt HTTP/1.1" 200 526 "-" "YandexSomething/1.0"

95.108.147.240 - - [22/Oct/2009:19:42:33 -0600] "GET /robots.txt HTTP/1.1" 200 526 "-" "YandexSomething/1.0"

95.108.147.241 - - [22/Oct/2009:21:43:07 -0600] "GET /feed/ HTTP/1.1" 200 53192 "-" "YandexBlog/0.99.101 (compatible; DOS3.30; Mozilla/5.0; B; robot) 1 readers"

95.108.147.242 - - [22/Oct/2009:18:45:34 -0600] "GET /robots.txt HTTP/1.1" 200 526 "-" "YandexSomething/1.0"

These are, respectively:
robot01b.feeds.yandex.net
robot05d.feeds.yandex.net
spider12.yandex.ru
robot03e.feeds.yandex.net
robot05e.feeds.yandex.net
robot07e.feeds.yandex.net
robot08e.feeds.yandex.net
robot10e.feeds.yandex.net
robot11e.feeds.yandex.net
robot12e.feeds.yandex.net
Reply With Quote
  #28  
Old 10-24-2009, 12:48 PM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
The Russians are coming, the Russians are coming !!!
Reply With Quote
  #29  
Old 11-27-2009, 09:38 PM
WebshoppeSolutions's Avatar
WebshoppeSolutions WebshoppeSolutions is offline
Member
  
Join Date: Oct 2009
Location: Southeast Texas
Posts: 7
The behaviour of Yandex is quite a lot like that of Google with regard to robots.txt .. The bot doesn't look at the robots.txt every single time it enters the domain.

Bots like Yandex, Baudi, and Sohu have all been fairly well behaved and as a result, are allowed. None of them have ever gone places I didn't want them to go, and parse rates don't break the bank with regard to bandwidth.

I'll be including these bots in my next robots.txt generator write, right along side of Yahoo, Bing, and Google ..
Reply With Quote
  #30  
Old 12-01-2009, 08:19 AM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
That sounds like a nice file you are going to write.

YandexBot hit us with this IP and host name today.

10:09 AM Guest Viewing Index
77.88.50.29 dech038.yandex.ru
Yandex/1.01.001 (compatible; Win16; H)
Reply With Quote
  #31  
Old 12-05-2009, 05:50 AM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
07:38 AM Guest Unknown Location
/links/browselinks.php?c=25
77.88.42.27 spider50.yandex.ru
Yandex/1.01.001 (compatible; Win16; I)
Reply With Quote
  #32  
Old 12-07-2009, 07:12 AM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
09:07 AM Guest Viewing Forum
93.158.151.25 spider66.yandex.ru
Yandex/1.01.001 (compatible; Win16; I)
Reply With Quote
  #33  
Old 12-29-2009, 02:43 PM
Ninjatiamat Ninjatiamat is offline
BANNED
  
Join Date: Dec 2009
Posts: 20
Thumbs down
How do I block yandex? This fucker averaged about 5gb bandwith/day by itself!!!
Reply With Quote
  #34  
Old 12-29-2009, 03:24 PM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
Yeah, YandexBot has been very active in the last few weeks for sure.

One way is to block is via robots.txt, I'm sure they honor it.

The only other way is to ban IP c-networks or ban by CIDR, but CIDR is not for the novice, you can screw up big time if you do not know what you are doing.
Reply With Quote
  #35  
Old 03-26-2010, 09:03 PM
SoundLizard SoundLizard is offline
Member
  
Join Date: Mar 2010
Posts: 1
Blocking for PhP users
To block this crawler from your site using php, simply copy this code into a text document. then upload it to your server in whichever directory you wish to block them from. rename it as .htaccess.
if you already have a .htaccess file, simply add the code to the existing file. It will pop up a forbidden page for them to see. if you want to block them from your whole site, simply install it into your root directory. here's the code for the main ip that ive seen them use...



order allow,deny
deny from 77.88.29.247
allow from all
Reply With Quote
  #36  
Old 05-10-2010, 07:38 AM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
09:26 AM Guest Viewing Forum

77.88.42.26 spider54.yandex.ru
Yandex/1.01.001 (compatible; Win16; I)

09:25 AM Guest Viewing Forum

93.158.151.25 spider66.yandex.ru
Yandex/1.01.001 (compatible; Win16; I)

We have not blocked Yandex here due to the fact that this is a legitimate search engine.
Reply With Quote
  #37  
Old 06-11-2010, 08:34 AM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
10:25 AM Guest Viewing Thread

95.108.240.251 spider02.yandex.ru
Yandex/1.01.001 (compatible; Win16; I)

inetnum: 95.108.240.0 - 95.108.240.255
netname: YANDEX-95-108-240
descr: Yandex enterprise network
country: RU
admin-c: YNDX1-RIPE
tech-c: YNDX1-RIPE
remarks: INFRA-AW
status: ASSIGNED PA
mnt-by: YANDEX-MNT
source: RIPE # Filtered

role: Yandex LLC Network Operations
address: Yandex LLC
address: 1 bld. 21 Samokatnaya St.
address: 111033
address: Moscow
address: Russian Federation
phone: +7 495 739 7000
fax-no: +7 495 739 7070
remarks: trouble: ------------------------------------------------------
remarks: trouble: Points of contact for Yandex LLC Network Operations
remarks: trouble: ------------------------------------------------------
remarks: trouble: Routing and peering issues: noc@yandex.net
remarks: trouble: SPAM issues: abuse@yandex.ru
remarks: trouble: Network security issues: abuse@yandex.ru
remarks: trouble: Mail issues: postmaster@yandex.ru
remarks: trouble: General information: info@yandex.ru
remarks: trouble: ------------------------------------------------------
admin-c: VLI1-RIPE
admin-c: TVB11-RIPE
tech-c: VLI1-RIPE
nic-hdl: YNDX1-RIPE
mnt-by: YANDEX-MNT
source: RIPE # Filtered
abuse-mailbox: abuse@yandex.ru

% Information related to '95.108.128.0/17AS13238'

route: 95.108.128.0/17
descr: Yandex enterprise network
origin: AS13238
mnt-by: YANDEX-MNT
source: RIPE # Filtered

% Information related to '95.108.240.0/21AS13238'

route: 95.108.240.0/21
descr: Yandex network
origin: AS13238
mnt-by: YANDEX-MNT
Reply With Quote
  #38  
Old 06-15-2010, 06:37 PM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
95.108.142.138 plane03.yandex.ru
Yandex/1.01.001 (compatible; Win16; H)
Reply With Quote
  #39  
Old 07-16-2010, 01:24 PM
SkydivingForums's Avatar
SkydivingForums SkydivingForums is offline
Member
  
Join Date: Oct 2009
Posts: 37
Lightbulb Yandex...
Hello Anthony,

I was wondering if the Russian search engine spiders should be left alone.

I 'm receiving a lot of activity from them, and noticed that my site is now listed in their search engines (great for me) at yandex.com

I have also noticed on other forums that there is in fact malicious code injections that redirect their forums to yandex. Their entire forum was infected on profiles, and all kinds of internal links. Although, I need to find out the rest of the details.

Just wondering what you have heard and what should be done about these aggressive crawlers...Thanks!
__________________
http://www.skydive-info.com
Reply With Quote
  #40  
Old 07-16-2010, 01:29 PM
AnthonyCea's Avatar
AnthonyCea AnthonyCea is offline
Publisher
  
Join Date: Feb 2006
Location: Deep South, USA
Posts: 29,228
Yandex is a legitimate Russian search engine, we have never blocked any of their IP's and have not had any problems with them.

Maybe Russian hackers or competitors are hacking forums and sending traffic to Yandex, but I highly doubt they are involved in this sort of activity, it may be designed to hurt them since they are the leading Russian language search engine.

If you have no need to be indexed by them, blocking Yandex by user agent via .htaccess is the way to go.
Reply With Quote
Reply
Page 1 of 2 1 2 >

« Previous Thread | Next Thread »


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -7. The time now is 05:00 AM.

Contact Us - Forum Posters Union - Archive - Top

Powered by vBulletin®
Copyright ©2000 - 2010, Jelsoft Enterprises Ltd.
2006-2009 ForumPostersUnion.com