Hi,

As of yesterday, this forum's being scanned periodically by the bot MSNPTC. This appears to be some (new?) microsoft search engine bot.
The problem here is that this bot executes literally hundreds of simultaneous requests, that actually completely flood the forum!

I've now disallowed the bot in robots.txt hoping it will leave us alone now. If it doesn't I guess I'll have to go for heavier measures like firewalling the entire IP range it uses. This shouldn't be needed though - bots simply shouldn't act like DoS attacks

Does anyone know about this or has experienced the same? Google didn't tell me much about it after a first search.
I'm just curious if this is a known problem or if anyone heard anything about it.

hm I havent had problems with that one yet, but another bot downloaded my whole website like 4000 times in a row, causing about 33GB traffic in a couple hours...
However, its not new - this thread is from 2004:
http://www.webmasterworld.com/forum11/2451.htm

can't view that link :

Forbidden

You don't have permission to access /forum11/2451.htm on this server.

I can look at it without problems...

Maybe try a proxy?

Works for me too

Guess that site has UK IPs blocked, as I got an "cannot access the site" when using an UK based proxy...

i'm in holland remember?

Anyway, i could read that thread with that proxy link you showed. No real answers still. Nothing official anywhere. So I guess if i spot it flooding again, I'll just disallow access for msnptc altogether.

Short version of the thread for you, Victor:

• the discussion is about MSNPTC/1.0
  
• it has been seen under the IP-Adresses 131.107.3.74, 131.107.3.84, 207.46.238.143
  
• it is confirmed that it does (or did) NOT request the robots.txt
  
• wild guesses are going that its Partner-Bot msnbot relays the robots.txt to MSNPTC
  

Our team recently had the same problem with one of Yahoo!'s bots. We had no choice except actually Firewalling the bot's ip.

so this bot exists for 3 years now and still it's behaving in the same rediculous manner? Heh, goooooooooo MS!

c'mon, it's MS. What did you expect?

[EDIT: tags: microsoft bashing]

Quote from Victor :

so this bot exists for 3 years now and still it's behaving in the same rediculous manner? Heh, goooooooooo MS!

well, intercrap exploder exists since 12 years and its still bugridden rubbish

saved the page from above, even though I just read you already read it...

For what it's worth, the inference appears to be that it's an ad-quality/adsense-like bot from Microsoft

hmm ok thanks. The ip's i have logged were not in that range. So I may just firewall them then (the attacking range - not the range MS states there)

Seems it's been hitting my server for some time too, although it would appear (at least at this time), not viscously.. but the UA name appears to have changed too since it started:


Now it appears to hit almost daily. Recent hits are coming from the 207.46.92.0/24 range.


Regards,

Ian

Just use robots.txt and Disallow the MSNPTC/1.0 user agent.

If you want to deny an entire subnet, you're better off doing that at the firewall level.

Quote from JamesF1 :

Just use robots.txt and Disallow the MSNPTC/1.0 user agent.

But that only works if the robot honors the deal a htaccess file on the other hand would stop it on the server level.


Something like that should work, if it does not you can find more help here:
http://httpd.apache.org/docs/1.3/mod/mod_access.html#allow

[edit]Damn you angel, you are quick[/edit]

Quote from Dygear :

[edit]Damn you angel, you are quick[/edit]

I use apache too much :o

This reminds me of Lemon Deamon's flash "When robots attack".

"ROBOTS! ROBOTS! No1 is safe when Robots attack!" lol.

Quote from freddybob :

Sorry, what I meant was how do I allow this user_agent when it comes from the 65.54.xxx.yyy IP range but disallow it when it comes from any other IP?

I did not tested it, just a guess, no warranty :-). Notice also Ian.H's post about a new IP range.....

Quote from Dygear :

But that only works if the robot honors the deal a htaccess file on the other hand would stop it on the server level.

You are entirely right, sir. I shouldn't stray near the forums when I'm as shattered as I was when I posted that last night

I'm just saying, your right, but so am I.