Welcome to NeoOffice developer notes and announcements
NeoOffice
Developer notes and announcements
 
 

This website is an archive and is no longer active
NeoOffice announcements have moved to the NeoOffice News website


Support
· Forums
· NeoOffice Support
· NeoWiki


Announcements
· Twitter @NeoOffice


Downloads
· Download NeoOffice


  
NeoOffice :: View topic - Abusive bandwidth usage
Abusive bandwidth usage
 
   NeoOffice Forum Index -> Server Outages
View previous topic :: View next topic  
Author Message
pluby
The Architect
The Architect


Joined: Jun 16, 2003
Posts: 11949

PostPosted: Wed Apr 16, 2008 11:30 am    Post subject: Abusive bandwidth usage

FYI. After getting a $600 excess bandwidth usage bill from our web hosting provider, I went through our web statistics and found that a small number of IP addresses were responsible for more than 50% of our total bandwidth usage!

All but one of these IP addresses were web proxy servers located in mainland China. While I understand that China's "content filtering" is down by routing all users through a small number of proxies, I find it hard to believe that 50% of our users are in mainland China. More likely, one or a few machine are repeatedly downloading every single NeoOffice binary for no good reason.

The other IP address was Google's search bot. Even though that server's robots.txt has a Disallow: / entry, Google's search bot still insisted on indexing every single NeoOffice binary.

Since this kind of abuse now costs real money, I have specifically disallowed access to the NeoOffice binaries for the following networks. I have also pushed this change to all of the NeoOffice mirrors running Apache as well as they are likely getting the same abuse of their bandwidth:

60.176.
66.249.73.
121.63.
125.70.

I have left open access to our main website, Trinity, Bugzilla, and the NeoWiki so that users in these networks can see why they cannot download from their mirrors and can either find alternate routes to our servers or can complain about the abuse to their ISP.

Patrick
Back to top
djpimley
The Anomaly
(earlier version)


Joined: Jun 11, 2006
Posts: 481
Location: Great Britain

PostPosted: Wed Apr 16, 2008 11:55 am    Post subject: Re: Abusive bandwidth usage

pluby wrote:
FYI. After getting a $600 excess bandwidth usage bill from our web hosting provider

Ouch! Shocked Sad
Back to top
ovvldc
Captain Naiobi


Joined: Sep 13, 2004
Posts: 2352
Location: Zürich, CH

PostPosted: Wed Apr 16, 2008 2:03 pm    Post subject:

Maybe someone is downloading the whole internet so that they can play with it at home? Or maybe the Chinese government wants its citizens to use a cached and appropriately purged version of the web?

Anyway, I am sorry to see that even Google doesn't play by the rules.

Best wishes,
Oscar

_________________
"What do you think of Western Civilization?"
"I think it would be a good idea!"
- Mohandas Karamchand Gandhi
Back to top
pluby
The Architect
The Architect


Joined: Jun 16, 2003
Posts: 11949

PostPosted: Thu Apr 17, 2008 8:48 am    Post subject:

Well woudn't you know that after blocking the subnets listed above, hundreds of users magically appeared yesterday from other Chinese subnets. Wink

What makes me laugh is that ever one of these Chinese subnets uses Windows NT running Internet Explorer so it is pretty clear to me that these are not Mac users but stupid, bandwidth-wasting search bots.

Anyway, here is the current list of blocked subnets:

60.176.
66.249.73.
121.63.
123.154.
125.70.
218.10.
218.79.
220.160.
222.35.

Patrick
Back to top
ovvldc
Captain Naiobi


Joined: Sep 13, 2004
Posts: 2352
Location: Zürich, CH

PostPosted: Thu Apr 17, 2008 11:06 am    Post subject:

What, you've accidentally stumbled upon the set of IP adresses used by the Great Firewall? Or is there just a whole lot of China-based web search engines? Question
Back to top
pluby
The Architect
The Architect


Joined: Jun 16, 2003
Posts: 11949

PostPosted: Fri Apr 25, 2008 11:48 am    Post subject:

About a week ago, I did some intensive review of our weblogs and I think that I found some pretty reasonable criteria for identifying the abusive robot requests while still allowing real users access to the downloads.

I have had this new filtering in place for nearly a week now and bandwidth usage has been very stable during that time. This is good news as my new filtering criteria allow Mac OS X users from the previously blocked Chinese subnets access.

Patrick
Back to top
Display posts from previous:   
   NeoOffice Forum Index -> Server Outages All times are GMT - 7 Hours
Page 1 of 1

 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum

Powered by phpBB © 2001, 2005 phpBB Group

All logos and trademarks in this site are property of their respective owner. The comments are property of their posters, all the rest © Planamesa Inc.
NeoOffice is a registered trademark of Planamesa Inc. and may not be used without permission.
PHP-Nuke Copyright © 2005 by Francisco Burzi. This is free software, and you may redistribute it under the GPL. PHP-Nuke comes with absolutely no warranty, for details, see the license.