View previous topic :: View next topic |
Author |
Message |
pluby The Architect
Joined: Jun 16, 2003 Posts: 11949
|
Posted: Wed Apr 16, 2008 11:30 am Post subject: Abusive bandwidth usage |
|
FYI. After getting a $600 excess bandwidth usage bill from our web hosting provider, I went through our web statistics and found that a small number of IP addresses were responsible for more than 50% of our total bandwidth usage!
All but one of these IP addresses were web proxy servers located in mainland China. While I understand that China's "content filtering" is down by routing all users through a small number of proxies, I find it hard to believe that 50% of our users are in mainland China. More likely, one or a few machine are repeatedly downloading every single NeoOffice binary for no good reason.
The other IP address was Google's search bot. Even though that server's robots.txt has a Disallow: / entry, Google's search bot still insisted on indexing every single NeoOffice binary.
Since this kind of abuse now costs real money, I have specifically disallowed access to the NeoOffice binaries for the following networks. I have also pushed this change to all of the NeoOffice mirrors running Apache as well as they are likely getting the same abuse of their bandwidth:
60.176.
66.249.73.
121.63.
125.70.
I have left open access to our main website, Trinity, Bugzilla, and the NeoWiki so that users in these networks can see why they cannot download from their mirrors and can either find alternate routes to our servers or can complain about the abuse to their ISP.
Patrick |
|
Back to top |
|
|
djpimley The Anomaly (earlier version)
Joined: Jun 11, 2006 Posts: 481 Location: Great Britain
|
Posted: Wed Apr 16, 2008 11:55 am Post subject: Re: Abusive bandwidth usage |
|
pluby wrote: | FYI. After getting a $600 excess bandwidth usage bill from our web hosting provider |
Ouch! |
|
Back to top |
|
|
ovvldc Captain Naiobi
Joined: Sep 13, 2004 Posts: 2352 Location: Zürich, CH
|
Posted: Wed Apr 16, 2008 2:03 pm Post subject: |
|
Maybe someone is downloading the whole internet so that they can play with it at home? Or maybe the Chinese government wants its citizens to use a cached and appropriately purged version of the web?
Anyway, I am sorry to see that even Google doesn't play by the rules.
Best wishes,
Oscar _________________ "What do you think of Western Civilization?"
"I think it would be a good idea!"
- Mohandas Karamchand Gandhi |
|
Back to top |
|
|
pluby The Architect
Joined: Jun 16, 2003 Posts: 11949
|
Posted: Thu Apr 17, 2008 8:48 am Post subject: |
|
Well woudn't you know that after blocking the subnets listed above, hundreds of users magically appeared yesterday from other Chinese subnets.
What makes me laugh is that ever one of these Chinese subnets uses Windows NT running Internet Explorer so it is pretty clear to me that these are not Mac users but stupid, bandwidth-wasting search bots.
Anyway, here is the current list of blocked subnets:
60.176.
66.249.73.
121.63.
123.154.
125.70.
218.10.
218.79.
220.160.
222.35.
Patrick |
|
Back to top |
|
|
ovvldc Captain Naiobi
Joined: Sep 13, 2004 Posts: 2352 Location: Zürich, CH
|
Posted: Thu Apr 17, 2008 11:06 am Post subject: |
|
What, you've accidentally stumbled upon the set of IP adresses used by the Great Firewall? Or is there just a whole lot of China-based web search engines? |
|
Back to top |
|
|
pluby The Architect
Joined: Jun 16, 2003 Posts: 11949
|
Posted: Fri Apr 25, 2008 11:48 am Post subject: |
|
About a week ago, I did some intensive review of our weblogs and I think that I found some pretty reasonable criteria for identifying the abusive robot requests while still allowing real users access to the downloads.
I have had this new filtering in place for nearly a week now and bandwidth usage has been very stable during that time. This is good news as my new filtering criteria allow Mac OS X users from the previously blocked Chinese subnets access.
Patrick |
|
Back to top |
|
|
|