Blocked 85.255.118.116 for requesting 9030 pages in under 5 minutes. That's 30 pages a second.

Come on, everybody will agree THAT is not normal.

*splutters and spills coffee*
30 a second?! where do these people get the bandwidth?

Blocked 85.182.33.32, hundreds of request in a row. This one even managed to get to load up to 8+.

you think they'd apply some thought and be a little more discrete, rather than making a lot of noise that you're bound to notice lol
fools

Willy,

Have you had any scraping from 194.97.107.175? Over the last 10 days, the IP has pulled about 500,000 pages from our server over at RieselSieve (yeah, I just noticed today). It seems to be the same 40 pages or so in 1-minute intervals. Thought I'd check, because I can't figure out the interest in userw.php.

Not a single hit on BS.

I used to get dozens of hits on userw.php from different IP addresses. The user-agent string was empty or at least surely not corresponding to a mobile phone. They're probably using it because the output is smaller than show_user.php, so it's good for scraping stats. I modified userw.php to give a message instead of returning the stats.

Blocked IP 216.32.90.58. This IP can't even get the index anymore. I never seen a scraper capable of getting even the old server as high as 120 load.

<rant mode>
You sir, are an idiot (of course you need another IP to be able to read this).

And no, switching browsers doesn't get you past the captcha. I wait, you didn't see that, because you kept requesting pages even though all you got was the captcha.
</rant mode>

[17:13:51] <Soul_keeper> http://sterlingdesktops.com/boincrace/test.png chart now auto updates daily at 10:10am PST
[17:32:16] <PovAddict> Soul_keeper: you aren't HTML-scraping boincstats.com are you?
[17:32:23] <Soul_keeper> PovAddict, yeah
[17:35:07] <PovAddict> what makes you think you're allowed to do that
[17:35:13] <PovAddict> http://boincstats.com/forum/forum_thread.php?id=1161
[17:35:50] <Soul_keeper> if they didn't want me accessing their website then it shouldn't be on the internet
[17:36:23] <PovAddict> that's like saying "if you didn't want me to steal that image you shouldn't have put it on the internet", just because it's on the internet doesn't mean anyone can copy it

I like how the parent directory (http://sterlingdesktops.com/boincrace/) does a directory listing, with plain text scripts that independently display for the world that scraping is exactly what he's doing.

His image files are showing quite recent "last updated" timestamps... I think Willy blocked the wrong IP or something.

Scraping.....??

Could someone describe in general what it is, and what benefits come from it?

**Nevermind** - just read the original post.

I assume the negatives for BOINCstats would be the extra bandwidth that is being taken....

And the extra demand on the CPU and database.

Scrapers profit from (read: steal) the work of others and present the data as their own.

Darn scrapers! Do your own work!!!!

Been monitoring it and the image still shows January 25th, so I think I've got him.

It's a shame that I was singled out for an example to be made of, I only wget 3 stats pages a day at a several minute delayed stagger.
If this is considered a load on your server as opposed to viewing the same 3 sites serveral times a day to compare results, then I appologize.

Originally I had it grabbing about 8 sites, then I was told by someone this is frowned on "scraping" so I reduced it to 3 and added a sleep delay.
This DOES use equal/less bandwidth than just viewing the 3 sites in my web browser, which this supplements.

Sounds to me like someone has nothing better to do than look for someone to mess with.

I've explained why I don't allow scraping and I also said I would take measures to prevent it. Be it one page or a thousand, it's my site and I decide whether I allow it or not.

You were warned not to scrape yet you still went on with it. Now live with the consequence.

Willy can come and see the full log with its context then. But I think there's no context that can 'save' things like "then their site is a load of crap", or the line at 16:47:32 which I won't repeat here.

Heh, that's funny. Making a donation costs about five minutes. You've been scraping for over four months, you have a BAM! account for more than two years. Couldn't find the time??

Anyway, your last post convinced me not to lift the ban.

If that´s the way you see it, I´d recommend you take yourself elsewhere and stop annoying people.

Just a thought ...

already done.


some random parting thoughts:

I did the calculations, I was taking a whipping 36KB per day from this website (lynx -dump 3 12KB pages staggered which means 12KB every min+, much less than viewing the website ONCE since I didn't grab any images)
for example if this site has 100KB/s upload that would take a whopping .4 seconds or so to download if done all at once.

to put that in perspective those smiley face icons that pop up every time someone posts a forum message take atleast 45KB alone. And viewing a single forum post by willy takes over 100KB .....

I think I hurt someone's feelings, I wasn't banned for "scrapping". povaddict decided he wanted to exercise his "power" on a website where he had pull because he felt threatened/insecure.


anyways this is my last post.
hopefully you all learned something.

you're welcome

Yes, well, you already said that once before.



This tme we hope you mean it and make it permanent!