Is this thread dead? No response for 6 weeks.

Concerning the free availability of LFSW stats which is limited by IP address and not (as it may seem) by PubStat key. My application using the data (see previous post) currently runs on 500servers and may appear on other server-hosting sites. I guess all servers (or many of them) on such site use the same IP address. Does it mean they will block each other, rendering the free access to LFSW data unreliable/useless?

Thanks for an answer/solution!

Quote from EQ Worry :

Is this thread dead? No response for 6 weeks.

Concerning the free availability of LFSW stats which is limited by IP address and not (as it may seem) by PubStat key. My application using the data (see previous post) currently runs on 500servers and may appear on other server-hosting sites. I guess all servers (or many of them) on such site use the same IP address. Does it mean they will block each other, rendering the free access to LFSW data unreliable/useless?

Thanks for an answer/solution!

As I cannot duplicate the problem, I can't really help you. The only person that could answer that question is on the dev team and they have remained silent. This might be due to the fact there is no real fix for this sort of issue, atlest none that I can think of at this point in time.

Checking IP and Key, and making sure they don't belong to the same account would be the only way I could think of allowing people to share an IP address while not allowing people to make a million keys and cycle though them as needed on a request basis. Checking that the IP & Key does not belong to the same user is the only fix I could come up with.

Well, I think it is relatively easy to test it. If you do not have 2 (or more) accounts yet, create one demo, generate key and try to send two requests for LFSW data using the keys of two different people one right after another (using a browser). The 2nd request will be refused, because it is coming from the same IP.

When I was testing this I also discovered a rather interesting fact. If you use 2 computers with different IPs, you may actually use the same key belonging to the same person in quick succession without getting the refuse message.

My conclusion is that the PubStat key does not play any role in "tarpitting" the service. It is the IP that matters and that is IMO very wrong. My proposal is simple: Allow only licensed people to have PubStat keys, but do not use IP in limiting the access, because it is a very unsafe way very likely to prevent even legitimate usage of LFSW data fully complying with set conditions.

And if three licensed people join their keys in one application to be able to get all data faster, is it a real problem? Obviously most people have no personal use of this service. The main thing is to remove this access from demo accounts, leave it only to S1/2, but without checking IP. Many IPs are shared, many, especially in LFS servers.

Quote from EQ Worry :

The main thing is to remove this access from demo accounts, leave it only to S1/2, but without checking IP.

Who cares if they are demo users, if they have a positive balance in their account for premium stats that's really all that matters. I would agree that Tar Pitting the IP address is wrong, and I can only assume a mistake. I'll see if I can't get Victors attention over this, see what he says about the matter.

Thanks a lot! I'd really be interested to know of some applicable solution from the author of the system.

The IP check was derived from an anti-flooding check that's been in place from the start. I can do that check super quick, while on the other hand first having to check userid requires a database query.

But I've had a thought for a while now to do away with tarpitting altogether and do it like google for example : only allow a limited number of requests per user per day. Then I can do away with the flood check, because that was put in place because certain users could not make their script wait 5 seconds and started to hammer the pubstats with requests until the 5 seconds tarpit were over.

And I might remake the pubstats one final time actually, because it's still a bit shabby due to my former inexperience. For example no error codes and I know there are some internal inefficiencies too.
I'd keep the current pubstat still working, but the new version 1.5 (and onwards) would output a bit different data then before ... or should it be a whole new url?

Hi Victor,

if you really plan to change output formats, I would prefer a new URL and a parallel availabilty of 1.4 and 1.5 for some time.

This would give us the chance to work on our tools while they are still running with 1.4.

Thanks!

Thanks for the reply Vic, much appreciated.

I assume you mean with the IP Tar pitting method that you currently have implemented, your using a database query with the IP Address of Client and the Time of the Last Query in a database with it's storage type set to memory, hence the very fast part. While that works wonderful on the old 'nameless client' system that you had running, I'm not sure if it still remains the best solution in the case of the mandatory IDKey implementation.

Further more I would have to add that, I was my thought that you did not wish the Pub Stats to be stored, it was meant to provide a much more 'live' experience to the end user. Where as what your currently proposing would pretty much mandate the use of storage to keep content flowing to the end user, or require a system (such as a cron task) that would run in the background every X amount of minutes as to not quite add up to the total query limit of the day and still provide a almost live experience. That being the only way to provide that 'live' status that was the original intention of the PubStats.

I'm a big fan of error handling when possible, so adding the error codes to the pubstats system could only be a good thing. With the current version 1.4 and the new version of 1.5 and onwards I don't really need a new URL and I would slightly discourage it as you have kind of set a sudo standard with the current convention. (Yes, I know that statement goes against some W3C document that I can't find at this time.) Makes no difference to me though.

Hi Victor and thanks for the reply! Generally I'm happy with the PubStats access, the data it offers, the form it uses, the way tarpitting works (5 seconds delay). I'm struggling with the IP check because if my application runs in several instances on one LFS servers farm, whey will in effect block each other, solution not possible (except when all instances use paid access, but that would, I guess, never happen).

I can understand the IP check is easy and fast, but it is in my view too limiting (just as was the IP kicking/banning used in demo prior to patch Z). Also limiting number of requests per user per day would not have the desired effect for you, I'm afraid, because anyone can have any number of demo accounts and demo people may generate PubStats keys. I believe such mechanism would also include database queries increasing some access count in each account. Maybe this DB access could be used rather for the key check. But of course I'm no expert in these matters.

My general feeling as a user is that it is very nice as it is now, just the IP check (although quick) should be replaced by the supplied key check. And to prevent people using cyclically keys from multiple accounts, limiting this access only to licensed accounts may be a good protection for you.

One again, thanks for the previous reply. Have a nice day!

Quote from Dygear :

Thanks for the reply Vic, much appreciated.

I assume you mean with the IP Tar pitting method that you currently have implemented, your using a database query with the IP Address of Client and the Time of the Last Query in a database with it's storage type set to memory, hence the very fast part. While that works wonderful on the old 'nameless client' system that you had running, I'm not sure if it still remains the best solution in the case of the mandatory IDKey implementation.

Further more I would have to add that, I was my thought that you did not wish the Pub Stats to be stored, it was meant to provide a much more 'live' experience to the end user. Where as what your currently proposing would pretty much mandate the use of storage to keep content flowing to the end user, or require a system (such as a cron task) that would run in the background every X amount of minutes as to not quite add up to the total query limit of the day and still provide a almost live experience. That being the only way to provide that 'live' status that was the original intention of the PubStats.

I'm a big fan of error handling when possible, so adding the error codes to the pubstats system could only be a good thing. With the current version 1.4 and the new version of 1.5 and onwards I don't really need a new URL and I would slightly discourage it as you have kind of set a sudo standard with the current convention. (Yes, I know that statement goes against some W3C document that I can't find at this time.) Makes no difference to me though.

well i use shared memory to store IP info and time, so i don't have to even connect to the database to check ip stuff. But that's besides the point for now.

as for the live experience for the user - i wouldn't be talking about 100 queries a day, but rather something like 10000 per day. If one really needs more than that then premium should be used.
But if noone really minds the tarpitting .. eh well i could just change the ip check to an idkey check and leave it at that.
I'll have a look soon.

Quote from Victor :

i wouldn't be talking about 100 queries a day, but rather something like 10000 per day. If one really needs more than that then premium should be used.

OMG, I see we're talking here about numbers different by several orders of magnitude. To say the truth, I was afraid it will be limited to something like 100 requests per day, 1000 at best. I'll be happy with 10K that as long as it is not counted per IP address but per key or account.

Quote from Victor :

But if noone really minds the tarpitting .. eh well i could just change the ip check to an idkey check and leave it at that.

Well, if that is easier for you... The good thing on this approach is it will remove a hidden obstacle without any code changes necessary on user's part. And it may be a quick and easy patch for you as well (I hope).

Would someone mind testing the changes?
I've a test url for that - it should output exactly the same stuff, but internally it's rewritten a bit and now tarpitting by user (no matter which or how many idkey's are used, unless of course with premium access)

The url is the same, but instead of the script name get_stat2.php, use get_stat2t.php
If everything appears to be ok I'll replace the original one with this.

---

a general error message note which I think I've not mentioned before is :

"You shouldn't flood me! You're locked out for 15 minutes now. DO NOT make your program do its requests as fast as it can, until the tarpit is over..."

This happens if you really hammer the script and do more than 15 queries within 3 seconds. There aren't many people who see this, but as dygear does the LFSWSDK he might want to know about this.

Quote from Victor :

"You shouldn't flood me! You're locked out for 15 minutes now. DO NOT make your program do its requests as fast as it can, until the tarpit is over..."

This happens if you really hammer the script and do more than 15 queries within 3 seconds. There aren't many people who see this, but as dygear does the LFSWSDK he might want to know about this.

Thanks going to be interesting to program for without some kind of persistent save mechanism, or I guess I could just die on this statement from LFSWorld.

Quote from Victor :

Would someone mind testing the changes?

OK, I've done some quick testing. From my perspective it looks good, very good. It seems the sent keys really matter and if they are from two different accounts, each one is tarpitted separately, no matter the requests come from the same IP. Two keys from the same account are tarpitted as one, as I would expect.

It looks that with this update my primary concern is a history. Thanks a million, both to Victor and also to Dygear who, I believe, called Victor's attention to my post(s). Again, thanks a lot.

Quote from EQ Worry :

Dygear who, I believe, called Victor's attention to my post(s). Again, thanks a lot.

Nope, not I, Victors just being his ever vigilant self. Credit goes to him.

Is it normal when it says can't reload this page that quickly after another?

Because when more people visit the page and do the request, this message will appear to them

Quote from Flame CZE :

Is it normal when it says can't reload this page that quickly after another?
Because when more people visit the page and do the request, this message will appear to them

Yes, very normal, it is the whole purpose of tarpitting. There must a pause of 5 seconds between requests coming from the same person (or earlier the same IP address), or this message is displayed...

Quote from EQ Worry :

Yes, very normal, it is the whole purpose of tarpitting. There must a pause of 5 seconds between requests coming from the same person (or earlier the same IP address), or this message is displayed...

ah but victor said 15 requests per 3 seconds

thats bad for me, I cant output more than one request

nono, the 15 requests per 3 seconds give you a 15 minute ban from pubstats. This is a bad thing!

Normal tarpitting is a mechanism that allows you to make 1 request per 5 seconds, which is normal.

Thanks

Quote from Dygear :

Thanks going to be interesting to program for without some kind of persistent save mechanism, or I guess I could just die on this statement from LFSWorld.

ideally they should be notified that they stink of lazy and should insert a sleep (5) or whatever in their loop.

Quote from Victor :

ideally they should be notified that they stink of lazy and should insert a sleep (5) or whatever in their loop.

LOL, yes. However as the SDK has that sleep it's very unlikely that I'll ever see that error message, never the less, I'll handle it. I might just quote you in the error message .

yes, honestly it rarely happens (anymore), so it's not a big deal. And if you're already sleeping then it indeed will never happen at all for those who use your kit.

When can we expect the new principle checking keys instead of IPs to be put into service? Would it replace the previous one or will it stay with t at the end of the PHP file name as an alternative?

yes I've already copied it in place. The one with the t doesn't exist anymore.