I presume that the Per Request and Per MB add up to form the total?
So if I make 10000 requests (@£0.00001 per req), and in those 10000 requests I transfer 261Mb (@£0.0015 per Mb)

£0.10 (10000 requests)
+
£0.3915 (261Mb)
=
£0.50 (rounded up)

(Approximation of data transfer for 10000 hostslist requests)

Correct?

You have a space at the start of $value3 for all but the first entries (presumably either \r, \n or both)

Solution?

Notice the 'trim($value3)'.

In addition to that though you have a blank entry in the table at the end of each player, but I'll let you solve that one

That is one thing I don't get... I can see it is well intended, not to have hosts IP addresses distributed 'open-air' so to speak, but if I, or anyone else want to know the IP of every single host, or just a single host, all it takes is either "netstat -a", or a more refined method would be ethereal while the real LFS client requests a list (in the case of getting all IPs), or whilethe real LFS client is connecting / is connected to a server (in the case of a single host).

So anyone that could pose any real threat, and indeed about a million folk who don't could easily get the IP address anyway.

@MonkOnAHotTinRoof
Indeed, aggressive multi-threading would work providing that you have a server where you can host applications in a language that can multithread. PHP does not have that ability, although I have been looking in to ways of speeding it up by a kind of psuedo multi-thread app.
It basically chunks the data up and makes smaller (say 30 hosts) async HTTP requests to another script that gets and returns host data, but that is another topic completely.

And thank you, although I cannot take full credit. Victor kindly supplied me with his base script for requesting demo hosts (way back in S1), and I just translated it from PHP to a spec, corrected a few bits, filled in some blanks & updated it for S2.

So credit really goes to Victor.

Actually it is way slower than getting data from lfsworld. Main reason being you have to establish HOST_COUNT+1 sockets and send ((HOST_COUNT/16) + (HOST_COUNT * 2) + 1) packets (Given 300 hosts, that is 620 packets)

It in no way competes against the lfsworld data sources, and can only be used for a hostlist.

And yeah; cons do outweigh the pros, although you did miss a pro, which is why I did this in the first place. By requesting a list say once every 10 minutes, you get an extra bit of data lfsworld does not provide. IP and port combos. By joining a list from lfsworld with a list from the master list server (by servername), you can associate the IPs (not perfect, but it'll work). This will then allow for listing clients outside of LFS that will be able to ping the servers.

I've had this a while; I even have a client in PHP I was going to do something with, but given my current path it seems unlikely.

It is still not 100%, but it certainly isn't far off for the packets I've seen.

I have a working (well it worked at .5p7) PHP cient, although it takes roughly 100 odd seconds to get a full list. You can also get an S1 / S2 hosts list with it by grabbing the password from an ethereal packet dump of the master list (regular) request packet and supplying it to the class. I'm just going to update it with comments and such, and make sure it still works , then you can have it (unless the devs dont want you to ).

For now though, here is my reversal spec. Any questions, fire away.

Appears to work ok. An invalid username / pass doesn't give any feedback though.

Sounds good to me

Quote from the_angry_angel :

I'm making a test *desktop* application with the barely working version of libLFSW. Do we really want a desktop user to have to work out their IP? Doubly so they're going to get pissed if they need to work it out everytime they connect to the intarweb (i.e. dynamic IPs). I'm fairly sure this might be a good idea for websites and such, but not other uses...

A special case "development key"? One that isn't tied to the IP. If I'm paying for this key, I want some assurance that if some gimp pinches it they aren't going to get far with it.

At least it's going in (vaguely) the right direction. Standards compliance and alpha enabled PNG support will allow for some of the more advanced design ideas to be brought in to the mainstream, and the plugin manager should at least aid in curbing those self installing toolbars and spyware / adware junk.

Not to mention you can use it to get around WGA by disabling the WGA plugin. Why you'd need to do that though I couldn't begin to imagine. *snig-ger* (dumb word filter)

When it comes out of beta, if Microsoft pushes this as a recommended download through auto update, at least it is good news for website implementors. It won't have an immediate impact, but if it does happen it signals the end of the IE6 workarounds in favour of the inevitable, mostly undiscovered, and hopefully less numerable IE7 workarounds.

That isn't far off the google maps 'magic key' idea. The only difference being is that the google maps API keys are generated from an URL. Any requests made from any other URL using that key just don't work.

Since you can't really rely on the referrer field using http requests (google have the advantage of their system being JS based, and can check the current location directly), may I suggest that multiple keys can be assigned to an account, and each key is tied to the server IP address. That still isn't 100% secure as people on vhost packages sharing a server would be able to use each others key if they got a hold of it, but it certainly does narrow the field down a bit.

Of course, you'd also want to generate it on username and/or password, and maybe have a bit of random junk thrown in there just to give anyone trying to create keys themselves (if they got hold of username / password & IP) a hard time.

I'm pretty sure most developers should be clued up enough to be able to figure out their servers IP address.

P.S Multiple keys might be required, because for instance my development server and hosting server are separate machines, but I'd like to be able to use both at the same time.

*edit* Unrelated note; with JS disabled all of my new lines get stripped, but enabled it's ok?!? Whats that about? */edit*

Last edited by Anarchi-H, Mon 6 Feb 2006, 22:58.

Ah, good idea! Most sites related to LFS that might want to use this are community resources, and as such are usually privately funded, but as long as it is reasonable I think it makes sense to charge for this as an additional / premium service. However, given that the login data passed on the url now has extra value, I think I'd be way more comfortable in either someway of seeing stats for the data requests made with my login data (so I can see requests made using a pinched login), or having a token (like google maps dev keys) that is tied to my server. The later would be preffered. The login data could be accidentally exposed a million and one ways, some of which are preventable, others less so.

<stupid><question>Why?</question></stupid> I can see it might be useful for tracking who is using how much of what, but beyond that I don't see a great deal of point. Unless that is going to become the basis of a new tarpit / fair usage system?

Quote from Dygear :

I like the idea of a file, but I also hate it. I like the idea, as you can use this across all platforms and there are no 'special' requirements like there are with databases. It is also relatively simple for the end user to understand. But you littler you file system with unwanted files, and you require write access.

A specific cache directory resolves this. If you actually think about it, the key lies in the key generation for the cache. (pun not intended )
If you generate the key based on data sent to LFSWorld then requests for the same data overwrite the old data when they are updated, thus keeping files to a minimal.

Quote from Dygear :

I would perfer a database setup, end users must have the database type that you specify. But its easy to maintain.

No easier than files, and way more complicated than you might expect. This is the current sticking point of PPF2. If you just want a really simple SQL cache (one where you can't fetch out of it by individual fields) then you can just use the SQL container for PEAR::Cache. However, if you want to be able to select fields individually, how do you go about that? Do you split the data in to tables thus requiring multiple inserts per update but sticking to normalization rules, or do you just slap it all in one table, but flying in the face of common database design practice given that it is highly dynamic data?
In addition, you need SQL schemas and queries specific to each datasource if you go the 'individual field' route. The payoff to this method is the full leverage of the power of the RDBMS you implement it on to filter and sort data.

Quote from Dygear :

With memory as soon the file exits, you have lost your information but can be used by eveyone.

Shared memory does not work on windows, hence it's a non starter for me.
*edit* Actually, it does work, but not widely deployed, so it's still a non starter*edit*

Quote from Dygear :

What do you guys use? How do you implament it? Do you mind giving code examples?

A full cache implementation sample would be a pretty big chunk of PHP. Especially for PPF2 because it uses a generic interface allowing for swapping between cache implementations. (i.e. file, sql, any others you care to code)
PPF2 also utilizes a fallback mechanism. The whole process goes something like...

• Generate cache key based on requested data
  
• Check cache for valid entry
  
• If one exists, return data else continue
  
• Request data from LFSWorld
  
• Decompress & check for errors
  
• If data has error, check cache for an old entry (if enabled)
  
• If old entry does not exist, return false
  
• Otherwise return data

Because of this, PPF also implements a request meta-data system that will allow you to see stuff like last update time of cache, if the data is fresh, if the data is fallback data, the query url, error log (if any), and some other useful data.

As you can imagine, this is a not-so-simple affair, so it's all hidden from the user of the class with a nice friendly interface if they don't want to deal with it. However, if they do, almost everything can be overriden either in settings for the interface itself, overriding a specific component or by writing your own interface.

In addition to all of that, everything is pretty independant. If you want to add a datasource, it just requires coding an api compatible class, instantiating it and passing it in as the provider class. You could use the same class with your own cache & request mechanism if you wanted to, just by calling the parsing function.

This post is getting pretty long now, so I'll wrap up my thread hi-jack

A list of components in PPF2

Providers (parsers for all datasources (ch, highlight, hl, hllog, hosts, pb, pst, teams & wr))
Format lookup tools (cars, tracks, rules, flags)
Main programmer interface (LFSWorldSimpleInterface & LFSWorldInterface)
Caches (atm there are file & sql caches, but others are easily added)
String processing (colour code processing, codepage translation & Join2LFS compatible link creation)
LFSWorld error checking function
LFSWorld decompression function

In addition to those, there will be usage docs & api docs. Possibly in a wiki format since phpDocumentor is turning out to be... well a bit naff as far as extended docs go.

There may be more stuff like a PB/PST request chain mechanism to make batch grabs of PB / PST data easier.

PPF was in development for 2 months, and PPF2 (extending & refining PPF1 architecture) has taken an additional month.

If you still want to have a bash though, don't let me stop you
I hope this insanely long post has provided some insight in to what may lie ahead

Last edited by Anarchi-H, Tue 31 Jan 2006, 22:35.

http://www.creativyst.com/Doc/ ... es/CSV/CSV01.htm#Overview

I'm with TAA on this one. Streaming LFS via video is a pretty nasty way of doing things. I'm not so sure it would be worth the effort & resources required, especially given that scawen mentioned something about research in to a potential (and I emphasise that because it was nothing concrete) dummy client exactly for the purpose of spectators when this was raised on RSC.

LFS has no programming interface for retrieving real time telemetry / engine data. It can be accessed in a RAF, and could also probably be gleaned if you can find it in memory, but otherwise you are SOL unless something has changed since I last checked.

Quote from thisnameistaken :

If I had a penny for the number of brothers who've listened to lame music on other peoples' computers, I'd have £4.57.

Sod it, I've got Wham - Young Guns on my drive somewhere. I put it there, and sometimes I listen to it! Willingly!

lol, by lame, I meant shit like bonkers & hardcore nation... wham? You're sick

Velcro cable ties.
Brightly coloured, cheap, effective & easily removable / relocatable.

I get mine from Boyes, but I'm pretty sure you could get them from numerous cheapo jack-of-all-trade stores.

They are quite long too, so they have a tail of velcro (the soft side)... so what I've done is put strips of velcro on the back of my desk. The power cables go along the floor, and the more sensitive cables (read network & audio) run above hanging off the velcro on the desk.

I've also colour coded mine so that audio cables are in green ties etc.

I believe I joined a way back; I'm ixulai on there for those that don't know.
You'll have to excuse the lame entries of late though; my brother has been using my comp a lot recently.

Quote from Victor :

<u>18:52 08-01-2006</u> <b>vishnou7022</b> had uploaded a new hotlap to 126th place: <b>1:25.030</b> on <b>BL gp</b> with the <b>XRT</b></a>


edit - maybe you meant the laptime was missing in format 1, in the case of a WR. I can't see anything wrong otherwise.

Doh! My bad. In my defence I was running purely on caffiene yesterday :doh:
Firstly, the position error isn't an error at all, but when log_filter=1 the position always becomes '1' because obviously a WR is always number 1

Second thing (format 1 missing data) seems to have been fixed. Yesterday I was getting stuff like;


Today it seems ok though.
I'll keep checking to see if it was a particular param combo that did it, but IIRC it was doing it all of the time so it's probably fixed.

Good stuff

*plug*

I'll be releasing PPF version 2 sometime shortly, as soon as I get PHPDoc to cooperate and finish up the usage docs.
In short, it is easier to install, easier to use, more robust, has more features & is more powerful.
Heres a sneak peek at the minimum code required to get a full hosts list.

However, the old API is there. It is just in a slightly different form. It will be backwards compatible through a wrapper.

In addition, an optional 'fallback' has been coded in so that if LFSWorld is unreachable, or givs you errors it will try to use old data instead of simply failing, and there are providers for all data sources, including highlights, which has a parsing option.

I don't think I could have made it more straight forward. In addtition, installation will be easier because I'll be distributing the required PEAR libs with it now. You can override that though.

There are lots of other things to mention, but I don't want to hijack the thread so I'll leave it there

Last edited by Anarchi-H, Mon 9 Jan 2006, 06:12.

Small issue with the XML output when the list returned should be empty. The other two formats simply supply 'no output', but the XML one supplies only '</hotlaplog>' which is invalid xml and may throw some parsers, or at least cause bitching about mismatched tags or lack of xml declaration.

Example;
http://www.lfsworld.net/pubsta ... p;control=ks&format=2

In addition, at least one new error message has been introduced. If 'version' is less than 1.2 when requesting 'hl_log', the script returns:


*edit*

hl_log appears to be pretty bustified
First, it says 'had uploaded a new WR'. Shouldn't that be 'has' and 'HL'?
Second, the position does not have a value for format 1, and always appears as '1' in format 2.
And lastly, format 1 is missing data, namely track and position.

Unless I missed something and that is how it is supposed to be :s

*edit2*
Just confirmed, format 3 has the same position problem as format 2.

Last edited by Anarchi-H, Sun 8 Jan 2006, 17:22.

Quote from Tooby :

Thanx Victor for giving us the hl_log.
I almost don't dare to ask but...Could some one explain the date stamp (
<entry uploaddate="1136678609">).

Thanx in advance
/Tooby

Edit: Is there any good (free) php editors where you can have auto complete on commands and easy to find help?

It's a unix timestamp. For information on how to translate it in to an easier read format, see http://www.php.net/manual/en/function.date.php

As for auto completing editors, I can't help you. I gave up on 'em long ago, and now instead use gVIM and the reference manual.

*edit*Skip to the 'The point' bit if you can't be arsed to read all of my babble. There is a point. I promise *edit*

Excellent stuff Vic, although the 'format' parameter seems to deviate from your (up until now) 'I'll give you the data, you do what you like with it' mantra.

I only have one 'gripe', if you like with the LFSWorld data now.
PBs and PSTs.
Anyone who has written a script to get these on a restricted server (especially if you want both of them because they share a tarpit) will tell you that the current method is a pain in the arse.

Consider a page with a team listing.
Each driver has basic information, car number, name, plate, plus a couple of stats and their best PB.
Lets assume that this team has 15 members.

Now, how to get the data?
The easy way around would be alter the max execution time & make the requests in a loop, sleeping for 5 seconds at the end of each iteration.
(15 * 5) * 2 = 150 seconds.
15 drivers, 5 second delays, once for PBs, again for PSTs.

But not everyone has permission (iirc safe_mode restricts this) to alter the max execution time. Some hosts even limit it to lower than default times.

So how to get it if you can't alter the execution time? A cyclic, or round robin cache mechanism, or alternatively a caching request chain.

The first involves picking a name from the top of the driver list, updating and putting them at the bottom, except each driver needs to appear twice. Once for PB, once for PST.

The second is a bit of a hack, but it gets the job done as quick as a for loop but with only a 5 second execution time required.
Basically, a root request is made that takes the same list a cyclic cache would use as an url parameter. The script takes the first one off the list, requests the data, sleeps for 5 seconds and then recurses by making a HTTP request for itself with the new list. This carries on until the list is empty, at which point the job is done.

IMO that is a very long way around, and incredibly wasteful.

The point

I think a separated list of usernames for PB and PST with separate tarpits of at least a minute would make a lot more sense, especially considering that the majority of users for the lfsworld data seem to be teams.
I don't know the request numbers, and I don't know the database schema, but taking our example above, 2 queries has to be better than 30, especially if there are any joins involved.

Of course thats just my opinion and could be total twaffle, but it makes sense to me.

Otherwise, keep up the good work

Not shoutcast, but I qutie often listen to pandora.com

ATM I'm listening to 'What the hell is stipulation - Reggie and the full effect' radio.

Basically, it takes an artist / track and gives you similar music. Kinda like the last.fm similar artist feature only better.

Quote from Dygear :

I did not know that PHP had struts, if you could show me somewhere I could learn about them, that would be awsome. I am taking it that a strut is a datatype, and I can't really find anything on them.

I think you are confusing "struts" with "structs"; Struts is an MVC framework originally written in Java (but now with clones / ports to other languages), where as a struct is as you assumed a data type.

PHP doesn't have structs, but then you don't need a struct in PHP. An assoc array or a custom class (if it has a few functions tied to it) would be perfectly suitable.