Yes, I know there is some major problems with the packet system in PRISM. I think it's discarding packet data (because, I don't use the Size packet header at all). So I'm going to have to rewrite the whole packet system. It seems to be that when two packets come in close to each other (we are talking about within 1 or 2 milliseconds) they become part of the same buffered variable space. This is causing a problem with when you start a race with multiple AI as the screen always outputs each AI's preference, wing level, fuel load, stuff like that. And this data floods the packet buffer, looks like we also fill up our own buffer space when this happens, this causes packet fragmentation.
If anyone can independently confirm this, that would be very helpful. I'm going to use InSimSniffer myself, but I've been up all night programming and debugging so now is the winter of our discontent, eh ... I mean time for me to go to sleep. .
The socket implementation needs work, the packet processing needs work, but neither should require a complete rewrite
I'll do my best to help, just going through a bureaucracy-intense matter IRL that's keeping me busy.
€: The fragmentation you describe should not happen, even if our first recv() after the select()'s return doesn't poll all the data for the initial packet from the buffer, it will not poll data from a completely different packet inbetween calls. Assuming TCP of course, with UDP it's a simple matter of recv(), checking Size and discarding mismatches
I don't know PHP, but it looks to me like you are just reading the first packet from the buffer on each receive call, so if multiple packets arrive in a single call only the first is getting processed. You need to read out all the completed packets from the buffer in turn, then store any incomplete packets so they can be completed on the next call. Basically you're treating TCP like UDP, instead of as a constant stream of data.
Just want to echo what DarkTimes has brought up and lend some additional thoughts for the packet handling; With TCP there's the possibility for split packets under TCP, however applying the logic generally shouldn't be a problem if you want to stick with the current architecture
If you need help with tcp stream handling, I'd be happy to lend a hand.
If you want some pointers then maybe pm me, or if you want me to write the code, I could do that too :P
Basically it should be :
read tcp data -> add data to stream buffer -> check if there is a whole packet in the buffer -> if yes, substract that packet data from the buffer (leaving a possible next packet or partial packet in the buffer) -> return the packet data you took from the buffer to packet processing functions.
Yeah, not a rewrite, but I need to add some things for sure to make sure there is no packet fragmentation, so I was thinking of adding PHPInSimMod::$packetBuffer and place any data I get from the packet in there. I would then change the PHPInSimMod:acketHandle() function read over the data, splinting off packets by using the Size attribute that's first up in the packet size (Thanks to a stroke of wisdom from Scawen & Vic). If there is less rawData in the packet buffer then the Size, wait until the next packet comes in.
Should I not be ready for the packet buffer to contain only a single byte? Would that not cause fragmentation of the packet, or am I missing something about how the TCP &| UDP protocol works? I might have to read up some more on the socket_select function.
That sir, would be the nail! (As in hit the nail on the head.)
Treating TCP like UDP. That reminds me, TCP connection for normal packets and UDP for MCI & NLP packets. I am going to need two packet buffers, PHPInSimMod::$packetBufferTCP & PHPInSimMod::$packetBufferUDP. Or should I not even bother with a UDP packet buffer? UDP should only return whole packets correct? There fore a buffer is not needed. I can't use one universal buffer encase the TCP has only part of one packet, then I get a new packet from UDP and concatenate that onto the end, then I have part of a TCP packet with a whole UDP packet in the middle of it and that's data corruption. (Not good.)
I agree that the socket system, as in the module that morpha wrote, works extremely well. It's just the code I put around it to read the packets missed some conditions that I had not thought of. So I would expect that to be fixed in the next release.
TAA, could I ask a favor of you? Could you submit 0.1.5 & 0.1.6 to github? I'd really like someone to be on top of version control and I really can't think of a better man then you, as it requires someone who is truth worthy, and a valued member of the community. I've upgrade your account with access there. I'll continue to submit updates here, in the manner that I am doing right now, but i'll include a link to github and let people who want to use it, use it.
Oh my god, that would be awesome, it would be an honor Vic!
Just a note to everyone: I will never say no to help. If people want to take up a section of code and submit new code for it more power to them! I'll review over it, put all of the patches I've received (and the ones I've made myself) and put them all together it into the next release. In fact anyone who is on the dev team can make a release, but due to the current way we submit patches it's best to have one point man.
However Vic, once your done and are happy with the results I don't have a problem with you calling it 0.1.7 and uploading it here. I'll apply my patches on top of that when I'm done and we should have nice rolling update process like that.
If you work on the packet handling system, Vic, I'll start work on the plugin system. I have enough of the packet system ready that I can still test in it's current state. When your done I'll be able to test all of the packets, and I'll merge my code with yours and make 0.1.8.
That's pretty much the idea that I had in mind from the ideas I got from up top of this post from TAA, DarkTimes and morpha.
Read TCP Data (From socket_select call, PHPInSimMod::$socketRead, PHPInSimMod::socket::recv())
Add Data to Stream Buffer (Add data to PHPInSimMod::$packetBuffer)
Check for whole packets in the buffer (If Packet::Size > PacketBuffer cut packet from buffer, and reduce PacketBuffer size by Packet::Size then return packet data to processing functions.)
Check for whole packets in the buffer (If no this time (not a whole packet), or packetBuffer is empty wait for next packet to come in, back to TCP Data read step.)
I was not working on anything, I was sleeping. I only woke up about 30 minutes ago. I might be in the USA, but I'm a night person 22:00 (10:00pm) is my morning, and I go to sleep around 15:00 (3:00pm) - 17:00(5:00pm) depending on when my girlfriend let's me go to sleep.
hmm, I've been reading through the code and came across what I think are some snags :
1)
What about if i wanted to make a tracking tool for a league that runs with multiple hosts?
Atm there is only a facility to have one socket (PHPInSimMod::$socket).
Similar if you'd want to use both a tcp and udp connection (for one host). This is currently not possible.
2)
Related to this I've been thinking if it might be a good idea to split the config file into two. One for generic settings and one for connection settings. This because the connection settings, if the multiple hosts idea would be implemented, might grow and get uncomfortable when combined with generic options. Maybe the connection settings could even be left to the plugin designer's own config file.
So the PHPInSimMod class wouldn't automatically start a connetion, but it would start reading plugins instead and let the plugins initiate all the connections that it needs.
This would require some changes to the structure of PHPInSimMod, so take some time to think about it (like I must as well )
3)
How will you handle console input?
A note - I think it's (STILL) not possible to have console input on windows. Correct me if I'm wrong - i'd be interested
I ask this because in my experience with php I found it very useful to use streams as opposed to socket functions. The reason for that is that you can mix STDIN and any network sockets using stream_select() rather than socket_select(). You wouldn't need two select checks then (one for network traffic and one for kb input).
eg.
$streamRead = array(STDIN, $somehoststream);
$status = stream_select($streamRead, .. ,... etc);
That way you can listen for both network traffic and kb input at the same time, sticking with the timeout mechanism you put in place.
When not using streams the script would not respond to kb input if it is in a socket_select with a long timeout. I hope you see what I mean
Or you could go the ncurses way, but I guess you don't wanna even think about that for multiple reasons
I had that planned from the start, just never had the time to provide any code. My only concern is that a single-threaded PHP based app might not deliver the required performance for multiple hosts, especially with high MCI/NLP frequencies.
It is possible, but we'd have to provide php binaries compiled with the necessary packets in cygwin. Alternatively we could introduce a second process for console I/O that connects to the PRISM instance via socket, thus integrating itself into the select() set. Problem there: stream_select() will return STDIN even if it is still blocking, i.e. if the window loses focus or if you start typing.
hmm ok. I guess that could be worked on later on. The STDIN in the select() call could be optional with a check or variable that tells us we're in windows or not.
I am replaying, and I've read everything. I have some ideas, just give me a second to post my thoughts.
(They will be edited in place here.)
Nope, we are going to spend some more time on the socket / stream functions. And that's awesome! I think it would be a good idea to get the core nailed down before we start moving huge plugins onto the trunk.
Ok, then we would also have a case where one plugin wants to function on just one server. So let's go big picture first. Multiple in and multiple concurrent InSim connections. We could make a new file called connections.ini in the configs dir that would allow for multiple concurrent connections. Something along the lines of this:
[PRISM Dev Server 1] ; Host 1 - PRISM Dev Server #1
; The IP Address we are going to attempt a connection too. ip = "1.2.4.8" ; The Port we are going to attempt the connection on. port = 29999 udpPort = 30000
; Default Password to Attempt; password = "password"
; Socket Connection Type ; 0 - Use Best ; 1 - Use TCP ; 2 - Use UDP socketType = 2
[PRISM Dev Server 2] ; Host 2 (Via Relay) - PRISM Dev Server #2
; 1 = TRUE, Use it, 0 = FALSE, Don't use it. useRelay = 1 ; The Hostname we should connect to. hostname = "PRISM Dev Server 2"
Then from startup we read the file, parse over it and start the connections we wish. Plugins may wish not only operate on one connection, so we will need a way to do that. That's a bridge we will cross when we get there. I also agree that not having concurrent TCP & UDP is a great failure on my part when I started design, as it's a great feature of the InSim engine.
I have no problem with plugins starting their own connections, as long as that connection is handled within PRISM's stream_select or socket_select call. Big ideas are good, I just want to make sure this can also focus on the single server, running multiple plugins without to much of a hassle. Perhaps the plugins.ini file should also describe what plugins will work on what connections. That should allow for the greatest flexibility.
I see us handling this in two ways. Bundling cygwin, like morpha says, on the windows side (that's not a problem at all, I got quite good at the NSIS thing back in 2004 - 2006). But I also think that it may be possible to do keyboard input now with PHP on windows natively. I did not get to read all of the links I had posted in the interactiveStartup() method. I think one of them details how to handle keyboard input async to the socket calls.
I agree stream_select is the way forward, and also I don't use time outs at all in my socket_select functions unless I can help it. The time outs are set to NULL unless a timer is set in that case the timeout is set to when the timeout is set to expire (and code must be executed.)
ncurses, yeah I really hope that's over kill. But there is a built in lib for it. So I'll hold it in reserve as kind of a console function on steroids.
Yeah, I was blown away when you told me that! I wonder just how many packets PHP can handle at once. I think the relay would be a good source of data for this. Vic, do you know how many packets the relay handles in a second both in and out, talking in bytes and packets a second? Also having an idea of internet connection and computer resources would give us all a better gauge on just how far we can push PHP when it comes to this.
I just wanted to expand on this just a little bit, now that I've had a second to take a breath and think about it.
connections.ini Defines the connections that are available.
plugins.ini Defines the plugins that are loaded on startup, and what connections they can utilize.
Yes any plugin can be started on any server, or made to act on any server, so long as the plugin is within the plugins directory, by use the the `prism load plugin` command by an admin. By using the load plugin command, your telling prism to check the plugin exists within the plugin directory and that it's loadable. You are also allowing it to read the packets that are sent from the server. If a plugin is already loaded, think of it as a plugin opting in, at run time to a new connection, and it becomes available for parsing of packets on that connection.
Plugins can be made to connect to all available connections within the plugins.ini file, by naming the connection they wish to, but by using the "*" parameter under their name. This states that the plugin should be connected on all available connections. You should also use the minus sign before a connection name to tell it to connect to all but that connection. An example of that can be found at the end.
Ah sorry dude, hadn't noticed your request, mostly because I've only just looked at the thread again since yesterday(? Starting to loose track of days).
I'll certainly try and assist in keeping GH upto date if you require it
I'm not sure how many packets it can handle, but it's more than prism will need.
For example if there are 250 clients on the relay all listening to a host with 24 racers, then that would mean just for MCI packets :
3pps * (24/8) * 250bytes = 2250 bytes per second per client
2250 * 250clients = 562500 bytes per second, sort of.
and ~ 2250 packets per second that the relay is sending out.
The relay would run at around 10% cpu on a 3ghz xeon at that network load. The relay would be doing more than described here though - it receives insim packets from all the other hosts as well for example and it's doing some other processing like forbidden area checking and simple user tracking.
In other words, easy peasy.
The PRISM app is not supposed to be run on a page request through a webserver but rather on as cli app and a frontend might just read date generated by the background app?
Basically, except the big "feature" of PRISM will be the plugin system. It's more along the lines of Lapper for that. It's written in PHP though, and is to use a cli installed PHP.
You did not miss it, I had posted twice and I don't think you had been on between the first time I posted it and the time I quoted it.
Could not agree with that statement more. At least I hope no one uses it for such a large project. Although if anyone does get there I'd love to know about it!
PHP originally meant Personal Home Page and was used for keeping track of hits on Rasmus Lerdorf's (Creator of PHP) résumé on his personal website. It got it's current recursive acronym PHP: Hypertext Preprocessor. Since then it has grown greatly into a fully featured programming language.
.
. Also we have 