If I send the following Packet

sTiny.Size = 4
sTiny.Type = E_InSim.ISP_TINY
sTiny.ReqI = 1
sTiny.SubT = E_TINY.TINY_NPL

LFS send back only the first two drivers on Server.
Even if I stop all other procedures in the Tool.

In Last InSim i had to send NPL for each driver. I think now i only have to send this one packet, wright ??

Any hint ???

Here are two things which happened to me, which caused that apparently not all NPL packets were received:

1. : Insecure receiving. For example the C-function recv(...) returns whatever amount of bytes it decides to, which means, that you may receive only two and a half packages at first. You need to catch this, because otherwise every packet after the second one can not be interpreted correctly. Nevertheless, it weirdly worked flawlessy in single player, but once LFS went multi player: BOOM!
2. (stupid one): The PLID can range from 0 to 255 not just to 31.

Quote from Blue-Ray :

2. (stupid one): The PLID can range from 0 to 255 not just to 31.

Not really, PLIDs start from 1.

Quote from hackerx :

Not really, PLIDs start from 1.

Well, the PLID 0 is not assigned to a player, but the PLID sent in a packet can be zero under certain circumstances.
Of course I didn't take that into account when I made this mistake.

Solved
I just had to cut the received stream into pieces

Yes, read into a buffer and if there are at least 4 bytes there, check if there are actually "Size" bytes there and if so, you have a packet, so process it, then shunt down the buffer by that amount. If you don't yet have a full packet, read some more bytes - perhaps next frame - and check again for a packet (added to the end of your buffer).

Not only can you receive multiple packets in one call to "recv" or whatever function you call to fill your buffer with incoming TCP data, you could also receive half a packet. TCP itself has no understanding of packets, and can transfer however many bytes in wants to at any time, so it's up to you to make sure the right amount of data is there before you process it.

confusing

Maybe I need to draw diagrams. But really, you can work it out yourself and that's probably better. It's just a case of realising that the packets don't necessarily arrive in complete or single packets. Any number of the sent bytes may arrive at any time. The good thing is (with TCP) they are guaranteed to arrive, and arrive in the correct order, unless the connection breaks down.

The multiple packets case is very easy to deal with - just see if there is more data there after reading the first packet.

The incomplete packets case is a bit more complicated but it's really just a case of your code realising that there isn't enough data received for the specified packet size, so you must leave that incomplete packet in the buffer until you get some more info and find that the packet is complete.

If you do these two steps, you'll never miss a packet, because TCP will take care that nothing is ever dropped from the stream. I hope that's a better explanation!

On the other hand, when TCP does break down. My what a mess! I put a timeout on my receive function that if after a period of about 40 seconds (whatever LFS' keep alive packet is + a little bit) then it breaks the connection off and tries again. Well worth it, because when TCP goes wrong, every packet afterwards sails up the kaiber with the carry on crew.

Hmm, it really shouldn't go wrong. Not AT ALL!

If it goes wrong, due to connection loss or whatever, no more packets will be sent. It's either working fine or it's broken, that is the great thing about TCP. You should absolutely never receive any wrong info. You should just get an error response to the operating system's receive function.

If you are receiving bogus packets that basically means your packet interpretation function is wrong.

Quote from Scawen :

The multiple packets case is very easy to deal with - just see if there is more data there after reading the first packet.

The incomplete packets case is a bit more complicated but it's really just a case of your code realising that there isn't enough data received for the specified packet size, so you must leave that incomplete packet in the buffer until you get some more info and find that the packet is complete.

I have been using the new insim over TCP/IP for quite some time now, including connecting to a local guest, a local host and a remote host over the internet and have yet to experience the "not receiving enough data" and have never yet had to buffer the received data and wait for some more to finish the LFS Packet.

Now, I am using VB6 with its standard WinSock control and it fires a routine called _DataArrival when data is received from the TCP socket. I do not know if this function somehow compensates for the fact that TCP can break up the initial "packet".

Getting a few LFS packets in altogether happens all the time and I have been able to accomodate this (as I had it happening and could test and debug it until it worked). However, with the split packet, I have never had it so I have not worked how what to do with it. I am of course looking for it and I just have a message box which will pop up the first time it happens!

Don't get me wrong, I fully understand the reasons that it can and will happen, I just wonder if VB6 compensates already.

Can anyone else confirm of clarify this?

Tim

Quote from t1ger :

Don't get me wrong, I fully understand the reasons that it can and will happen, I just wonder if VB6 compensates already.

It shouldn't do - VB has no understanding of how the insim data is packetized (all TCP data is basically a constant stream, its upto you how you packetize it), so you're probably just very lucky, or you've got a huge buffer..

Connect your InSim client upto LFS and join a really busy server with a fair bit of chat, and start watching the packets, odds are you'll probably see a split one.

Quote from the_angry_angel :

It shouldn't do - VB has no understanding of how the insim data is packetized (all TCP data is basically a constant stream, its upto you how you packetize it), so you're probably just very lucky, or you've got a huge buffer..

Connect your InSim client upto LFS and join a really busy server with a fair bit of chat, and start watching the packets, odds are you'll probably see a split one.

Thanks, that is a good point, I haven't yet connected to a "busy" server and tried.

Tim

For stress testing I use FernBay Club with the maximum ammount of AI driver with BF1 and enable receiving MCI packets with 100ms interval.

It's not the load of a 32 player server, but for initial testing it's ok.

The easiest way to deal with splitted packages is very simple. Just read the first byte of the incoming stream, but without removing the byte (!!). And then test if the ammount of bytes in the input buffer is greater or equal the byte you read. If yes read in the packet, if not wait for more bytes.

If you do this in a loop you have no problem to read multiple packets and splitted packets.

Thanks LupusC for that clear explanation. And of course that is the reason I put "Size" as the first byte.

I think that people get problems because they like to create a buffer just for THIS frame only, and not store any extra info. But that's not possible, the buffer you read your bytes into must be a static buffer that is still there in the next iteration of the loop. You need a buffer and a "WritePos" or "BytesInBuffer" variable, indicating where the next bytes to arrive will be written to.

For clarification a short (simplified) snipplet of my java program that shows the part where the packets are read in.

receiveBuffer: used for the incoming data stream
inSimData: ByteBuffer with the read packet you can convert into a structure you need (I do it with the factory pattern)

My program is in an alpha state so i cannot make it yet public.

Quote from LupusC :

The easiest way to deal with splitted packages is very simple. Just read the first byte of the incoming stream, but without removing the byte (!!). And then test if the ammount of bytes in the input buffer is greater or equal the byte you read. If yes read in the packet, if not wait for more bytes.

If you do this in a loop you have no problem to read multiple packets and splitted packets.

Hi LupusC,

Thanks, and yes that is exactly what I do. I have the routine that receives data from the TCP stream which passes the array of bytes received to a routine which checks the first byte ("size") and then checks it against the size of bytes received on the stream.

As you know, there are three possiblities:

1. The size received matches the size of the LFS packet. I then pass the data to the process routine. The process routine only knows how to deal with full and complete LFS packets.

2. The size received is greater than the size of the LFS packet. In this case, I pass the first LFS packet to the process routine and then re-process the rest of the actual data received.

3. The size received is less than the size of the LFS packet. In this case I know that I need to wait for more data and then add it to the end, and then process it. I have as yet, never encountered this case. All I have here is a message box waiting to pop up and say "Not enough data received".

Last night, I connected to a very full server, which had about 23 people connected and people joining and leaving and talking and pitting and racing and everything. Now, apart from my PC having some issues and my application not being able to keep up with where everybody was (but I know this is a limitation of what I have coded) then I never saw case 3 as described above.

I then connected to another server with about 7 racers and I got the same results again.

Like I say, I do not deny that this is not possible, I am just saying that I would have expected to have seen it by now with the amount of testing I have done.

Thanks

Tim

Quote from t1ger :

All I have here is a message box waiting to pop up and say "Not enough data received".

Hmm, don't make a messagebox. If there is a splitted package then just wait for the rest of the package. As Scawen said, TCP is streaming, splitted packages are normal and no exception. The messagebox has no useful information for the user. If you want you can make an entry in a log file. Personally I use Java and log4j and for such information the log level DEBUG.

Thinking about how to get splitted packages maybe the following wil be a solution:
1. No MCI packets (they are too large and not so good to handle for that)
2. Set receive buffer size to 100-120 Bytes (not kByte!!!)
3. Maximum AI player or a replay from a server with more player
4. Fire out TINY_NPL requests in a loop to get many many many packets

For every request you get at least 20 times 76 bytes in return (with maxium AI)

The IS_NPL packet is 76 bytes long, so your buffer will be smaller than 2 packets and IMHO a splitted package has to occur.

You are using VB6 so have a look at http://support.microsoft.com/kb/237688/en-us to change socket options.

Quote from LupusC :

Hmm, don't make a messagebox.

I know, don't worry I am just testing at the moment. And even when this is complete I do not intend releasing it to the masses - only a select few

Quote from LupusC :

Thinking about how to get splitted packages maybe the following wil be a solution:
1. No MCI packets (they are too large and not so good to handle for that)
2. Set receive buffer size to 100-120 Bytes (not kByte!!!)
3. Maximum AI player or a replay from a server with more player
4. Fire out TINY_NPL requests in a loop to get many many many packets

For every request you get at least 20 times 76 bytes in return (with maxium AI)

The IS_NPL packet is 76 bytes long, so your buffer will be smaller than 2 packets and IMHO a splitted package has to occur.

This is the type of thing I need! I am not, by any means, an expert programmer. I know what I need to do when I get less data than I need - wait for some more and then add on the bits I am missing. BUT I can guarantee that it will not work if I code it and just wait for it to happen. I need to test it and make sure, and I can only do that when it happens.

Quote from LupusC :

You are using VB6 so have a look at http://support.microsoft.com/kb/237688/en-us to change socket options.

I will try this, because the one thing I am so far unable to do is find out and change how big the receive buffer is. If I can do this, then I can test it and then I will know I am covered when it occurs.

Thanks very much for the info LupusC.

Tim

Ehm, one thing I forgot to mention. If you cannot fetch all incoming packets fast enough then you might loose the connection to LFS because LFS thinks that there is a problem with the InSim client. I didn't test this but I think it could be possible. Have to test this also for my app.

Quote from t1ger :

Like I say, I do not deny that this is not possible, I am just saying that I would have expected to have seen it by now with the amount of testing I have done.

FYI, I have now seen this 3 times in total. However, most of the time it does not happen. I can now confirm that VB6 is not "hiding" me from it and it is possible, just nowhere near often enough for me to successfully write the code to handle it.



Tim

Gah - my brain hurts.

Trying to make an app in VB6, and I cannot seem to get the "buffer" to work correctly.

Anybody got a snippet?