Hi folks!

I've come up with this problem when programming my insim application, the famous TCP - WOULDBLOCK.

After doing some research I've been reading some interesting old threads that discuss the topic. I'm no expert, so first of all I'd like to know what is exactly happening and what would be the best ways to tackle this problem. I'll explain first what is happening in detail.

My insim app is server oriented. It's a league managing insim. Players can request tables to see the TOP PBs and pre-qualifying times. Each table is at most 65 buttons, with not a lot of text.

Players can click on other players names in the tables to see a detail of their times, and each time they do that the table is cleared and completely re-sent.

So I'm testing with my teammates clicking on names like hell to force the insim to send a lot of buttons to them.

The server is located on my home computer, but it hangs on pretty well until at some point the server sends a TCP - WOULDBLOCK messsage and that player loses connection to the server.


As I understand it, what's happening is the following:

• A player requests a table. The insim sends all the buttons to the server, and then the server sends the same buttons to the client.
  
• If too much info is being sent to the client then the TCP buffer for that specific client's connection gets full and that sometimes causes the client to timeout or lose some important TCP packets that he should be receiving. This causes the client to lose connection.
  

Now I'm wondering some things... First of all, my friends have reported that they don't lose connection unless they are using any p2p or direct download software that is taking most of their bandwidth. Then again, my insim or the server never seem to die, they keep alive pretty nicely

Is it possible that the server might end up diying also because the insim is sending too many buttons to it? Which is the *best* way to prevent clients from getting their TCP buffer full?

What I want the most is help and ideas about how to avoid having people losing connection to the server. Any help would be appreciated on techniques to avoid this.

As I understand it, (I'm sure you've read more about the problem than I have) TCP_WOULDBLOCK happens when there are too many packets 'happening' in a short period of time.

Are you still sending buttons with a fixed size? If so, each packet will take longer to send thus make it more likely for a packet collision/flood to happen. Reducing the size of the packet to the smallest size possible (by cropping off the unused portions of the text field) may reduce the likelihood of this happening.

It's not a proper solution, but it may help.

Edit: I've just read the 0.5 update to your library - the above is irrelevant with that version, feel free to ignore.

Yes, this is what has finally brought me to send the buttons trimmed to avoid the overhead. A full table with max text width could mean about 252 bytes/button x 60 buttons = 15120 bytes! (14.7 KB for a single table!). Now a full table is at max 28 bytes/button x 60 buttons = 1680 bytes (1.64 KB).

That is sent in a single burst by the insim to the host, and then from the host to the client. I don't know how big the client connection TCP buffer is, but is sending 1.5 KB that much?

I would think it would be much larger then the MTU for any given connection. Seeing as that in the range of 1,500 Bytes on Ethernet. I would think the buffer size should be many times that.

WOULDBLOCK just means the connection is busy at the moment and is saying if I were to wait for it, it would block/freeze your application to do so. So it sends that message and the proper response is to stop sending/receiving and wait for another send/recv event from winsock to notify you to continue..

This shouldn't cause a timeout or a disconnection as its not a connection problem but one of bandwidth. But since this one is implemented as an "error" most people treat it like that and close the connection entirely.

When you send/recv a packet, just put the whole thing into a buffer and track the position, number of bytes. The send/recv functions tell you how much was transferred so just handle that in a temporary place and resume when the next event fires.

If the clients are loosing connection from the server, wouldn't that mean that the problem is between the LFS server and the LFS client? Surely if it was an error with the InSim application, the InSim connection would be terminated, not a client's?

In which case, it would be an LFS error (as the previous WOULDBLOCK issue was that Scawen fixed a few years back IIRC) not an error in the InSim application itself. It may be triggered by a lot of packets being sent over InSim in a short period, and reducing the number/size of InSim packets would reduce the likelyhood of it occurring, but wouldn't solve the problem - nothing in an InSim application would for sure.


We also have this problem sometimes on our InSim enabled server. Usually it happens to the same few people, indicating a client-side bandwidth problem, but has occasionally disconnected a good proportion of the server's clients at the same time.


I suppose the way to find out if it's an InSim<->Server or Server<->Client problem would be to see if your InSim application's winsock actually receives a WOULDBLOCK error/message. If it doesn't, other than over MSO, then I would assume it's an LFS Server<->Client issue.

I believe it's a server<->client issue. My insim socket is a blocking one (which is by default), so I believe that if I did everything right my socket would have no problem handling this situation, as it can be blocked and wait in send and recv calls. I think WOULDBLOCK must be manually handled only when you use a nonblocking socket, in which case you are in charge of checking and controlling when you can send or receive data.

I have to say that I see that message in the LFS dedicated server console, I don't know if it also shows in the client's chat when they lose connection. The first time I saw the wouldblock message was when I was still using max sized buttons. When a client requested a table I would see first the wouldblock message and if he kept requesting more tables then I saw repeating green messages "TCP: Cleared emergency store". But actually during this test the client never lost connection to the server, and I was the only one seeing those messages in the server console.

After this I changed my insim library to trim button packets and reduce dramatically the network usage for these button-tables. With this change I no longer see "TCP: Cleared emergency store", but if the client goes crazy requesting buttons a wouldblock message may *or may not* appear and it comes with a client disconnection.

My concern right now is that I'm testing this right now with only two people online, and one of them is me, and I have all LFS client, LFS host and the insim application running on the same machine -though my connection is pretty poor in upload bandwidth!- I'm worried that disconnetions may occur more often in a crowded server.

Anyway, my insim and my dedicated server seem to survive pretty fine. I think I'll wait to do some testing in a *real* server hosted by some serious hosting company and have a bunch of people online requesting buttons and we'll see what happens!

Quote from Stuff :

WOULDBLOCK just means the connection is busy at the moment and is saying if I were to wait for it, it would block/freeze your application to do so. So it sends that message and the proper response is to stop sending/receiving and wait for another send/recv event from winsock to notify you to continue..

This shouldn't cause a timeout or a disconnection as its not a connection problem but one of bandwidth. But since this one is implemented as an "error" most people treat it like that and close the connection entirely.

When you send/recv a packet, just put the whole thing into a buffer and track the position, number of bytes. The send/recv functions tell you how much was transferred so just handle that in a temporary place and resume when the next event fires.

Mod to Plus 5, Insightful!

Quote from MaKaKaZo :

Anyway, my insim and my dedicated server seem to survive pretty fine. I think I'll wait to do some testing in a *real* server hosted by some serious hosting company and have a bunch of people online requesting buttons and we'll see what happens!

Give me a date and time and I'll show up to help you test out.

Quote from Dygear :

Give me a date and time and I'll show up to help you test out.

When I'm done with the user interface I'll post a public request for beta testers somewhere visible (probably in the programmers forum and somewhere else). But first I have to re-do some things, like help/rules/commands windows and add a couple of commands to show info about the next race and PB analysis compared to the best lap.