A couple of days ago I spent several hours thanks to these two little design quirks.

Quirk #1. Deadlock prone protocol design. Our program uses two client-server protocols. Let’s call them encrypted and plaintext to protected the innocent. On an unencrypted connection things are business as usual. Client connects to the server, sends a request, and waits for the reply. On an encrypted connection, things are more interesting. Immediately upon client connection, the server is supposed to send a greeting. Client that uses encrypted connection waits for this greeting and will not proceed any further until the greeting has been received.

This causes a deadlock when the client thinks the connection is encrypted, and the server things it is not. The server waits for an unencrypted client request, while the client waits for an encrypted server greeting. Nothing happens. In our team this is a well-known problem that every developer experienced at least once.

It is important to note, that due to various technical issues, debug servers are always unencrypted.

Quirk #2. Visual Studio and the command line. By default client works encrypted. There is a command line switch like -unencrypted that changes that. Command line parser is written by hand and does something like GetCommandLine().Split(' '). Normally, you cannot add “ENTER” characters (\r\n) to the command line in Visual Studio debugger. However, if you copy and paste the command line from another source like Notepad, enters are happily inserted. Thus you end up with a command line looking like

-port foo -host bar -unencrypted\r\n

The command line parser looks at this, finds unknown switch “unencrypted\r\n”, ignores it and moves on, treating connection as encrypted. And encrypted client would wait for the server greeting, which will never come…

Quirk #3. In our system it is possible to intercept messages being sent out. However, higher level interceptor reports messages as “sent” before they actually hit the wire. While in reality the client is waiting for the server greeting (that will never come, remember?), the log says a request has been sent to the server.

End result. So, you see in the client log “sending request xyzbght” and you observe that the server is not responding. You check for the -unencrypted flag in the command line, and it’s obviously there. Characters \r\n are also there, but after all, they are invisible and not really expected to be in a command line. So, the client sent the request, and the server did not respond. This means the problem is somewhere on the server side, right?

You debug the server to death, and come to the conclusion that the server never receives client request. So, client says it sent it, but the server never received it. At this point you turn to some monitoring agent that looks what actually was transmitted over the wire. I use PocketSoap TcpTrace. Then you find out that nothing was actually sent and the client log is basically lying.

Then you turn to debug the client, find out about the -unencrypted\r\n flag and it all falls into place. After 3 or 4 hours of intense server debugging of course.

Happy end.