FIN_WAIT_2 status at Client side

[Ahmad Jalil Qarshi]

Hi,

I have developed a socket based (Connection Less) client application
on AIX 5.2 for a third party server.

The protocol is somewhat like:

1) Connect with server at port 5555
2) Send Request to Server at specified port.
3) Close connection with server. (This is must. Client must close the
port after sending request otherwise the server will not send us
response).
4) Open a listening port at Client (Port: 9999) to receive the
response from Server.
5) Receive Response and process at Client side. (Server is responsible
to Close the connection at port 9999).

Now the problem is that when I close connection with server (using the
close() function) after sending request to server at port 5555,
somehow the port 5555 is not properly closed. The close function
returns 0, which indicates successful close.

When I use netstat on my client machine I can see following port
status.

Proto Recv-Q Send-Q Local Address Foreign
Address (state)
-------- ---------- -----------
--------------------- --------------------------
------------------
tcp4 0 0 server1.59959
200.10.196.3.5555 FIN_WAIT_2

According to TCP/IP protocol it indicates that the Close function is
being called at Client side but server has not properly closed the
port. The third party server vendor is claiming that I am not properly
closing the port at client side so this problem is at client side.

A short description of Client/Server TCP/IP protocol is given at:
http://www.softlab.ntua.gr/facilities/documentation/unix/unix-socket-faq/unix-socket-faq-2.html#ss2.7

Now what I want to know is where the actual problem, at client side or
server side and what is the solution.

Thanks in anticipation,

Ahmad Jalil Qarshi

[Kenny McCormack]

Post by Ahmad Jalil Qarshi
Hi,
I have developed a socket based (Connection Less) client application
on AIX 5.2 for a third party server.

(Posting from clc - note the "CLC effect" in full bloom...)

With love and affection:

Off topic. Not portable. Cant discuss it here. Blah, blah, blah.

[David Schwartz]

Post by Ahmad Jalil Qarshi
Now the problem is that when I close connection with server (using the
close() function) after sending request to server at port 5555,
somehow the port 5555 is not properly closed. The close function
returns 0, which indicates successful close.

And that's a problem because ... ?

Post by Ahmad Jalil Qarshi
According to TCP/IP protocol it indicates that the Close function is
being called at Client side but server has not properly closed the
port. The third party server vendor is claiming that I am not properly
closing the port at client side so this problem is at client side.

And the problem is ... ?

Post by Ahmad Jalil Qarshi
Now what I want to know is where the actual problem, at client side or
server side and what is the solution.

What's the problem? You've shown your attempts to diagnose the problem
but not what the actual problem is. What goes wrong?

DS

[Rick Jones]

First, a netnanny nit:

When you cross-post to so many groups, you should select one for the
Followup-To: header. I've trimmed comp.lang.c from the followup-to:
header on this, please further trim it down to one group of your
chosing. My initial suggestion would be comp.protocols.tcp-ip.

Post by Ahmad Jalil Qarshi
1) Connect with server at port 5555
2) Send Request to Server at specified port.
3) Close connection with server. (This is must. Client must close the
port after sending request otherwise the server will not send us
response).
4) Open a listening port at Client (Port: 9999) to receive the
response from Server.
5) Receive Response and process at Client side. (Server is responsible
to Close the connection at port 9999).

Why can't the server simply send the response back down the connection
to port 5555? It seems really silly to go to all the trouble of
establishing a second TCP connection when you already have one which
is perfectly capable. And by requiring connection establishment by
both ends, it makes life much more complicated when dealing with
firewalls.

Post by Ahmad Jalil Qarshi
Now the problem is that when I close connection with server (using the
close() function) after sending request to server at port 5555,
somehow the port 5555 is not properly closed. The close function
returns 0, which indicates successful close.
When I use netstat on my client machine I can see following port
status.
Proto Recv-Q Send-Q Local Address Foreign
Address (state)
-------- ---------- -----------
--------------------- --------------------------
------------------
tcp4 0 0 server1.59959
200.10.196.3.5555 FIN_WAIT_2
According to TCP/IP protocol it indicates that the Close function is
being called at Client side but server has not properly closed the
port. The third party server vendor is claiming that I am not properly
closing the port at client side so this problem is at client side.

The third party server vendor is confused or there is a device between
your client and the server doing evil things. You have done
everything you should on your end.

FIN_WAIT_2 does indeed mean that endpoint has sent a FINished segment,
and that segment has been ACKnowldeged by the remote. It is now
sitting there waiting for a FINished segment from the remote.

Perhaps the third party server code is doing something bad like using
an abortive close on the connection. That will send a ReSeT segment
rather than a FINished segment, and RST's are not retransmitted when
lost.

What you need is a packet trace at both ends.

rick jones

[Barry Margolin]

In article

Post by Ahmad Jalil Qarshi
Hi,
I have developed a socket based (Connection Less) client application
on AIX 5.2 for a third party server.
1) Connect with server at port 5555
2) Send Request to Server at specified port.
3) Close connection with server. (This is must. Client must close the
port after sending request otherwise the server will not send us
response).
4) Open a listening port at Client (Port: 9999) to receive the
response from Server.

That's a very strange protocol design. Why doesn't it just send the
response on the same connection? If it needs to get an EOF to indicate
the end of the request, that's what half-closed connections are for; the
client would use shutdown(s, SHUT_WR) to send a FIN without closing the
connection.

Shouldn't you open the listening port before closing the connection with
the server? Otherwise, what happens if the server is faster than the
client, so it tries to connect to your port 9999 before you open it?

Post by Ahmad Jalil Qarshi
5) Receive Response and process at Client side. (Server is responsible
to Close the connection at port 9999).
Now the problem is that when I close connection with server (using the
close() function) after sending request to server at port 5555,
somehow the port 5555 is not properly closed. The close function
returns 0, which indicates successful close.
When I use netstat on my client machine I can see following port
status.
Proto Recv-Q Send-Q Local Address Foreign
Address (state)
-------- ---------- -----------
--------------------- --------------------------
------------------
tcp4 0 0 server1.59959
200.10.196.3.5555 FIN_WAIT_2
According to TCP/IP protocol it indicates that the Close function is
being called at Client side but server has not properly closed the
port. The third party server vendor is claiming that I am not properly
closing the port at client side so this problem is at client side.
http://www.softlab.ntua.gr/facilities/documentation/unix/unix-socket-faq/unix-
socket-faq-2.html#ss2.7
Now what I want to know is where the actual problem, at client side or
server side and what is the solution.

Sounds to me like the problem is on the server. FIN_WAIT_2 means you've
sent a FIN, received an ACK of the FIN, and you're waiting for the other
side to send its FIN. That happens when the server closes its end of
the connection.

[Ahmad Jalil Qarshi]

Post by Barry Margolin
In article

Post by Ahmad Jalil Qarshi
Hi,
I have developed a socket based (Connection Less) client application
on AIX 5.2 for a third party server.
1) Connect with server at port 5555
2) Send Request to Server at specified port.
3) Close connection with server. (This is must. Client must close the
port after sending request otherwise the server will not send us
response).
4) Open a listening port at Client (Port: 9999) to receive the
response from Server.

That's a very strange protocol design. Why doesn't it just send the
response on the same connection? If it needs to get an EOF to indicate
the end of the request, that's what half-closed connections are for; the
client would use shutdown(s, SHUT_WR) to send a FIN without closing the
connection.
Shouldn't you open the listening port before closing the connection with
the server? Otherwise, what happens if the server is faster than the
client, so it tries to connect to your port 9999 before you open it?

Post by Ahmad Jalil Qarshi
5) Receive Response and process at Client side. (Server is responsible
to Close the connection at port 9999).
Now the problem is that when I close connection with server (using the
close() function) after sending request to server at port 5555,
somehow the port 5555 is not properly closed. The close function
returns 0, which indicates successful close.
When I use netstat on my client machine I can see following port
status.
Proto Recv-Q Send-Q Local Address Foreign
Address (state)
-------- ---------- -----------
--------------------- --------------------------
------------------
tcp4 0 0 server1.59959
200.10.196.3.5555 FIN_WAIT_2
According to TCP/IP protocol it indicates that the Close function is
being called at Client side but server has not properly closed the
port. The third party server vendor is claiming that I am not properly
closing the port at client side so this problem is at client side.
http://www.softlab.ntua.gr/facilities/documentation/unix/unix-socket-...
socket-faq-2.html#ss2.7
Now what I want to know is where the actual problem, at client side or
server side and what is the solution.

Sounds to me like the problem is on the server. FIN_WAIT_2 means you've
sent a FIN, received an ACK of the FIN, and you're waiting for the other
side to send its FIN. That happens when the server closes its end of
the connection.
--
Arlington, MA
*** PLEASE don't copy me on replies, I'll read them in the group ***

Thanks all (especially Rick Jones and Barry Margolin) for your kind
illustrative response.

Dear Barry, as I told you that its a third party server so I have to
follow this protocol. Our response listener module is always running
so it doesn't matter how fast the server is. As far as problem is
concerned I am totally agree with you that the problem is at server
side.

Regards,

Ahmad Jalil Qarshi

[Niel Lambrechts]

Post by Ahmad Jalil Qarshi

Post by Barry Margolin
Sounds to me like the problem is on the server. FIN_WAIT_2 means you've
sent a FIN, received an ACK of the FIN, and you're waiting for the other
side to send its FIN. That happens when the server closes its end of
the connection.

Not likely on the server side.

The server (when closing a connection), sends a FIN. The client has to
respond with ACK and then sends its own FIN which then requires a last
ACK from the server side.

The state that the connection is in during the period between when the
server gets the ACK from the client and the server gets the FIN from the
client is known as FIN_WAIT_2.

So really, the server could (legitimately) be waiting for the final FIN
from the client. It commonly happens where the client has a buggy O/S,
TCP stack or application bug - a couple of years ago it still commonly
happened in web browsers.

If the server-side is running on AIX, you could try something like:
no -o tcp_keepintvl=30

so that the O/S will clean connections in this state faster (2 minutes),
but this is a workaround and the time may still be too long.

<original post>

Post by Ahmad Jalil Qarshi
Now the problem is that when I close connection with server (using the
close() function) after sending request to server at port 5555,
somehow the port 5555 is not properly closed. The close function
returns 0, which indicates successful close.

If you do not care for any buffered TCP data that may remain on the
socket at the point where you are currently calling close() in your
client code, you should call shutdown(socketName,SHUT_RDWR) before the
close().

refer to
http://publib.boulder.ibm.com/infocenter/systems/index.jsp?topic=/com.ibm.aix.progcomm/doc/progcomc/skt_shutdn.htm

Niel

[Niel Lambrechts]

Post by Barry Margolin
Sounds to me like the problem is on the server. FIN_WAIT_2 means you've
sent a FIN, received an ACK of the FIN, and you're waiting for the other
side to send its FIN. That happens when the server closes its end of
the connection.

Apologies Barry I read the post in haste - the "server" you are
referring to actually is the client side of the TCP connection in this
case.

I agree the problem must lie on the "third party server".

Niel

[Rick Jones]

Post by Niel Lambrechts
no -o tcp_keepintvl=30

If there is still a TCP endpoint at the other end, will TCP keepalives
really do anything? The keepalive probe will be sent, and the remote
endpoint will send an ACK in response.

rick jones

[Niel Lambrechts]

Post by Rick Jones

Post by Niel Lambrechts
no -o tcp_keepintvl=30

If there is still a TCP endpoint at the other end, will TCP keepalives
really do anything? The keepalive probe will be sent, and the remote
endpoint will send an ACK in response.
rick jones

In the FIN_WAIT_2 state the client has already ACK'd the
server/initiator FIN.

The socket will be cleaned up in (tcp_keepintvl * tcp_keep_alive_retry)
half seconds server side.

Niel

[Rick Jones]

Post by Niel Lambrechts

Post by Rick Jones

Post by Niel Lambrechts
no -o tcp_keepintvl=30

If there is still a TCP endpoint at the other end, will TCP
keepalives really do anything? The keepalive probe will be sent,
and the remote endpoint will send an ACK in response.

In the FIN_WAIT_2 state the client has already ACK'd the
server/initiator FIN.

Fine.

Post by Niel Lambrechts
The socket will be cleaned up in (tcp_keepintvl *
tcp_keep_alive_retry) half seconds server side.

How? Only if the other endpoint is no longer there. However, if it
is still there, in CLOSE_WAIT, all that will (should) happen IIRC is
that the keepalive probe will be ACKnowleldged and so the side sending
the keepalives will remain. As far as it can tell, it is still a
valid simplex TCP connection right?

TCP keepalives only cull connections when the remote end is no longer
responding.

Post by Niel Lambrechts
Niel

[Niel Lambrechts]

Post by Rick Jones

Post by Niel Lambrechts
The socket will be cleaned up in (tcp_keepintvl *
tcp_keep_alive_retry) half seconds server side.

How? Only if the other endpoint is no longer there.

Maybe tcp_keepintvl is not ideal for this but most operating systems do
not strictly follow RFC793 any longer, and provide a timeout to
specifically cater for FIN_WAIT_2 states.

Post by Rick Jones
However, if it
is still there, in CLOSE_WAIT, all that will (should) happen IIRC is
that the keepalive probe will be ACKnowleldged and so the side sending
the keepalives will remain. As far as it can tell, it is still a
valid simplex TCP connection right?

The AIX specific option for the timer I am thinking about is actually
'tcp_finwait2', and seems to have been around since AIX 4.3:

# no -h tcp_finwait2
Help for tunable tcp_finwait2:
Specifies the length of time to wait in the FIN_WAIT2 state before
closing the connection, measured in half seconds. The default is 1,200
half seconds (600 seconds or 10 minutes). Ranges from 0 to USHRT_MAX.
tcp_finwait2 is a runtime (dynamic) attribute.

This should close the TCP connection unconditionally, as long as the
state is FIN_WAIT_2.

I'm wondering - wouldn't shutdown(s,SHUT_RDWR) before close() prevent
the keepalives you are referring to in the first place?

Niel

[Henry]

Post by Ahmad Jalil Qarshi
Hi,
I have developed a socket based (Connection Less) client application
on AIX 5.2 for a third party server.
1) Connect with server at port 5555
2) Send Request to Server at specified port.
3) Close connection with server. (This is must. Client must close the
port after sending request otherwise the server will not send us
response).
4) Open a listening port at Client (Port: 9999) to receive the
response from Server.
5) Receive Response and process at Client side. (Server is responsible
to Close the connection at port 9999).
Now the problem is that when I close connection with server (using the
close() function) after sending request to server at port 5555,
somehow the port 5555 is not properly closed. The close function
returns 0, which indicates successful close.
When I use netstat on my client machine I can see following port
status.
Proto Recv-Q Send-Q Local Address Foreign
Address (state)
-------- ---------- -----------
--------------------- --------------------------
------------------
tcp4 0 0 server1.59959
200.10.196.3.5555 FIN_WAIT_2
According to TCP/IP protocol it indicates that the Close function is
being called at Client side but server has not properly closed the
port. The third party server vendor is claiming that I am not properly
closing the port at client side so this problem is at client side.
A short description of Client/Server TCP/IP protocol is given at:http://www.softlab.ntua.gr/facilities/documentation/unix/unix-socket-...
Now what I want to know is where the actual problem, at client side or
server side and what is the solution.
Thanks in anticipation,
Ahmad Jalil Qarshi

Only time I've had issues with FIN_WAIT_2 was with a printer being
timed-out by a PIX Firewall.
Stupid Firewalls. Waste of money