Server Backup Delay Over Gigabit Ethernet Articles | Network Analysis Tips - Analyzers - Packet Sniffer

What was the root cause in this instance?

The TCP Window on the backup server was filling, requiring the source server to halt while the buffer was cleared by the application. This occurred thousands and thousands of times, causing enormous delay in the backup process.

We will now discuss why a problem with TCP Window can have this kind of effect, how we found this problem with Wireshark, and what we can do to address this issue and improve performance.

TCP Window – What is it, and why did it cause the delay?

A TCP Window can be described as a receive buffer on a given station. It is the amount of data that can be sent to that station at one time, without receiving an ACK for the data sent. With every ACK, a receiver will inform the sender how much space it has remaining in its TCP Window.


In many cases, we don’t see this number drop, showing that we have 65,535 bytes of room in the buffer.

If this window does fill, the receiver will send a window size of Zero to the sender, indicating a full window, which should cause the sender to halt transmission until further notice. One reason the TCP window can fill is that the application is not clearing this buffer as quickly as traffic is coming into it. The overall halt caused by this event can add seconds, minutes, or even hours to a backup.

How can this problem be found with Wireshark?

Often, hours of extra delay is caused by several thousand smaller delays. The first key to finding these delays in Wireshark is to use the time column to look for any halt in the data stream. In the screenshot below, the time column was set to view the time between the current packet and the previously displayed packet. This is also known as delta time. Note packet 144.

When compared to the time between the packets above it, which are sub-millisecond, this packet experiences a relatively long delay. It is a 64 byte ACK from the backup server to the source server. What caused this packet to be delayed for 20 milliseconds?

Look at the Window value of the previous packet in this direction – packet 142. This packet has a window of 2299 Bytes, meaning it only has 2299 bytes left in the receive buffer before it is full. When the sender received this ACK, it saw that there was only room for one more full size packet. This final packet is sent in frame 145, then the sender waits for the receive buffer to clear. In packet 144, the window has returned to 65,535 bytes and transfer can resume. After this frame, the delta times return to microseconds.

Note: In this situation, we never actually saw the TCP Window completely fill, resulting in a Zero Window event. As this example shows, zero windows are not the only thing to look for when troubleshooting window problems.

This 20 millisecond delay occurred once for every 10 milliseconds of transfer. In some cases, it took even longer for the TCP Window to return to 65,535. These delays caused the overall backup time to slow to over 5 hours.

What can be done to resolve this?

The backup application in this example is single-threaded. Only one TCP connection is used to transfer data from one machine to another, and the application on the receiver is not processing data out of the receive buffer as fast as it is coming in. In some cases, backup apps can be tuned to use more TCP connections, however this is not always the case. It may be that a new backup application is needed in order to use multi-threading. Also, check with the vendor to see if there are any tuning possibilities in the application itself that will cause it to clear the TCP buffer more frequently.

There are many things that can impact server backups. So far we have discussed the impact of file handling and delays caused by full TCP Windows.