Tag Archives: IOPS

What does Microsoft mean by low / moderate / high / very high / extremely high Azure network bandwidth (part 2.3)

This article is part of a series of 5 where I am talking about the Microsoft Azure network bandwidth. For a better understanding please make sure you read also the other parts:

  1. What does Microsoft mean by low / moderate / high / very high / extremely high Azure network bandwidth (part 1)
  2. The isolated network setup (the environment used for network analysis) (part 2)
  3. The IOmeter benchmark tests who reproduce as close as possible the HTTP/HTTPS, SMB and MS SQL network traffic (part 3)
  4. The Azure Virtual Machines used to run the IOmeter benchmarks (part 4)
  5. Results and interpretations

Now let’s see what exactly happens when a file is copied over the network (file share)

Because the file share is a bit more complex, I will explain in advance what you will see in the next Wireshark and Process Monitor screenshots.
In a non-academic explanation: when a file is copied over the network, the file is read from the disk in 1 MB chunks. Each 1 MB chunk is then sent through the network. At this point SMB is the communication protocol. SMB divides each 1 MB chunks in smaller chunks (up to 64 KB each) and is requesting the network adapter to send them to the destination.
Let’s pause a bit the explanation and mention Large Segment Offload (LSO) – which “is a technique for increasing outbound throughput of high-bandwidth network connections by reducing CPU overhead. It works by queuing up large buffers and letting the network interface card (NIC) split them into separate packets.”
Continuing the file share explanation: actually those SMB smaller chunks (up to 64 KB each) are sent over the network in much smaller size packets. The size of those packets is in direct relation with the Maximum Transmission Unit size.
Let’s pause again the explanation and mention in the absence of Large Segment Offload (LSO) (TSO / TCP Chimney Offload) the 1 MB chunks will be directly divided to packets in size with the Maximum Transmission Unit.

On the other “end of the wire”, the system where the file is written to, will receive network packets who will not exceed the Maximum Transmission Unit size.
The purpose of this explanation is to help reproduce as close as possible the file share network traffic using IOmeter.

 

Of course, the way the data is handled (processed) before being passed to the network interface card has its performance implications – but this is a different subject. Remember, we look to benchmark only the network. We will try in our analysis to have the minimum influence from CPU, RAM and DISK point of view.

 

Here I will provide couple of important links for those who want to have a deeper reading in SMB, Wireshark, LSO / TSO / TCP Chimney Offload, MTU
https://en.wikipedia.org/wiki/Maximum_transmission_unit
https://en.wikipedia.org/wiki/Protocol_data_unit
https://en.wikipedia.org/wiki/Network_layer
https://ask.wireshark.org/questions/24699/tcp-packet-length-was-much-greater-than-mtu
https://msdn.microsoft.com/en-us/library/cc246482.aspx
https://en.wikipedia.org/wiki/Large_segment_offload
http://www.peerwisdom.org/2013/04/25/disabling-large-send-offload-windows/
https://en.wikipedia.org/wiki/IP_fragmentation

 

This is the Process Monitor capture performed on OPIOA3_01 VM while an EXE file (en_.net_framework_4_full_x86_x64_508940.exe) was copied via network (file share).

In the picture the top-red rectangle indicates the fact the OS is reading the file from disk in 1 MB chunks.
The middle-red rectangle indicates the WriteFile operation that will be analyzed in the next pictures (Wireshark screens).
The bottom-red rectangle indicates the TCP Send & TCP Receive operations performed by the System as result of the WriteFile (via network) commands.

what_does_microsoft_means_by_low_moderate_high_very_high_extremely_high_azure_network_bandwidth_14

 

The WriteFile Event Properties -> this is the Process Monitor event who will be tracked later in the Wireshark screens.

what_does_microsoft_means_by_low_moderate_high_very_high_extremely_high_azure_network_bandwidth_15

 

Here we have the Wireshark capture (performed on OPIOA3_01 VM) filtered to show what exactly does “WriteFile Offset: 2,097,152, Length: 1,048,576” operation means in terms of network traffic. Please note the 1,048,576 bytes (the 1 MB data chunk from the file sent over the network) are actually split in multiple (28) packets (no larger than 64 KB each).

what_does_microsoft_means_by_low_moderate_high_very_high_extremely_high_azure_network_bandwidth_16

 

Here we have the same Wireshark capture (performed on OPIOA3_01 VM). Note the fact Wireshark reassembled all those 28 packets. We can see the Write Length, File Offset and the amount of data match the numbers from Process Monitor (Write Length=1048576, File Offset=2097152, Data Length=1048576 bytes).

what_does_microsoft_means_by_low_moderate_high_very_high_extremely_high_azure_network_bandwidth_17

 

 

BUT, the Wireshark capture performed on OPIOA3_02 VM (the system where the file was written to) indicates that maximum size of the received packets from OPIOA3_01 is not exceeding the default Ethernet MTU (1500 bytes).

Please note the capture is sorted based on the maximum size of the frames.
Frame no. 16933 length: 1514 bytes (ETH = 14 bytes | IP = 20 bytes | TCP = 20 bytes | TCP segment data = 1460 bytes)

what_does_microsoft_means_by_low_moderate_high_very_high_extremely_high_azure_network_bandwidth_18

 

what_does_microsoft_means_by_low_moderate_high_very_high_extremely_high_azure_network_bandwidth_19

 

what_does_microsoft_means_by_low_moderate_high_very_high_extremely_high_azure_network_bandwidth_20

 

what_does_microsoft_means_by_low_moderate_high_very_high_extremely_high_azure_network_bandwidth_21

Frame no. 16933 length: 1514 (ETH = 14 bytes | IP = 20 bytes | TCP = 20 bytes | TCP segment data = 1460 bytes)

MTU (by default 1500 bytes for Ethernet)“maximum size of an IP packet that can be transmitted without fragmentation – including IP headers but excluding headers from lower levels in the protocol stack.”
From the above screenshots: IP (20 bytes) + TCP (20 bytes) + TCP segment data (1460 bytes) = 1500 bytes

 

So, let’s make a short summary of what we’ve seen so far about file share network traffic usage:

  • when a file copy over the network is performed, initially the file is read from the disk in 1 MB chunks.
  • each of those 1 MB chunks are requested to be written to a remote network resource.
  • at the application layer the SMB protocol is used for handling the data transfer between the systems.
  • SMB divides each 1 MB chunks in smaller chunks (up to 64 KB each) and request the network adapter to send them to the destination.
  • the 64 KB chunks are actually sent over the network in much smaller size packets (the size of those packets is influenced by the MTU size).
  • on the other “end of the wire”, the system where the file is written to, will receive network packets who will not exceed the Maximum Transmission Unit size.

Again, this explanation will help reproduce as close as possible the file share network traffic using IOmeter.

 

Go back to the previous page.