In this blog we will compare the network performance of instances in OCI and AWS.

The Network is a sensitive resource in multi-tenant environments, and is usually one of the first contributors leading to diminished performance when the environment is oversubscribed.

In AWS there is a concept of Placement groups. Network performance is optimized between instances belonging to the same Placement group. But the downside of having instances in one and the same placement group is that it limits the amount of resources available for each instance. Therefore if a big cluster has to be created, there is a good chance that resource limitations for specific instance types will be reached.

In Oracle Cloud infrastructure this problem does not exist. For OCI’s flat network the concept of Placement groups is not really applicable because the whole AD is basically one large placement group in itself.

To cover as much as possible the following tests have been performed:

  • OCI: network test between 2 instances, Same AD
  • AWS: network test between 2 instances in same Placement group, same AZ
  • AWS: network test between 2 instances in different Placement group, same AZ
  • OCI: network test between 2 instances, different AD
  • AWS: network test between 2 instances, different AZ

We measured network latency and throughput, with all tests running 10GBit instances.

Instances in same AD/AZ belong to same subnet.

We use uperf as a benchmarking tool. It is one of the most common tools available on the market for network performance testing.

I. Pick instance shapes

Used cheapest AWS 10Gbit instance: c4.8xlarge (60.0 GiB Memory, 36 vCPUs):

24*30*1.59$/hr = $1144.8 /month.

On OCI the cheapest 10Gbit instance was used: BM.Standard1.36 (256GB Memory, 36 OCPU):

24*30*36*0.0638$/(OCPU*hour) = $1653.69/month

There is no exact match on shapes between the 2 vendors. But as far as network testing chosen instances can be considered equivalent: Memory would not make a difference. Both shapes have enough CPU power to load 10Gbit.

Baremetal vs VM does not make a big difference either as 10Gbit AWS shapes has SR-IOV support.

SR-IOV enables network traffic to bypass the software switch layer of virtualization stack.

As a result, the I/O overhead in the software emulation layer is diminished and achieves network performance that is nearly the same performance as in non-virtualized environments.

II. Install benchmarking tools

RHEL 7.4 is OS on AWS, and OEL 7.4 was picked on OCI.

Install uperf. It has to be installed on target and destination instances.

[ec2-user@ip-10-10-5-145 ~]$ sudo yum install git
[ec2-user@ip-10-10-5-145 ~]$ sudo yum install gcc
[ec2-user@ip-10-10-5-145 ~]$ sudo yum install autoconf
#run below step only if rpm is missing. Find rpm in public repo
[ec2-user@ip-10-10-5-145 ~]$ sudo yum install /tmp/lksctp-tools-devel-1.0.17-2.el7.x86_64.rpm   
[ec2-user@ip-10-10-5-145 ~]$ git clone https://github.com/uperf/uperf
[ec2-user@ip-10-10-5-145 ~]$ cd uperf
[ec2-user@ip-10-10-5-145 ~]$ touch aclocal.m4 configure Makefile.am Makefile.in
[ec2-user@ip-10-10-5-145 uperf]$ mkdir /tmp/uperf1
[ec2-user@ip-10-10-5-145 uperf]$ export PATH=/tmp/uperf1:$PATH
[ec2-user@ip-10-10-5-145 uperf]$ ./configure --prefix=/tmp/uperf1
[ec2-user@ip-10-10-5-145 uperf]$ make
[ec2-user@ip-10-10-5-145 uperf]$ sudo make install

Same test for instance in OCI.

III. Test

1. OCI: network test between 2 instances, Same AD

First is ping test, to get a feel of latency for decent size payload.

[opc@net-test-ad1-1 uperf]$ ping -c 50 -s 1400 -i 0.2 -q   10.0.0.14

PING 10.0.0.14 (10.0.0.14) 1400(1428) bytes of data.

--- 10.0.0.14 ping statistics ---

50 packets transmitted, 50 received, 0% packet loss, time 9799ms

rtt min/avg/max/mdev = 0.153/0.178/0.225/0.024 ms

Ping test may be not very accurate as ICMP traffic can be de-prioritize in some networks. It gives rough numbers, but will not use it for comparison.
Uperf does not have this problem as it generates TCP traffic.

Uperf test.

Run uperf in slave mode on destination. Also for duration of this test stop firewall if it blocks needed ports.

[opc@net-test-ad1-2 uperf]$ sudo systemctl stop firewalld.service

[opc@net-test-ad1-2 uperf]$ uperf -s

Run uperf latency test on source:

Use all default settings in netperf.xml

[opc@net-test-ad1-1 uperf]$ export h=10.0.0.14

[opc@net-test-ad1-1 uperf]$ export proto=TCP

[opc@net-test-ad1-1 uperf]$ cat workloads/netperf.xml

<?xml version="1.0"?>

<profile name="netperf">

  <group nthreads="1">

        <transaction iterations="1">

            <flowop type="accept" options="remotehost=$h protocol=$proto

      wndsz=50k tcp_nodelay"/>

        </transaction>

        <transaction duration="30s">

            <flowop type="write" options="size=90"/>

            <flowop type="read" options="size=90"/>

        </transaction>

        <transaction iterations="1">

            <flowop type="disconnect" />

        </transaction>

  </group>

</profile>

 

[opc@net-test-ad1-1 uperf]$ uperf -m workloads/netperf.xml -a -e -p

    Starting 1 threads running profile:netperf ...   0.00 seconds

    Txn1            0 /   1.00(s) =            0           1op/s                                                                  
    Txn2      72.09MB /  30.23(s) =    20.00Mb/s       27781op/s                                                                  
    Txn3            0 /   0.00(s) =            0           0op/s                                                                  
    -------------------------------------------------------------------------------------------------
    Total     72.09MB /  32.34(s) =    18.70Mb/s       25975op/s

    Group Details
    -------------------------------------------------------------------------------------------------
    Group0    144.18M /  31.23(s) =    38.72Mb/s       53782op/s

    Strand Details
    -------------------------------------------------------------------------------------------------
    Thr0      72.09MB /  32.24(s) =    18.76Mb/s       26055op/s

    Txn                Count         avg         cpu         max         min
    -------------------------------------------------------------------------------------------------
    Txn0                   1      1.28ms      0.00ns      1.28ms      1.28ms
    Txn1              419963     71.45us      0.00ns      2.81ms     61.32us
    Txn2                   1     36.66us      0.00ns     36.66us     36.66us


    Flowop             Count         avg         cpu         max         min
    -------------------------------------------------------------------------------------------------
    accept                 1      1.28ms      0.00ns      1.28ms      1.28ms
    write             419963      2.34us      0.00ns     92.21us      1.68us
    read              419962     68.99us      0.00ns      2.81ms      6.51us
    disconnect             1     31.75us      0.00ns     31.75us     31.75us


    Netstat statistics for this run
    -------------------------------------------------------------------------------------------------
    Nic       opkts/s     ipkts/s      obits/s      ibits/s
    ens2f0       12990       12990    16.24Mb/s    16.21Mb/s
    lo            192         192   397.42Kb/s   397.42Kb/s
    -------------------------------------------------------------------------------------------------


    Run Statistics
    Hostname            Time       Data   Throughput   Operations      Errors
    -------------------------------------------------------------------------------------------------
    10.0.0.14         32.34s    72.09MB    18.70Mb/s       839928        0.00
    master            32.34s    72.09MB    18.70Mb/s       839928        0.00
    -------------------------------------------------------------------------------------------------
    Difference(%)     -0.00%      0.00%        0.00%        0.00%       0.00%

Run uperf throughput test on source:

For throughput test we bumped wndsz 10 times from default in iperf.xml. This is needed, as with higher latencies, and specific configurations of TCP sliding window, RTT could be a bottleneck. With hight wndsz truly maximum throughput can be checked. These settings were used in all of our tests.

[opc@net-test-ad1-1 uperf]$ export nthr=8

[opc@net-test-ad1-1 uperf]$ uperf -m workloads/iperf.xml -a -e -p

[opc@net-test-ad1-1 uperf]$ cat workloads/iperf.xml

<?xml version="1.0"?>

<profile name="iPERF">

  <group nthreads="$nthr">

        <transaction iterations="1">

            <flowop type="connect" options="remotehost=$h protocol=$proto

      wndsz=500k  tcp_nodelay"/>

        </transaction>

        <transaction duration="30s">

            <flowop type="write" options="count=10 size=8k"/>

        </transaction>

        <transaction iterations="1">

            <flowop type="disconnect" />

        </transaction>

  </group>

 

</profile>

 

[opc@net-test-ad1-1 uperf]$ uperf -m workloads/iperf.xml -a -e -p

    Starting 8 threads running profile:iperf ...   0.00 seconds

    Txn1            0 /   1.00(s) =            0           8op/s                                                                  
    Txn2      34.47GB /  30.23(s) =     9.79Gb/s      149434op/s                                                                   
    Txn3            0 /   0.00(s) =            0           0op/s                                                                  
    --------------------------------------------------------------------------------------------------
    Total     34.47GB /  32.33(s) =     9.16Gb/s      139718op/s

     
    Group Details
    --------------------------------------------------------------------------------------------------
    Group0     3.45GB /  31.23(s) =   947.93Mb/s      144644op/s

     
    Strand Details
    --------------------------------------------------------------------------------------------------
    Thr0       4.31GB /  32.23(s) =     1.15Gb/s       17539op/s
    Thr1       4.30GB /  32.23(s) =     1.15Gb/s       17500op/s
    Thr2       4.31GB /  32.23(s) =     1.15Gb/s       17519op/s
    Thr3       4.32GB /  32.23(s) =     1.15Gb/s       17559op/s
    Thr4       4.30GB /  32.23(s) =     1.15Gb/s       17504op/s
    Thr5       4.31GB /  32.23(s) =     1.15Gb/s       17512op/s
    Thr6       4.30GB /  32.23(s) =     1.15Gb/s       17491op/s
    Thr7       4.31GB /  32.23(s) =     1.15Gb/s       17526op/s

     

     

    Txn                Count         avg         cpu         max         min
    --------------------------------------------------------------------------------------------------
    Txn0                   8    446.91us      0.00ns    587.04us    384.51us
    Txn1              451763    531.73us      0.00ns      2.61ms      8.74us
    Txn2                   8      8.30us      0.00ns      9.99us      6.09us

     

     

    Flowop             Count         avg         cpu         max         min
    --------------------------------------------------------------------------------------------------
    connect                8    446.44us      0.00ns    586.65us    384.02us
    write            4517570     53.16us      0.00ns      2.61ms      8.69us
    disconnect             8      8.00us      0.00ns      9.65us      5.73us


    Netstat statistics for this run
    --------------------------------------------------------------------------------------------------
    Nic       opkts/s     ipkts/s      obits/s      ibits/s
    ens2f0      134770       35059     9.23Gb/s    18.54Mb/s
    lo            192         192   401.81Kb/s   401.81Kb/s
    --------------------------------------------------------------------------------------------------

     
    Run Statistics
    Hostname            Time       Data   Throughput   Operations      Errors
    --------------------------------------------------------------------------------------------------
    10.0.0.14         32.33s    34.46GB     9.16Gb/s      4517171        0.00
    master            32.33s    34.47GB     9.16Gb/s      4517594        0.00
    --------------------------------------------------------------------------------------------------
    Difference(%)     -0.00%      0.01%        0.01%        0.01%       0.00%

As we can see network latency is 71.45us and throughput is 9.79Gb/s. Number are within SLA limits.

2. AWS: Network test between 2 instances in one Placement group, same AZ

Ping test.

[ec2-user@ip-10-10-5-145 uperf]$ ping -c 50 -s 1400 -i 0.2 -q 10.10.6.238

PING 10.10.6.238 (10.10.6.238) 1400(1428) bytes of data.

--- 10.10.6.238 ping statistics ---

50 packets transmitted, 50 received, 0% packet loss, time 9799ms

rtt min/avg/max/mdev = 0.108/0.158/0.214/0.029 ms

Uperf test.

Here only the essential part of output is listed, to keep content short.

Run uperf latency test:

[ec2-user@ip-10-10-5-145 uperf]$ export h=10.10.6.238

[ec2-user@ip-10-10-5-145 uperf]$ export proto=TCP

[ec2-user@ip-10-10-5-145 uperf]$ uperf -m workloads/netperf.xml -a -e -p

    …

    Txn                Count         avg         cpu         max         min
    --------------------------------------------------------------------------------------------------
    Txn0                   1      5.48ms      0.00ns      5.48ms      5.48ms
    Txn1              423172     70.79us      0.00ns      3.43ms     56.18us
    Txn2                   1     58.23us      0.00ns     58.23us     58.23us

Run uperf throughput test:

[ec2-user@ip-10-10-5-145 uperf]$ uperf -m workloads/iperf.xml -a -e -p

    Starting 8 threads running profile:iperf ...   0.00 seconds

    Txn1            0 /   1.00(s) =            0           8op/s                                                                  
    Txn2      34.44GB /  30.23(s) =     9.79Gb/s      149333op/s                                                                  
    Txn3            0 /   0.00(s) =            0           0op/s                                                                  
    --------------------------------------------------------------------------------------------------
    Total     34.44GB /  32.33(s) =     9.15Gb/s      139622op/s

Network latency is 70.79us and throughput is 9.79Gb/s. Numbers are within SLA limits.

This is not bad at all.

Confirm what SR-IOV is enabled:

[ec2-user@ip-10-10-5-145 ~]$ ethtool -i eth0
driver: ixgbevf                               ---<<-------- ixgbevf used
version: 3.2.2-k-rh7.4
firmware-version: 
expansion-rom-version: 
bus-info: 0000:00:03.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

These test results appear to confirm the following theory: When AWS Placement group is enforced, performance of 10Gbit OCI Baremetal shape is basically identical to 10Gbit AWS VM shape.

3. AWS: network test between 2 instances in different Placement group, same AZ

Ping test.

[ec2-user@ip-10-10-5-145 uperf]$ ping -c 50 -s 1400 -i 0.2 -q 10.10.3.181

PING 10.10.3.181 (10.10.3.181) 1400(1428) bytes of data.

--- 10.10.3.181 ping statistics ---

50 packets transmitted, 50 received, 0% packet loss, time 9802ms

rtt min/avg/max/mdev = 0.191/0.336/2.551/0.456 ms

Uperf test.

Run uperf latency test:

[ec2-user@ip-10-10-5-145 uperf]$ export h=10.10.3.181
[ec2-user@ip-10-10-5-145 uperf]$ export proto=TCP
[ec2-user@ip-10-10-5-145 uperf]$ uperf -m workloads/netperf.xml -a -e -p

    …

    Txn                Count         avg         cpu         max         min
    --------------------------------------------------------------------------------------------------
    Txn0                   1    605.02us      0.00ns    605.02us    605.02us
    Txn1              152148    197.06us      0.00ns     10.92ms    146.15us
    Txn2                   1     40.05us      0.00ns     40.05us     40.05us

Run uperf throughput test:

[ec2-user@ip-10-10-5-145 uperf]$ uperf -m workloads/iperf.xml -a -e -p

    Starting 8 threads running profile:iperf ...   0.00 seconds

    Txn1            0 /   1.00(s) =            0           8op/s                                                               
    Txn2      29.10GB /  30.23(s) =     8.27Gb/s     126164op/s                                                                  
    Txn3            0 /   0.00(s) =            0           0op/s                                                                 
    --------------------------------------------------------------------------------------------------
    Total     29.10GB /  32.33(s) =     7.73Gb/s     117959op/s

Key numbers are highlighted in bold. All numbers will be summaraized in the comparison table at the end of this blog.

4. OCI: network test between 2 instances, different AD

Ping test.

[opc@net-test-ad1-1 uperf]$ ping -c 50 -s 1400 -i 0.2 -q  10.0.1.5

PING 10.0.1.5 (10.0.1.5) 1400(1428) bytes of data.

--- 10.0.1.5 ping statistics ---

50 packets transmitted, 50 received, 0% packet loss, time 9800ms

rtt min/avg/max/mdev = 0.706/0.724/0.786/0.018 ms

Uperf test.

Run uperf latency test:

[opc@net-test-ad1-1 uperf]$ export h=10.0.1.5

[opc@net-test-ad1-1 uperf]$ export proto=TCP

[opc@net-test-ad1-1 uperf]$ uperf -m workloads/netperf.xml -a -e -p

    …

    Txn                Count         avg         cpu         max         min
    --------------------------------------------------------------------------------------------------
    Txn0                   1      1.89ms      0.00ns      1.89ms      1.89ms
    Txn1               43090    696.85us      0.00ns      1.42ms    679.91us
    Txn2                   1     16.21us      0.00ns     16.21us     16.21us

Run uperf throughput test:

[opc@net-test-ad1-1 uperf]$ uperf -m workloads/iperf.xml -a -e -p

Starting 8 threads running profile:iperf ...   0.00 seconds

    Txn1            0 /   1.00(s) =            0           8op/s                                      
    Txn2      34.46GB /  30.23(s) =     9.79Gb/s      149424op/s                               
    Txn3            0 /   0.00(s) =            0           0op/s                                                                  
    --------------------------------------------------------------------------------------------------
    Total     34.46GB /  32.33(s) =     9.16Gb/s      139708op/s

5. AWS: network test between 2 instances, different AZ

Ping test.

[ec2-user@ip-10-10-5-145 uperf]$ ping -c 50 -s 1400 -i 0.2 -q 10.10.2.207

PING 10.10.2.207 (10.10.2.207) 1400(1428) bytes of data.

--- 10.10.2.207 ping statistics ---

50 packets transmitted, 50 received, 0% packet loss, time 9801ms

rtt min/avg/max/mdev = 0.638/0.678/0.741/0.038 ms

Uperf test.

Run uperf latency test:

[ec2-user@ip-10-10-5-145 uperf]$ export h=10.10.2.207

[ec2-user@ip-10-10-5-145 uperf]$ export proto=TCP

[ec2-user@ip-10-10-5-145 uperf]$ uperf -m workloads/netperf.xml -a -e -p

    …

    Txn                Count         avg         cpu         max         min
     --------------------------------------------------------------------------------------------------
    Txn0                   1      2.67ms      0.00ns      2.67ms      2.67ms
    Txn1               46138    650.44us      0.00ns    201.36ms    542.42us
    Txn2                   1     22.06us      0.00ns     22.06us     22.06us

Run uperf throughput test:

[ec2-user@ip-10-10-5-145 uperf]$ uperf -m workloads/iperf.xml -a -e -p

Starting 8 threads running profile:iperf ...   0.00 seconds

Txn1            0 /   1.00(s) =            0           8op/s                                                                  
Txn2      17.57GB /  30.23(s) =     4.99Gb/s      76208op/s                                                                  
Txn3            0 /   0.00(s) =            0           0op/s                                                                  
--------------------------------------------------------------------------------------------------
Total     17.57GB /  32.33(s) =     4.67Gb/s       71252op/s

IV. Conclusion

Below is a summary of our obtained results:

Env Test description Latency Throughput
OCI Test between two instances, Same AD 71.45us 9.79GB/s
AWS Test between two instances, in same Placement group, same AZ 70.79us 9.79GB/s
AWS Test between two instances, in different Placement group, same AZ 197.06us 8.27GB/s
OCI Test between two instances, different AD 696.85us 9.79GB/s
AWS Test between two instances, different AZ 650.44us 4.99GB/s

The configuration of AWS with the Placement group applied performed very similar to OCI running two instances in the same AD.

Removing the AWS placement group increased latency significantly and shaved off about 10% of throughput.

Cross AZ/AD traffic added more latency, which was expected, but cross AZ also decreased
throughput in AWS - which was rather un-expected. Potential causes for this could be examined in a future blog.