In this blog we compared Oracle Cloud Infrastructure baremetal local NVMe SSD with AWS instance NVMe store on MemSQL use case. MemSQL load mostly concerned about concurrent writes, and relatively large reads.

Thins blog will do broader test, covering most performance aspects of SSD.

Will use Cloud Harmony Block storage project for benchmarking. In my opinion it is one of the most comprehensive benchmarking suites for SSD. It replicates the SNIA Solid State Storage (SSS) Performance Test Specification (PTS) Enterprise v1.1.

More details on the test specification is here.

Pick shapes

There is no exact 1-1 shape matches between AWS and OCI but will pick as close as possible, and compensate differences on test level.


Oracle: BM.HighIO1.36 (36 OCPU, 500GB of RAM, 4x3TB NVMe SSD drives)         $4.46/hr. 
AWS: i3.16xlarge (64 vCPU, 488GB of RAM, 8x1.9TB instance NVMe SSD drive)    $4.99/hr.

On Multithreaded systems 36 Oracle OCPU would be equivalent 72 Amazon vCPU, as OCPU=1core, vCPU=1 thread.

OS/kernel

On AWS will do latest RHEL release:

  • Red Hat Enterprise Linux Server release 7.4 (Maipo)
  • hostname: aws-rhel7test

On OCI will use latest Oracle Enterprise Linux with UEK kernel:

  • Oracle-Linux-7.4
  • hostname: oci-oel7test

As we saw in previous blog OEL and CentOS behaved almost same on Oracle Baremetal Cloud - will skip CentOS here for simplicity.

Here are kernel versions:

[ec2-user@aws-rhel7test ~]$ uname -r
3.10.0-693.el7.x86_64

 

[opc@oci-oel7test ~]$ uname -r
4.1.12-103.6.1.el7uek.x86_64

Install Benchmarking Suite

sudo yum install git 
sudo yum install php
sudo yum install fio
sudo yum install gnuplot
sudo yum install wkhtmltopdf
sudo yum install zip

git clone https://github.com/cloudharmony/block-storage.git

Run Test

Will run test on IOPS, Latency and Throughput with 32 parallel threads, so we do not hit CPU limit on either box.

Test can be run on raw drives.

Test does several iterations, and each iteration takes a while to get SSD to Steady State, so test takes hours to run. Be patient.

Test configuration looks same on AWS and Oracle OCI barametal instance:

sudo ./block-storage/run.sh  --test=iops --test=latency --test=throughput --threads=32 --nopurge --precondition_ones --target /dev/nvme3n1 --timeout 3600

Test Results

IOPS

First test IOPS Steady State broken down by Block Sizes:

IOPS Steady State Convergence Plot - All Block Sizes. TOP: Oracle OCI, baremetal BOTTOM AWS
IOPS Steady State Convergence Plot - All Block Sizes.
TOP: Oracle OCI, baremetal
BOTTOM: AWS

For IOPS, small block sizes graphs are most relevant in practical use. As big blocks hit limit on throughput.

Next one is very Informative graph. IOPS as function of Read/Write ratio (graph color), and block size.

IOPS - All Read-Write Mix & Block Size - 2D Plot TOP Oracle OCI baremetal BOTTOM AWS
IOPS - All Read/Write Mix & Block Size - 2D Plot
TOP: Oracle OCI, baremetal
BOTTOM: AWS

As on graph scale is logarithmic, it is hard to see numbers.. Oracle hit up to 982378 IOPS on Read-only small block load. While AWS pushes 413157 IOPS for same load.

Now - same thing in 3d, and no logarithmic scale: color is Read/Write Ratio.

IOPS - All Read-Write Mix & Block Size - 3D Columns TOP Oracle OCI baremetal BOTTOM AWS
IOPS -All Read/Write Mix & Block Size - 3D Columns
TOP: Oracle OCI, baremetal
BOTTOM: AWS

To conclude IOPS benchmark - Oracle Cloud infrastructure holds about 2.5 times more load than Amazon.

Latency

While you can scale IOPS by adding more drives, and number above are pretty high for most practical use, - latency problem you can not fix: it is what it is for particular hardware. And latency is critical factor for most “random lookup” like load. Lets compare it side-by-size.

First graph represents latency for Write-only activity. Color represent I/O block size.

Steady State Convergence Plot - Average Latency - 100 Writes TOP Oracle OCI baremetal BOTTOM AWS
Steady State Convergence Plot - Average Latency - 100% Writes
TOP: Oracle OCI, baremetal
BOTTOM: AWS

A bit hard to see numbers. For Oracle Cloud it is around 0.018ms, for Amazon 0.044ms.

Next graph show Read and Write average latencies. Color represents Read/Write ratio.

Average Latency vs Block Size and Read-Write Mix - 3D Plot TOP Oracle OCI baremetal BOTTOM AWS
Average Latency vs. Block Size and Read/Write Mix - 3D Plot
TOP: Oracle OCI, baremetal
BOTTOM: AWS

Note how read latency and write latency changes little with block size increase.

Interesting to note what write latency is lower than read.

To conclude Latency benchmark - Oracle Cloud Infrastructure latency is about half of what it is on AWS.

Throughput

Throughput is common bottleneck for Big data, Machine Learning and RDBMS workloads, where full dataset needed to be scanned. So nice to the have the bar high.

Below is a graph of throughput as function of Read/Write ratio.

Throughput - All Read-Write Mix & Block Size - 2D Plot 1024KiB TOP Oracle OCI baremetal BOTTOM AWS
Throughput - All Read/Write Mix & Block Size - 2D Plot 1024KiB
TOP: Oracle OCI, baremetal
BOTTOM: AWS

The graph above is for 1MB block size. For 128kB graph looks pretty much same.

To conclude Throughput benchmark - Oracle Cloud Infrastructure single disk throughput is about double of Amazon.

Here is OCI full report and AWS full report excluding 3d graphs.

What SSDs are we testing by the way?

It is relatively easy to say for Oracle Cloud, as baremetal been used.

[opc@oci-oel7test ~]$ sudo lspci -v

 ...
 25:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 
 172X (rev 01) (prog-if 02 [NVM Express])
  Subsystem: Oracle/SUN Device a803
  Physical Slot: 13
  Flags: bus master, fast devsel, latency 0, IRQ 28, NUMA node 0
  Memory at c6000000 (64-bit, non-prefetchable) [size=16K]
  Capabilities: [40] Power Management version 3
  Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit+
  Capabilities: [70] Express Endpoint, MSI 00
  Capabilities: [b0] MSI-X: Enable+ Count=129 Masked-
  Capabilities: [100] Advanced Error Reporting
  Capabilities: [148] Device Serial Number c8-01-00-61-99-38-25-00
  Capabilities: [158] Power Budgeting 
  Capabilities: [168] Alternative Routing-ID Interpretation (ARI)
  Capabilities: [178] #19
  Capabilities: [1d8] Latency Tolerance Reporting
  Kernel driver in use: nvme
  Kernel modules: nvme

  ...

So it is this NVMe:
http://www.samsung.com/semiconductor/global/file/insight/2016/08/Samsung_PM1725a-1.pdf

And our benchmark hit specification numbers from vendor.

For Amazon Web services get to details of hardware is not easy as Xen abstracts PCI layer. So we can only guess how far we are from Manufacturing specifications.

Conclusion

I/O numbers looks about 2 time better on Oracle Cloud Infrastructure. And some of them can be compensated by throwing more disks to it, but others, like Latency can not..

Most probably latency improvements comes from elimination of virtualization layer, and so - lighter stack of handling interrupts. When stack is already as thin as NVMe, it matters.

We gave AWS a try.. But in practical use even if it would be on same level of performance as Oracle, we can not use Amazon NVMe instance store for any persistent data, as its get wiped out on instance shutdown.

Next fastest persistent storage option on AWS would be EBS io1. But those is another 10times slower in latency, 20 times less IOPS, and 5 times less throughput!

So with this layout, Oracle OCI instance is a lot faster on I/O than AWS.