In this blog, we show how we created a Datastax Cassandra cluster on Oracle Cloud Infrastructure (OCI) using Terraform and benchmark Oracle Cloud baremetal machines running Cassandra stress.

1. Steps to Create the Cluster

Go here to download the Terraform project
Follow the guide to setup your environment

Edit the file env-vars and fill in all the relevant info for your OCI account

edit variables.tf and select the shape of your OCI instance, we used DenseIO1.36.

variable "DSE_Shape" 
{ 
default = "BM.DenseIO1.36" 
# default = "VM.DenseIO1.8" 
# default = "VM.DenseIO1.16" 
}  

# DSE cluster deployment topology by availability domain (Phoenix region: PHX) 
variable "DSE_Cluster_Topology_PHX_Region" { 
type = "map"  
 default = { 
  AD1_Count = "2" 
  AD2_Count = "2" 
  AD3_Count = "2" 
  } 
 }  
# DSE cluster deployment topology by availability domain (Ashburn region: IAD) 
variable "DSE_Cluster_Topology_IAD_Region" { 
 type = "map"  
 default = { 
  AD1_Count = "2" 
  AD2_Count = "2"
  AD3_Count = "2" 
  } 
 }  

Run % terraform plan

Run % terraform apply

(It takes 20 - 30 minutes to run, and you will have a 12 nodes baremetal up and running)

2. Benchmark

Using the Cassandra-stress default workload

Write Work Load

First we try writing 100 million records

cassandra-stress write n=100000000 cl=LOCAL_QUORUM -mode native cql3 user=cassandra password=xxxxxx -schema keyspace="keyspace1" "replication(factor=3)" -pop seq=1..100000000 -rate threads=100 -log file=~/load100M`date +%Y%m%d%H%M%S`.log -node 10.0.0.4,10.0.0.5,10.0.1.2,10.0.1.3,10.0.2.2,10.0.2.3 -graph file=oci_bm_local_quorum.html title=oci_bm revision=benchmark-1  

Results:  
Op rate : 77,087 op/s [WRITE: 77,087 op/s]  
Partition rate : 77,087 pk/s [WRITE: 77,087 pk/s]  
Row rate : 77,087 row/s [WRITE: 77,087 row/s]  
Latency mean : 1.3 ms [WRITE: 1.3 ms]  
Latency median : 1.2 ms [WRITE: 1.2 ms]  
Latency 95th percentile : 1.8 ms [WRITE: 1.8 ms]  
Latency 99th percentile : 1.9 ms [WRITE: 1.9 ms]  
Latency 99.9th percentile : 15.1 ms [WRITE: 15.1 ms]  
Latency max : 591.9 ms [WRITE: 591.9 ms]  
Total partitions : 100,000,000 [WRITE: 100,000,000]  
Total errors : 0 [WRITE: 0]  
Total GC count : 0  
Total GC memory : 0.000 KiB  
Total GC time : 0.0 seconds  
Avg GC time : NaN ms  
StdDev GC time : 0.0 ms  
Total operation time : 00:21:37  

Datastax Cassandra stress writing 100 million records. pic1
Datastax Cassandra stress writing 100 million records. pic2
Next we ran 9 more Cassandra stress client at the same time. One of the subsequent results is shown below:

Results:  
Op rate : 24,607 op/s [WRITE: 24,607 op/s]  
Partition rate : 24,607 pk/s [WRITE: 24,607 pk/s]  
Row rate : 24,607 row/s [WRITE: 24,607 row/s]  
Latency mean : 4.0 ms [WRITE: 4.0 ms]  
Latency median : 2.0 ms [WRITE: 2.0 ms]  
Latency 95th percentile : 11.4 ms [WRITE: 11.4 ms]  
Latency 99th percentile : 18.0 ms [WRITE: 18.0 ms]  
Latency 99.9th percentile : 221.8 ms [WRITE: 221.8 ms]  
Latency max : 1022.4 ms [WRITE: 1,022.4 ms]  
Total partitions : 100,000,000 [WRITE: 100,000,000]  
Total errors : 0 [WRITE: 0]  
Total GC count : 0  
Total GC memory : 0.000 KiB  
Total GC time : 0.0 seconds  
Avg GC time : NaN ms  
StdDev GC time : 0.0 ms  
Total operation time : 01:07:43  

Finally we tried to write 1 billion records using one Cassandra stress client with 1000 threads:

cassandra-stress write n=1000000000 cl=LOCAL_QUORUM -mode native cql3 user=cassandra password=xxxxxxxxx -schema keyspace="keyspace1" "replication(factor=3)" -pop seq=1000000001..2000000000 -rate threads=1000 -log file=~/load100M`date +%Y%m%d%H%M%S`.log -node 10.0.0.4,10.0.0.5,10.0.1.2,10.0.1.3,10.0.2.2,10.0.2.3 -graph file=oci_bm_local_quorum_11.html title=oci_bm revision=benchmark-11  

Results:  
Op rate : 156,651 op/s [WRITE: 156,651 op/s]  
Partition rate : 156,651 pk/s [WRITE: 156,651 pk/s]  
Row rate : 156,651 row/s [WRITE: 156,651 row/s]  
Latency mean : 6.4 ms [WRITE: 6.4 ms]  
Latency median : 2.0 ms [WRITE: 2.0 ms]  
Latency 95th percentile : 18.1 ms [WRITE: 18.1 ms]  
Latency 99th percentile : 31.7 ms [WRITE: 31.7 ms]  
Latency 99.9th percentile : 474.5 ms [WRITE: 474.5 ms]  
Latency max : 1959.8 ms [WRITE: 1,959.8 ms]  
Total partitions : 1,000,000,000 [WRITE: 1,000,000,000]  
Total errors : 0 [WRITE: 0]  
Total GC count : 0  
Total GC memory : 0.000 KiB  
Total GC time : 0.0 seconds  
Avg GC time : NaN ms  
StdDev GC time : 0.0 ms  
Total operation time : 01:46:23  

Datastax Cassandra stress write 1 billion records. client with 1000 threads. pic1
Datastax Cassandra stress write 1 billion records. client with 1000 threads. pic2

Read work load

For read stress test we use 2 Cassandra stress clients.

Each client will read random record from the cluster, where the key to read is chosen randomly, with a normal distribution, mean at 1 billion and standard deviation of 500 millions.

cassandra-stress read duration=240m cl=LOCAL_QUORUM -mode native cql3 user=cassandra password=xxxxxxxx -schema keyspace="keyspace1" -pop dist=gaussian\(1..2000000000,1000000000,500000000\) -rate threads=1500 -log file=~/read_1500_`date +%Y%m%d%H%M%S`.log -node 10.0.0.4,10.0.0.5,10.0.1.2,10.0.1.3,10.0.2.2,10.0.2.3 -graph file=oci_bm_local_quorum_read.html title=oci_bm revision=benchmark-1  
Results:  
Op rate : 47,646 op/s [READ: 47,652 op/s] 
Partition rate : 47,646 pk/s [READ: 47,652 pk/s] 
Row rate : 47,646 row/s [READ: 47,652 row/s] 
Latency mean : 31.4 ms [READ: 31.4 ms] 
Latency median : 14.4 ms [READ: 14.4 ms] 
Latency 95th percentile : 117.0 ms [READ: 117.0 ms] 
Latency 99th percentile : 182.2 ms [READ: 182.2 ms] 
Latency 99.9th percentile : 345.8 ms [READ: 345.8 ms] 
Latency max : 9781.1 ms [READ: 9,781.1 ms] 
Total partitions : 686,126,907 [READ: 686,126,907] 
Total errors : 0 [READ: 0] 
Total GC count : 0 
Total GC memory : 0.000 KiB 
Total GC time : 0.0 seconds 
Avg GC time : NaN ms 
StdDev GC time : 0.0 ms 
Total operation time : 04:00:00  

Datastax Cassandra stress read. Read random record from the cluster, where the key to read is chosen randomly, with a normal distribution, mean at 1 billion and standard deviation of 500 millions. Pic1
Datastax Cassandra stress read. Read random record from the cluster, where the key to read is chosen randomly, with a normal distribution, mean at 1 billion and standard deviation of 500 millions. Pic2

3. Cost

Based on the pricing published by Oracle as of March 1, 2018 with OCPU per hour price of $0.1275, the price to deploy 6 DenseIO.36 servers will be $27.54 per hour, or $19,828.80 per month. Sounds expensive? well.. keep reading.

Conclusion

Our testing showed that write operations for Cassandra clusters running on OCI baremetal machines was extremely fast. For read operations, Cassandra performed significantly better on OCI baremetal machines compared to running clusters on infrastructures of other vendors in similar price range.

While we acknowledge that a direct apples-to-apples counterpart for Oracle’s DenseIO.36 instances is not possible with the instance configurations offered in AWS at this time, we know that Amazon’s new EC2 Baremetal i3 instances may be able to compete.

AWS i3.metal instances are powered by Intel Xeon E5-2686 v4 (Broadwell) processors with 36 hyperthreaded cores, 512 GiB of memory, and 15.2TB of NVMe SSD-backed instance storage. Leveraging next gen networking technologies based on Elastic Network Adapter (ENA), the instances can deliver high networking throughput and lower latency with up to 25 Gbps of aggregate network bandwidth. This could allow AWS to come closer to drawing level with OCI’s performance benchmarks, at least for Cassandra. .. But AWS bill will be much higher then OCI. And this is only for compute. Do not forget - you will be billed for cross-AZ traffic as well.

OCI keep the title of the best ROI as of March 2018.

In next blogs we will benchmark Cassandra on OCI new X7 generation compute.