Setting up AWS EC2

Instance Type: m4.xlarge, 4 node cluster, 2 in each AZ
Storage: Two EBS volumes, data volume 400GB, 150GB log volume, root volume 150GB (General Purpose SSD)
OS:Amazon Linux AMI 2016.09.0 (HVM), SSD Volume Type - ami-b953f2da
The Amazon Linux AMI is an EBS-backed, AWS-supported image. The default image includes AWS command line tools, Python, Ruby, Perl, and Java. The repositories include Docker, PHP, MySQL, PostgreSQL, and other packages.
Root device type: ebs Virtualization type: hvm

File System:

xfs is not available in amazon linux, so using ext4

$ lsblk  
xvda 202:0 0 10G 0 disk  
└─xvda1 202:1 0 10G 0 part /  
xvdb 202:16 0 10G 0 disk  
xvdc 202:32 0 50G 0 disk  
$ sudo mkfs -t ext4 /dev/xvdc => data volume  
$ sudo mkfs -t ext4 /dev/xvdb => log volume  
$ sudo mkdir /dsedata /dselog  
$ sudo mount /dev/xvdc /dsedata  
$ sudo mount /dev/xvdb /dselog  
$ sudo vi /etc/fstab  
/dev/xvdc /dsedata ext4 defaults 0 2  
/dev/xvdb /dselog ext4 defaults 0 2  


Setup a private subnet one each for one AZ for the cassandra nodes
2 ENIs for each of 2 seed nodes (one in each AZ)
Setup a NAT instance/NAT Gateway in public subnet for software download from internet
Setup bastion host for ssh access to cassandra nodes in a public subnet, one for each AZ

Setting up Cassandra


$ sudo useradd cassandra  
$ cat /sys/class/block/xvdc/queue/scheduler  

$ cat /sys/class/block/xvdc/queue/rotational  

$ cat /sys/class/block/xvdc/queue/read_ahead_kb  

$ touch /var/lock/subsys/local  

$ echo 8 > /sys/class/block/xvdc/queue/read_ahead_kb  
$ echo 8 > /sys/class/block/xvdb/queue/read_ahead_kb  

net.core.rmem_max = 16777216  
net.core.wmem_max = 16777216  
net.core.rmem_default = 16777216  
net.core.wmem_default = 16777216  
net.core.optmem_max = 40960  
net.ipv4.tcp_rmem = 4096 87380 16777216  
net.ipv4.tcp_wmem = 4096 65536 16777216  
vm.max_map_count = 1048575  

$ sudo cat /sys/kernel/mm/transparent_hugepage/enabled  
always [madvise] never  

$ sudo cat /sys/kernel/mm/transparent_hugepage/defrag  
[always] madvise never  

$ sudo echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag  

$ sudo cat /sys/kernel/mm/transparent_hugepage/defrag  
always madvise [never]  

$ cat /proc/sys/vm/zone_reclaim_mode  

session required  

$ sudo yum install java-1.8.0  
$ sudo yum remove java-1.7.0-openjdk  
$ sudo /usr/sbin/alternatives --config java  
$ java -version  
openjdk version "1.8.0_111"  
OpenJDK Runtime Environment (build 1.8.0_111-b15)  
OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode)  
$ sudo /usr/sbin/alternatives --config java  
There is 1 program that provides 'java'.  
 Selection Command  
*+ 1 /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java  
Enter to keep the current selection[+], or type selection number:  

$ python  
Python 2.7.12 (default, Sep 1 2016, 22:14:00)  
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2  
Type "help", "copyright", "credits" or "license" for more information.  

$ chkconfig --list ntpd  
ntpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off  
$ ps -fe | grep ntp  

cassandra - memlock unlimited  
cassandra - nofile 100000  
cassandra - nproc 32768  
cassandra - as unlimited  

sudo chown cassandra:cassandra /dsedata  
sudo chown cassandra:cassandra /dselog  
sudo chown cassandra:cassandra /dsedata/*  
sudo chown cassandra:cassandra /dselog/*  


$ curl --user <email_id>:<password> -O  

$ chmod u+x  

$ sudo ./ --optionfile datastax-DC1-cassandra.prop --mode unattended  

$ cat datastax-DC1-cassandra.prop | grep -v ^# | sed '/^$/d'  

hints_directory: /dsedata/hints  
authenticator: PasswordAuthenticator  
authorizer: CassandraAuthorizer  
role_manager: CassandraRoleManager  
- /dsedata/data  
commitlog_directory: /dselog/commitlog  
saved_caches_directory: /dsedata/saved_caches  
- seeds: "" (add one node in each AZ as seed, give ENI private IP)  
endpoint_snitch: Ec2Snitch  




cat .cassandra/cqlshrc  
request_timeout = 3600  

cqlsh> ALTER KEYSPACE "system_auth"  
WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'ap-southeast_1_cassandra' : 4};  
cqlsh>alter KEYSPACE dse_leases WITH replication = {'class': 'NetworkTopologyStrategy', 'ap-southeast_1_cassandra': '3'};  
cqlsh>create role admin with password='******' and superuser=true and login=true;  
cqlsh>ALTER ROLE cassandra WITH PASSWORD='SomePasswordLikeThisIsBlocked' and superuser=false;  

$ dsetool perf cqlslowlog enable  


$ sudo service dse start  
Starting DSE daemon : dse  
DSE daemon starting with only Cassandra enabled (edit /etc/default/dse to enable other features)  

$ sudo service dse status  
dse is running  

$ cat /proc/16345/limits  
Limit Soft Limit Hard Limit Units   
Max cpu time unlimited unlimited seconds   
Max file size unlimited unlimited bytes   
Max data size unlimited unlimited bytes   
Max stack size 8388608 unlimited bytes   
Max core file size 0 unlimited bytes   
Max resident set unlimited unlimited bytes   
Max processes 32768 32768 processes  
Max open files 100000 100000 files   
Max locked memory unlimited unlimited bytes   
Max address space unlimited unlimited bytes   
Max file locks unlimited unlimited locks   
Max pending signals 64118 64118 signals   
Max msgqueue size 819200 819200 bytes   
Max nice priority 0 0   
Max realtime priority 0 0   
Max realtime timeout unlimited unlimited us   

$ dsetool ring  
$ dsetool status  

Cassandra on AWS - Some Best Practices:

    1. SSD is the recommended storage mechanism for Cassandra.
    2. The recommended file system for all volumes is XFS. Ext4 might be used by preference.
    3. Choose instance types with enhanced networking enabled. Enhanced networking offers better network performance.
    4. Insert heavy workloads are CPU-bound in Cassandra before becoming IO-bound.
    5. With Amazon EC2 for write-heavy workloads, you should look at instance types with at least 4 vCPUs.
    6. If you are just starting out with Cassandra, plan for cluster growth from the beginning.
    7. Choose Ec2MultiRegionSnitch to avoid complications when you decide to expand your cluster.
    8. We highly recommend that you spread your seed nodes across multiple Availability Zones.
    9. Attach an ENI to each seed node and add the ENI IP address to the list of seed nodes in the .yaml configuration file.
    10. Multiple EBS volumes for data and one separate ebs volume for commit log.
    11. Amazon EBS also provides a feature for backing up the data on your EBS volumes to Amazon S3 by taking point-in-time snapshots.
    12. Setup Autoscaling and automation scripts - Autoscaling groups (seed, non seed)
    13. Setup Bastion Host
    14. HA for OpsCenter and separate autoscaling group for it
    15. Custom AMI with Cassandra package is also available
    16. Netflix-built and open-sourced tool Priam or tablesnap for backups
    17. We should go for M4 series, since we are not using ephemeral disks that come with M3, so 10$ per server reduced cost