c. Create a Cluster Config

Now that AWS ParallelCluster is installed, you will create a configuration file to build a simple HPC system. This file will be generated in your home directory.

Please double check you are using the correct key-pair.

Configuring Your HPC cluster

We will be generating a cluster with the following settings:

  • Head-node and compute nodes will be c4.xlarge instances. Change the instance type if you like but you may run into EC2 limits that may prevent you to create some or too many instances.
  • We use a placement group to maximize the bandwidth between instances and reduce the latency. This packs instances close together inside an Availability Zone.
  • The cluster will have 0 compute nodes when starting, the minimum size is set to 0 and the maximum size of the cluster is set to 8 instances. We are using auto-scaling groups that will grow and shrink between the min and max limits based on the cluster utilization and job queue backlog.
  • A GP2 Amazon EBS volume will be attached to the head-node then shared through NFS to be mounted by the compute nodes on /shared. It is generally a good location to store applications or scripts. Please keep in mind that the /home directory is shared on NFS as well.
  • SLURM will be used as a job scheduler but there are other options you may consider in the future such as SGE.
  • We disable Intel Hyper-threading by setting disable_hyperthreading = true in the configuration file.

For more details about the configuration options of AWS ParallelCluster please review the user guide.

For now you can paste the following commands in your terminal:

IFACE=$(curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/)
SUBNET_ID=$(curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/${IFACE}/subnet-id)
VPC_ID=$(curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/${IFACE}/vpc-id)
AZ=$(curl http://169.254.169.254/latest/meta-data/placement/availability-zone)
REGION=${AZ::-1}

cd ~/environment
cat > my-cluster-config.conf << EOF
[aws]
aws_region_name = ${REGION}

[global]
cluster_template = default
update_check = false
sanity_check = true

[cluster default]
key_name = lab-3-your-key
base_os = alinux2
vpc_settings = public
ebs_settings = myebs
compute_instance_type = c4.xlarge
master_instance_type = c4.xlarge
cluster_type = ondemand
placement_group = DYNAMIC
placement = compute
initial_queue_size = 2
max_queue_size = 8
disable_hyperthreading = true
s3_read_write_resource = *
scheduler = slurm


[vpc public]
vpc_id = ${VPC_ID}
master_subnet_id = ${SUBNET_ID}

[ebs myebs]
shared_dir = /shared
volume_type = gp2
volume_size = 20

[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}
EOF

Now you are ready to launch a cluster, proceed to the next step.