From csi702

Contents Amazon Web Service HowTo Signing Up Adding Credits How this Works Running A Job Maximum Number of Instances Modifying an Existing AMI AMI Configuration For CSI702 Working Ubuntu AMI Configuration Shared Storage on EC2 References

1 Amazon Web Service HowTo

1.1 Signing Up

Go to AWS and click on 'Sign up now'. Create an amazon account if you don't already have one. You'll have to link a credit card to the account, this whole process should be self explanatory. Once you've verified your account through the email they send you, you can now log in.

1.1.1 Adding Credits

Starting from the main AWS page, log in. At the top center of the page, click on 'Account'. Click on 'Payment Method'. At the bottom of this page 'Redeem an AWS Credit Coupon'. Its all down hill from here...

1.2 How this Works

You have to sign up for Amazon S3 and Cloudfront. It looks like S3 is the storage end of things, and Cloudfront is the computing end.

1.3 Running A Job

After you've signed up for these services and verified your email, check out AWS Resources for tons of info and more clearly written tutorials!

1.4 Maximum Number of Instances

"You are limited to running 20 On-Demand or Reserved Instances and running 100 Spot Instances per region. If you need more instances, complete the Amazon EC2 instance request form and your use case and instance increase will be considered. Limit increases are tied to the region they were requested for."

1.5 Modifying an Existing AMI

If you want to create an image that has your own software configuration, you need to modify a pre-existing AMI by bundling it up into a private AMI. Details for this can be found here Customize An Existing AMI.

1.6 AMI Configuration For CSI702

In order to reduce duplicate efforts in configuring EC2 for our work, it would probably be good to have a standard AMI to share among students. After doing some searching around, there is an AMI that "can be used to run a small beowulf cluster on Amazon EC2 and perform on-demand parallel computations with C, Fortran, or Python." More information on this AIM is available at Data Wrangling Image: Fedora Core 6 MPI Compute Node with Python Libraries.

I tried this AMI, but can't get it to run more than one thread under OpenMP. Any suggestions?

1.6.1 Working Ubuntu AMI Configuration

Ubuntu provides a set of official AMI configurations which are preloaded on Amazon EC2. Complete information on using the Ubuntu AMI configurations is available at Ubuntu AMI.

The following is a general outline of the steps I followed to launch an Ubuntu AMI with 8 nodes and test with some basic OpenMP code:

1. From the ENC dashboard select Spot Requests and click the "Request Spot Instances" button.
   1. I used a Spot Instance instead of an On-Demand Instance because the 8 node AMI I used is classified as a "High-CPU Extra Large" instance. This costs $0.68 per hour according to the Pricing Guidelines. In comparison, the price for spot instances as of April 16 was only about $0.20 per hour for "High-CPU Extra Large" instances.
2. There is an excellent tutorial that walks you through the steps necessary to launch your first AMI at GettingStartedGuide
3. The identifier for the AMI I used is ami-55739e3c
4. Once the instance is started, ssh into the node using: ssh -i <your .pem file> ubuntu@<public DNS>
5. Loading the necessary software is simple on Ubuntu using aptitude
   1. Note: I install subversion because I store my homework code in a svn repository, which, besides providing version control and backup, provides an easy way of loading the code onto a new box. Google Code provides free svn hosting at Project Hosting. You can create a new project, with its own svn repository, at Create Project.
   2. To install subversion and gcc, type the following in the terminal:
      1. sudo apt-get update
      2. sudo apt-get install subversion
      3. sudo apt-get install gcc
   3. Alternatively, you can install gcc by typing:
      1. sudo apt-get update
      2. sudo apt-get build-essential
   4. To install mpich2, type the following in the terminal:
      1. sudo apt-get install mpich2
   5. Then configure the .mpd.conf file:
      1. vi .mpd.conf
   6. insert a line like
      1. MPD_SECRETWORD=opensesame
   7. save then type:
      1. chmod 600 .mpd.conf
   8. You should be able to run mpich2 by typing mpd&
   9. If you want to simply upload files from your computer to the AWS instance:
      1. scp -i <your .pem file> <the file to upload> ubuntu@<public DNS>:/home/ubuntu/<whatever directory you want>
6. Now you're ready to go! After following these steps I've compiled some basic OpenMP codes and seen all 8 cores in use.

1.7 Shared Storage on EC2

One of the challenges in running MPI in EC2 is configuring the environment with shared storage ^[1]. This is going to require that each node to install s3fs and mount an S3 bucket to the local environment. SSH keys would have to be stored in this location.

1.8 References