Overview

Amazon Web Services provide to the user an entire machine, charged per the hour, or reserved and paid by the year. Storage and computation are charged separately. Computation is called EC2, Elastic Cloud Computing, and storage is called EBS, Elastic Block Storage. An EC2 instance is a running computer that you access over the internet, for us, mostly by ssh.

Better than "elastic" computing, it could be called "plastic throw-away" computing. It is easy to set up instances when you need them, and destroy them when you don't need them. Rather than reboot, you reinstall fresh. For that reason, you need to be able to configure a fresh instance programmatically, or else to snapshot your instance and store the snapshot permanently somewhere; to clone additional instances. Most of this is about cloning instances. This isn't so much about having someone supply you one box, it's about scaling to have someone supply you between one and a thousand boxes, growing as your business grows, with very little capital outlays on your part.

In the course, you will create your own instances, and run as root. For now, user accounts have been created for you, using delightfully difficult to remember names of Pokemon characters. Following the usual CS standards, the lead account, in this case pikachu (who else!) can log into any account. It is the grader's account, and that way the grader can help you as well as grade your assignments.

I will talk about how to setup an instance farther down this page. For now, let's just focus on how you will log in.

Logging into our AWS instance.

To log in, you will need three things:

  1. An IP address, of where the instance is on the internet.
  2. A username, for your account on the instance.
  3. A secret key file, with extension .pem, which allows the user to log into that instance.
For various reasons, EC2 use public-key cryptography to authenticate, rather than passwords. There are no passwords. Instead there is a key file, which you keep secret and present to the ssh software in order to log in. Treat the file as if it contained your password. It is not exactly a password in that the contents of the file never leave your side of the ssh connection. The server has the public part of the key in the .ssh/authorized_keys file of the user's home directory. If the ssh proves that the private and public parts fit together, you get to log in. 


    hohokus-2:csc521 burt$ ./ec2ssh.sh  pikachu

     db d88888D   j88D      db .d888b. .d888b.     db    dD       db  db .d888b. 
    o88 VP  d8'  j8~88     o88 VP  `8D 88' `8D    o88   d8'      o88 o88 VP  `8D 
     88    d8'  j8' 88      88    odD' `V8o88'     88  d8'        88  88    odD' 
     88   d8'   V88888D     88  .88'      d8'      88 d8888b.     88  88  .88'   
     88  d8'        88  db  88 j88.      d8'   db  88 88' `8D db  88  88 j88.    
     VP d8'         VP  VP  VP 888888D  d8'    VP  VP `8888P  VP  VP  VP 888888D 

            /\       __                  Hello! The Pokemon have moved
            | \_____/ /                  to a new server. Our IP is:
            /         | 
           / ^.   ^   |                       174.129.16.112
           \0 _   0   |     _
           _\   _   __|    | |           TTYL!
           \/   \  <__|  _/ / 
            |       __|_/ _/             Your friend Pikachu.
            |      <__|__/   
             \______/
             /__||__\   


    Connection closed by 75.101.164.86

	-------
	[root@ip-10-251-43-16 ~]# date
	Tue Nov 24 11:14:53 EST 2009
	[root@ip-10-251-43-16 ~]# uptime
	 11:14:56 up 36 days, 15:14,  1 user,  load average: 0.00, 0.00, 0.00
	[root@ip-10-251-43-16 ~]# shutdown -h now
	
	Broadcast message from root (pts/0) (Tue Nov 24 11:15:04 2009):
	
	The system is going down for system halt NOW!
	[root@ip-10-251-43-16 ~]# Connection to 75.101.164.86 closed by remote host.
	Connection to 75.101.164.86 closed.
	hohokus:csc521 burt$

As of 22 Nov 09, the IP address is 174.129.16.112 (Previous IP: 75.101.164.86)

That takes care of step one.

Warning- the IP address might change every once in a while. I will post the new address here if it does.

I have nothing fancy for the username and pem file. Choose a name from the list below, email me or aravind, and we will send you the pem file.

Usernames:

  1. bulbasaur
  2. charizard
  3. jigglypuff
  4. jolteon
  5. mewtwo
  6. pikachu (taken)
  7. rayquaza
  8. squirtle
  9. starmie
  10. vaporeon

SSH to account:

Assuming you are vaporeon, and that 174.129.16.112 is the IP address, cd to the directory with the vaporeon.pem file in it, and run: 


    ssh -i vaporeon.pem vaporeon@174.129.16.112

Transferring files in and out of your account

Use the ftp or the copy functionality of ssh to copy files to and from the instance. You will need to know how to use tar to make archives, so read the man page on tar. In short, it's like this: 


    tar cvf my_directory.tar my_directory
Then transfer my_directory.tar. At the destination: 

    tar xvf my_directory.tar
Use tvf instead of xvf to peek inside a tar file without extracting the contents.

To move files from machine X to machine Y (I'm supposing Y is the aws machine, but whatever you want), on the X machine use: 


    sftp -oIdentityFile=vaporeon.pem vaporeon@174.129.16.112
Sorry about the painful command line syntax. The -o is a sort of quasi-standard for including the options of one program inside another. It means "the sftp option which is to pass to ssh the ssh option named IdentityFile, giving it value vaporeon.pem".

Once connected, there are get, put, ls, cd, and ls and lcd (local ls and local change directory) commands. Also scp can be used with syntax more like cp (no "connection").

As always, consult the man pages. It's all there.

Windows:

I have mostly talked about having a command line interface to ssh and sftp. I am not as familiar with Windows as I am with unix, but I will try in this section to help with using AWS in the manner described above with a typical Windows client.

The way I find most people using ssh in Windows is by the Putty program. Download the entire package, including PuTTY, puTTYgen, plink, pscp and psftp. At the time of this writing the PuTTY download site has an installer for one-click installation of the PuTTY suite.

PuTTY cannot use .pem files directly. The .pem must be converted to PuTTY's special ppk format. Run PuTTYgen, load the pem file and then save it with the ppk extension. The conversion will be done on save.

The PuTTY graphical tool can be used with the ppk file. Drill down among the options of the PuTTY configuration panel to ssh->keys, and browse for the ppk file. Returning to the session item of the configuration panel, write the IP address and go. When the black terminal window opens, it will ask for the login name, this doesn't seem to be an option to set in the configuration panel.

Plink can be used to run ssh from the command line. It worked but it didn't set the terminal correctly, so it only barely worked. I didn't pursue making that work better. However, psftp worked well. It was invoked at the command line with the same syntax as for unix, -i name-of-key-file.ppk and all.

As for the several other inconveniences (tar, end-of-line character inconsistencies), you have my best wishes and deepest sympathy.

Starting an instance:

The steps to fully setting up an instance ar:

  1. Log into your aws management console (browser).
  2. Starting an instance:
    1. Go to the instance panel of the management console.
    2. Create instance; select 1, small instance, a key-pair for root.
    3. (continued) In advanced instance creation options, select us-east-1a.
  3. Attach the EBS volume to the instance.
    1. Go to the volumes panel.
    2. Select the volume to attach and use attach button.
    3. In the pop-up, select the instance and the device (leave at /dev/sdf).
    4. To make an EBS, well, you don't make and EBS, you create a filesystem on a device. If you attached as /dev/sdf, then run:
      yes | mkfs -t ext3 /dev/sdf
      to make a partition-less disk. Then mount /dev/sdf where you want.
  4. Attach a Elastic IP to the instance. Go to the IP panel, select, and apply.
    1. Go to the Elastic IP panel.
    2. Select the IP
    3. In the button panel, select Associate.
    4. In the popup, select the instance.
  5. Configure the fresh instance.
    1. Log in as root, using the key pair select in the instance creation panel, at the IP selected in the Elastic IP panel.
    2. Mount the EBS onto the filesystem. (The association that you have already done is analogous to attaching hardware; mounting is attaching the filesystem on the software level.) A typical mount command is mount /dev/sdf /opt.
    3. yum is used to update and install software. User accounts are created to reconnect with the user home directories that are on the EBS filesystem.
    4. A Perl script contains everything that needs to be done; login as root to the fresh instance; navigate the root's permanent home directory (not /root, that is a per-instance home); find the configure script, and run it. (E.g. cd /opt/home/root; ./config.pl)
  6. When it's time to pack up and move, all we need is the EBS.
    1. umount the filesyste, e.g., /dev/sdf
    2. dump 0af ebs.dmp /dev/sdf
    3. move the ebs.dmp file to a new location.
    4. yes | mkfs -t ext3 /dev/sdf (a partition-less disk)
    5. mount and restore: cd /opt; restore -rf ebs.dmp
  7. EBS snapshots can be shared. To move, rather than dump/restore:
    1. On the FROM instance/ebs, snapshot the ebs. Give permissions to the AWS Account ID of the TO instance/ebs.
    2. On the TO aws account, create an ebs referencing the snapshot as a creation option.
    3. When created, attach and mount the ebs; you can then delete the FROM snapshot.

Configuration Script 

#!/usr/bin/perl

# configuration tasks for Amazon Web Services
# Csc521 configuration

# author: burt rosenberg
# created: oct 2009
# upadate:

# variables to set:

# where /dev/sdf (the EBS) will be attached:
$MyAttach = "/opt" ;

# where the home directories will be. The homes will be on
# the EBS, and mounted at $MyAttach
# a symlink from /home to here will be made. In /etc/passwd,
# /home/username wil appear.
$MyHomeDir = $MyAttach. "/home" ;

# yum updates
$YumTasks = <<EOF;
update
install gcc
EOF

# user accounts to create
$UserTasks = <<EOF;
pikachu:5001
bulbasaur:5002
charizard:5003
jigglypuff:5004
squirtle:5005
starmie:5006
mewtwo:5007
rayquaza:5008
jolteon:5009
vaporeon:5010
EOF


##### code

sub runyum{
   local($cmd) = $_[0] ;
   local($line) ;
   $cmd = "yum -y ".$cmd ;
   printf "** running: %s\n", $cmd ;
   open(F, $cmd." |") ;
   while ($line=<F>) {
      printf "%s", $line ;
   }
}
 
# -M suppress making the homedirectory
# -u gives the specified UID
sub runadduser{
   local($uname, $pid) = ($_[0], $_[1])  ;
   local($line) ;
   $line = `id -u $uname 2>&1` ;
   if ( $line =~ /No such user/ ) {
      printf "** adding user: %s\n", $uname ;
      $cmd = "useradd -M -u ". $pid . " " . $uname ;
      open(F, $cmd." |") ;
      while ($line=<F>) {
         printf "%s", $line ;
      }
   } else {
      printf "** user %s exists, skipping.\n", $uname ;
   }
}

@yums = split(/\n/,$YumTasks) ;
foreach (@yums) {
   runyum( $_ ) ;
}

unless (-l "/home" ) {
   if ( -d "/home" ) {
      `rmdir /home ; ln -s $MyHomeDir /home` ;
      printf "** relinked home directory\n"  ;
   }
} else { 
   printf "** homedir is a link, leaving alone\n" ; 
}

@users = split(/\n/,$UserTasks) ;
foreach (@users) {
   ($un,$pid) = split(/:/,$_) ;
   runadduser( $un, $pid ) ;
}

$df = `/bin/df 2>&1` ;
if ( $df =~ /\/dev\/sdf/ )  {
   printf "** EBS already attached.\n" ;
}
else {
   printf "** Attaching EBS to %s\n", $MyAttach ;
   `mount /dev/sdf $MyAttach` ;
}

##### end of file