OpenStack Persistent Instances

What is Persistent?

Persistent VMs behave like ‘normal’ systems rather than ‘cloud’ systems. Each VM has it’s own unique backing storage where any changes during run time are permanently stored.

What are they good for

These are good for one off systems that collect local state.

What are they NOT good for

They are not good for launching multiple systems.

They are also not the place to store important data. The volumes hosting persistent VMs are on redundant storage but are not backed up. Important data should still be stored on AFS or NFS network storage where regular backups happen.

Pre-requisites:

• have an OpenStackAccount
• have an ssh key uploaded
• have security groups defined, at least one to open ssh

Creating a bootable volume

The easiest way to get a bootable volume is to start from an image, the same sort used in Ephemeral VMs.

Steps:

• Login to Horizon
• If you belong to more than one project select the correct project from the drop-down menu in the left column.
• Select Volumes from the left column.
• Select + Create Volume button at top right.
• Fill in a descriptive Volume Name, for example <hostname>-root
• Fill in Size (GB), this needs to be large than the expanded size of the base image. A size of 16 or 32 is recommended.
• From Volume Source select Image
• From Use image as a source select the image you want to use. recommended options are:

Fully supported images

• CSAIL-Ubuntu-24.04LTS a standard CSAIL Ubuntu system (recommened)
• CSAIL-Ubuntu-22.04LTS

Beta / Best effort images

• CSAIL Ubuntu 26.04LTS (deta has AFS issues)
• CSAIL Ubuntu 22.04LTS
• CSAIL-AlmaLinux-10
• CSAIL-Debian-13
• CSAIL-Fedora-43

Upstream Vendor Images (no CSAIL integration)

• Ubuntu-24.04LTS
• Ubuntu-24.04LTS

The volume will be in Downloading state while it copies the image onto the new volume. Wait until its status is Available before proceeding.

Basic web based launch process

This describes the minimal steps needed to boot. Optional steps such as booting multiple instances at once or setting a fixed IP address are not covered here.

Steps, it is much faster to do than to explain, honest:

• Login to Horizon
• If you belong to more than one project select the correct project from the drop-down menu in the left column.
• Select Instances from the left column menu
• Select + Launch Instance button at top right
• Fill in an arbitrary Instance Name
• Select a Flavor this is a preset selection of CPU,RAM, and Disk resources. When in doubt “m1.4core” is a good option.
• Select Boot from volume from Instance Boot Source
• Select the volume created above from Volume
• Select the Access & Security tab, check that your key is listed in Keypair, it should be selected by default.
• Alos in the Access & Security select the security groups you need to ope required pots like ssh of https all network access is blocked by default

The security groups page has info on settign up security groups

• Select the networking tab
• Select the + in the block labeled inet under Available Networks, this will then move the inet block to Selected Networks and be labled nic:1
• Finally select the Launch button.

You should then be directed back to the Instances page where you will see your new instance in the spawning state. After some time (a few seconds up to about 2min) the system will be updated to Active. This means the VM has been successfully built and launch, the operating system is active but still booting at this point. It may be another minute for the system to boot up to the point where ssh is running and you can connect.

To connect to the system note the IP your system was assigned, this is shown on the instances page. Then connect as the ‘ubuntu’ user using your ssh private key. If you launched an “CSAIL” image you could also connect whit your CSAIL usrename and password but the ‘ubuntu’ user has passwordless sudo access where your normal user does not.

ssh -i </path/to/your/private_key> ubuntu@<IP>

After first boot, you should login and remove the cloud-init package by runningsudo apt-get remove cloud-init. This provides very useful services for Ephemeral VMs but some of these, for example regenerating the ssh host key, are problematic for persistent systems.

Turning off cloud-init after initial configuration

In Ubuntu, the cloud-init package is responsible for setting the hostname, resizing the root disk (if the image is smaller than the instance type you’re booting) and other cloudy things. The package fetches much of this info from the OpenStack metadata service. Users reported errors from “url_helper.py” when their VMs could not reach the service, and boot times of several minutes.

If you’re running a persistent VM that already has your key in it and knows its host name, it is recommended to remove the cloud-init package, to remove the dependency on the metadata service in case it is down. This also prevents regeneration of the host RSA key on each boot.

User accessible fault recovery

On the Instances web page there are several options available in the Actions column (under “more”) that are helpful in recovering from faults.

• Soft reboot instance this is the equivalent of CTRL-ALT-DEL on the console, it asks the running operating system to cleanly reboot if it can.
• Hard reboot instance this is the equivalent of pulling the power cord then restarting.
• Terminate instance followed by relaunching. This is similar to the hard reboot above, but also recreates the virtual container possibly on a different physical system, so this will help you self recover for faults in the underlying hardware.

Occasionally persistent system get in a state that needs interaction on the console to resolve, for example hanging at the grub prompt or requiring a manual fsck. The web interface provides a javascript and VNC based console, but the VNC protocol connection from your browser to the system is not encrypted. Do not type passwords over VNC, if you must type passwords immediately log in over a secure (ssh) channel and change the password as soon as the underlying issue is resolved!

The VNC console can be found by clicking on the instance name on the Instances page then clicking on the Console tab at the top. Some browsers, notably recent versions of Firefox, will not render the console script because it is insecure content embedded in a secure page. For those browsers click the Click here to show only console link.

If you are still unable to recover your system contact help@csail.mit.edu and we’ll help recover your system.