Chapter 8. Lay of the Land

To install (or upgrade) a package from the PyPI archive with pip, as root:

# pip install [--upgrade] <package-name>

To remove the package:

# pip uninstall <package-name>

If you need even newer versions of the clients, pip can install directly from the upstream git repository using the -e flag. You must specify a name for the Python egg that is installed. For example:

# pip install -e git+https://github.com/openstack/
                 python-novaclient.git#egg=python-novaclient

If you support the EC2 API on your cloud, you should also install the euca2ools package or some other EC2 API tool so that you can get the same view your users have. Using EC2 API-based tools is mostly out of the scope of this guide, though we discuss getting credentials for use with it.

There are also several *-manage command-line tools. These are installed with the project's services on the cloud controller and do not need to be installed separately:

Unlike the CLI tools mentioned above, the *-manage tools must be run from the cloud controller, as root, because they need read access to the config files such as /etc/nova/nova.conf and to make queries directly against the database rather than against the OpenStack API endpoints.

	Warning
	The existence of the *-manage tools is a legacy issue. It is a goal of the OpenStack project to eventually migrate all of the remaining functionality in the *-manage tools into the API-based tools. Until that day, you need to SSH into the cloud controller node to perform some maintenance operations that require one of the *-manage tools.

You must have the appropriate credentials if you want to use the command-line tools to make queries against your OpenStack cloud. By far, the easiest way to obtain authentication credentials to use with command-line clients is to use the OpenStack dashboard. From the top-right navigation row, select Project, then Access & Security, then API Access to access the user settings page where you can set your language and timezone preferences for the dashboard view. This action displays two buttons, Download OpenStack RC File and Download EC2 Credentials, which let you generate files that you can source in your shell to populate the environment variables the command-line tools require to know where your service endpoints and your authentication information are. The user you logged in to the dashboard dictates the filename for the openrc file, such as demo-openrc.sh. When logged in as admin, the file is named admin-openrc.sh.

The generated file looks something like this:

#!/bin/bash

# With the addition of Keystone, to use an openstack cloud you should
# authenticate against keystone, which returns a **Token** and **Service
# Catalog**. The catalog contains the endpoint for all services the
# user/tenant has access to--including nova, glance, keystone, swift.
#
# *NOTE*: Using the 2.0 *auth api* does not mean that compute api is 2.0.
# We use the 1.1 *compute api*
export OS_AUTH_URL=http://203.0.113.10:5000/v2.0

# With the addition of Keystone we have standardized on the term **tenant**
# as the entity that owns the resources.
export OS_TENANT_ID=98333aba48e756fa8f629c83a818ad57
export OS_TENANT_NAME="test-project"

# In addition to the owning entity (tenant), openstack stores the entity
# performing the action as the **user**.
export OS_USERNAME=demo

# With Keystone you pass the keystone password.
echo "Please enter your OpenStack Password: "
read -s OS_PASSWORD_INPUT
export OS_PASSWORD=$OS_PASSWORD_INPUT

Warning

This does not save your password in plain text, which is a good thing. But when you source or run the script, it prompts you for your password and then stores your response in the environment variable OS_PASSWORD. It is important to note that this does require interactivity. It is possible to store a value directly in the script if you require a noninteractive operation, but you then need to be extremely cautious with the security and permissions of this file.

EC2 compatibility credentials can be downloaded by selecting Project, then Access & Security, then API Access to display the Download EC2 Credentials button. Click the button to generate a ZIP file with server x509 certificates and a shell script fragment. Create a new directory in a secure location because these are live credentials containing all the authentication information required to access your cloud identity, unlike the default user-openrc. Extract the ZIP file here. You should have cacert.pem, cert.pem, ec2rc.sh, and pk.pem. The ec2rc.sh is similar to this:

#!/bin/bash

NOVARC=$(readlink -f "${BASH_SOURCE:-${0}}" 2>/dev/null) ||\
NOVARC=$(python -c 'import os,sys; \
print os.path.abspath(os.path.realpath(sys.argv[1]))' "${BASH_SOURCE:-${0}}")
NOVA_KEY_DIR=${NOVARC%/*}
export EC2_ACCESS_KEY=df7f93ec47e84ef8a347bbb3d598449a
export EC2_SECRET_KEY=ead2fff9f8a344e489956deacd47e818
export EC2_URL=http://203.0.113.10:8773/services/Cloud
export EC2_USER_ID=42 # nova does not use user id, but bundling requires it
export EC2_PRIVATE_KEY=${NOVA_KEY_DIR}/pk.pem
export EC2_CERT=${NOVA_KEY_DIR}/cert.pem
export NOVA_CERT=${NOVA_KEY_DIR}/cacert.pem
export EUCALYPTUS_CERT=${NOVA_CERT} # euca-bundle-image seems to require this

alias ec2-bundle-image="ec2-bundle-image --cert $EC2_CERT --privatekey \
$EC2_PRIVATE_KEY --user 42 --ec2cert $NOVA_CERT"
alias ec2-upload-bundle="ec2-upload-bundle -a $EC2_ACCESS_KEY -s \
$EC2_SECRET_KEY --url $S3_URL --ec2cert $NOVA_CERT"

To put the EC2 credentials into your environment, source the ec2rc.sh file.

The command-line tools can be made to show the OpenStack API calls they make by passing the --debug flag to them. For example:

# nova --debug list

This example shows the HTTP requests from the client and the responses from the endpoints, which can be helpful in creating custom tools written to the OpenStack API.

Tip

Keyring Support enables you to securely save your OpenStack password in an encrypted file. This feature is disabled by default. To enable it, add the --os-cache flag or set the environment variable OS_CACHE=1. Configuring OS_CACHE causes the command-line tool to authenticate on each and every interaction with the cloud. This can assist with working around this scenario. However, it increases the time taken to run commands and also the load on the server.

As an administrator, you have a few ways to discover what your OpenStack cloud looks like simply by using the OpenStack tools available. This section gives you an idea of how to get an overview of your cloud, its shape, size, and current state.

First, you can discover what servers belong to your OpenStack cloud by running:

# nova-manage service list | sort

The output looks like the following:

Binary           Host              Zone Status  State Updated_At
nova-cert        cloud.example.com nova enabled  :-)  2013-02-25 19:32:38
nova-compute     c01.example.com   nova enabled  :-)  2013-02-25 19:32:35
nova-compute     c02.example.com   nova enabled  :-)  2013-02-25 19:32:32
nova-compute     c03.example.com   nova enabled  :-)  2013-02-25 19:32:36
nova-compute     c04.example.com   nova enabled  :-)  2013-02-25 19:32:32
nova-compute     c05.example.com   nova enabled  :-)  2013-02-25 19:32:41
nova-conductor   cloud.example.com nova enabled  :-)  2013-02-25 19:32:40
nova-consoleauth cloud.example.com nova enabled  :-)  2013-02-25 19:32:36
nova-network     cloud.example.com nova enabled  :-)  2013-02-25 19:32:32
nova-scheduler   cloud.example.com nova enabled  :-)  2013-02-25 19:32:33

The output shows that there are five compute nodes and one cloud controller. You see a smiley face, such as :-), which indicates that the services are up and running. If a service is no longer available, the :-) symbol changes to XXX. This is an indication that you should troubleshoot why the service is down.

If you are using cinder, run the following command to see a similar listing:

# cinder-manage host list | sort

host              zone
c01.example.com   nova
c02.example.com   nova
c03.example.com   nova
c04.example.com   nova
c05.example.com   nova
cloud.example.com nova

With these two tables, you now have a good overview of what servers and services make up your cloud.

You can also use the Identity Service (keystone) to see what services are available in your cloud as well as what endpoints have been configured for the services.

The following command requires you to have your shell environment configured with the proper administrative variables:

$ keystone catalog

Service: image
+-------------+----------------------------------------+
|   Property  |                 Value                  |
+-------------+----------------------------------------+
|   adminURL  | http://cloud.internal.example.com:9292 |
| internalURL | http://cloud.example.com:9292          |
|  publicURL  | http://cloud.example.com:9292          |
|    region   |               RegionOne                |
+-------------+----------------------------------------+

Service: identity
+-------------+----------------------------------------------+
|   Property  |                   Value                      |
+-------------+----------------------------------------------+
|   adminURL  | http://cloud.internal.example.com:35357/v2.0 |
| internalURL | http://cloud.example.com:5000/v2.0           |
|  publicURL  | http://cloud.example.com:5000/v2.0           |
|    region   |                 RegionOne                    |
+-------------+----------------------------------------------+

The preceding output has been truncated to show only two services. You will see one service block for each service that your cloud provides. Note how the endpoint domain can be different depending on the endpoint type. Different endpoint domains per type are not required, but this can be done for different reasons, such as endpoint privacy or network traffic segregation.

You can find the version of the Compute installation by using the nova-manage command:

# nova-manage version list

You can obtain extra information about virtual machines that are running—their CPU usage, the memory, the disk I/O or network I/O—per instance, by running the nova diagnostics command with a server ID:

$ nova diagnostics <serverID>

The output of this command varies depending on the hypervisor because hypervisors support different attributes. The following demonstrates the difference between the two most popular hypervisors. Here is example output when the hypervisor is Xen:

+----------------+-----------------+
|    Property    |      Value      |
+----------------+-----------------+
| cpu0           | 4.3627          |
| memory         | 1171088064.0000 |
| memory_target  | 1171088064.0000 |
| vbd_xvda_read  | 0.0             |
| vbd_xvda_write | 0.0             |
| vif_0_rx       | 3223.6870       |
| vif_0_tx       | 0.0             |
| vif_1_rx       | 104.4955        |
| vif_1_tx       | 0.0             |
+----------------+-----------------+

While the command should work with any hypervisor that is controlled through libvirt (e.g., KVM, QEMU, or LXC), it has been tested only with KVM. Here is example output when the hypervisor is KVM:

+------------------+------------+
| Property         | Value      |
+------------------+------------+
| cpu0_time        | 2870000000 |
| memory           | 524288     |
| vda_errors       | -1         |
| vda_read         | 262144     |
| vda_read_req     | 112        |
| vda_write        | 5606400    |
| vda_write_req    | 376        |
| vnet0_rx         | 63343      |
| vnet0_rx_drop    | 0          |
| vnet0_rx_errors  | 0          |
| vnet0_rx_packets | 431        |
| vnet0_tx         | 4905       |
| vnet0_tx_drop    | 0          |
| vnet0_tx_errors  | 0          |
| vnet0_tx_packets | 45         |
+------------------+------------+