How to Perform an Upgrade from Havana to Icehouse—Ubuntu

For this section, we assume that you are starting with the architecture provided in the OpenStack Installation Guide and upgrading to the same architecture for Icehouse. All nodes should run Ubuntu 12.04 LTS with Linux kernel 3.11 and the latest Havana packages installed and operational. This section primarily addresses upgrading core OpenStack services such as Identity (keystone), Image Service (glance), Compute (nova), Networking (neutron), Block Storage (cinder), and the dashboard. The Networking upgrade includes conversion from the Open vSwitch (OVS) plug-in to the Modular Layer 2 (M2) plug-in. This section does not cover the upgrade process from Ubuntu 12.04 LTS to Ubuntu 14.04 LTS.

 Impact on Users

The upgrade process interrupts management of your environment, including the dashboard. If you properly prepare for this upgrade, tenant instances continue to operate normally. However, instances might experience intermittent network interruptions while the Networking service rebuilds virtual networking infrastructure.

 Upgrade Considerations

Always review the Icehouse Release Notes before you upgrade to learn about newly available features that you might want to enable and deprecated features that you should disable.

 Perform a Backup

Save the configuration files on all nodes:

# for i in keystone glance nova cinder neutron openstack-dashboard; \
  do mkdir $i-havana; \
  done
# for i in keystone glance nova cinder neutron openstack-dashboard; \
  do cp -r /etc/$i/* $i-havana/; \
  done
[Note]Note

You can modify this example script on each node to handle different services.

Back up all databases on the controller:

# mysqldump -u root -p --opt --add-drop-database --all-databases > havana-db-backup.sql
[Note]Note

Although not necessary, you should consider updating your MySQL server configuration as described in the MySQL controller setup section of the OpenStack Installation Guide.

 Manage Repositories

On all nodes, remove the repository for Havana packages and add the repository for Icehouse packages:

# apt-add-repository -r cloud-archive:havana
# apt-add-repository cloud-archive:icehouse
[Warning]Warning

Disable any automatic package updates.

 Upgrade Notes

  • Disable Compute file injection:

    Icehouse disables file injection by default per the Icehouse Release Notes.

    If you plan to deploy Icehouse in stages, you must disable file injection on all compute nodes that remain on Havana.

    Edit the /etc/nova/nova-compute.conf file:

    [libvirt]
    ...
    libvirt_inject_partition = -2
  • Convert from the OVS plug-in to the ML2 plug-in:

    You must convert the configuration for your environment contained in the /etc/neutron/neutron.conf and /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini files from OVS to ML2. For example, the OpenStack Installation Guide covers ML2 plug-in configuration using GRE tunnels.

    Keep the OVS plug-in packages and configuration files until you verify the upgrade.

 Upgrade the Controller Node

Upgrade packages on the controller node to Icehouse, as follows:

# apt-get update
# apt-get dist-upgrade
[Note]Note

Depending on your specific configuration, performing a dist-upgrade might restart services supplemental to your OpenStack environment. For example, if you use Open-iSCSI for Block Storage volumes and the upgrade includes a new open-scsi package, the package manager restarts Open-iSCSI services, which might cause the volumes for your users to be disconnected.

The package manager prompts you to update various configuration files. Reject these changes. The package manager appends .dpkg-dist to the newer versions of existing configuration files. You should consider adopting conventions associated with the newer configuration files and merging them with your existing configuration files after completing the upgrade process. You can find newer versions of existing configuration files with the following command:

# find /etc -name *.dpkg-dist

 Upgrade Each Service

The upgrade procedure for each service typically requires that you stop the service, run the database synchronization command to update the associated database, and start the service to apply the new configuration. You need administrator privileges for these procedures. Some services require additional steps.

  • OpenStack Identity:

    Update the configuration file for compatibility with Icehouse.

    Edit the /etc/keystone/keystone.conf file:

    Add the [database] section.

    Move the connection key from the [sql] section to the [database] section.

    Stop services, upgrade the database, and start services.

    # service keystone stop
    # keystone-manage token_flush
    # keystone-manage db_sync
    # service keystone start
  • OpenStack Image Service:

    Before upgrading the Image Service database, you must convert the character set for each table to UTF-8.

    Use the MySQL client to execute the following commands:

    # mysql -u root -p
    mysql> SET foreign_key_checks = 0;
    mysql> ALTER TABLE glance.image_locations CONVERT TO CHARACTER SET 'utf8';
    mysql> ALTER TABLE glance.image_members CONVERT TO CHARACTER SET 'utf8';
    mysql> ALTER TABLE glance.image_properties CONVERT TO CHARACTER SET 'utf8';
    mysql> ALTER TABLE glance.image_tags CONVERT TO CHARACTER SET 'utf8';
    mysql> ALTER TABLE glance.images CONVERT TO CHARACTER SET 'utf8';
    mysql> ALTER TABLE glance.migrate_version CONVERT TO CHARACTER SET 'utf8';
    mysql> SET foreign_key_checks = 1;
    mysql> exit
    [Note]Note

    Your environment might contain different or additional tables that you must also convert to UTF-8 by using similar commands.

    Update the configuration for compatibility with Icehouse.

    Edit the /etc/glance/glance-api.conf and /etc/glance/glance-registry.conf files:

    Add the [database] section.

    Rename the sql_connection key to connection and move it to the [database] section.

    Edit the /etc/glance/glance-api.conf file:

    Add RabbitMQ message broker keys to the [DEFAULT] section.

    Replace RABBIT_PASS with the password you chose for the guest account in RabbitMQ.

    [DEFAULT]
    ...
    rpc_backend = rabbit
    rabbit_host = controller
    rabbit_password = RABBIT_PASS

    Stop services, upgrade the database, and start services:

    # service glance-api stop
    # service glance-registry stop
    # glance-manage db_sync
    # service glance-api start
    # service glance-registry start
  • OpenStack Compute:

    Update the configuration for compatibility with Icehouse.

    Edit the /etc/nova/nova.conf file:

    Change the rpc_backend key from nova.rpc.impl_kombu to rabbit.

    Edit the /etc/nova/api-paste.ini file:

    Comment out or remove any keys in the [filter:authtoken] section beneath the paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory statement.

    Stop services, upgrade the database, and start services.

    # service nova-api stop
    # service nova-scheduler stop
    # service nova-conductor stop
    # service nova-cert stop
    # service nova-consoleauth stop
    # service nova-novncproxy stop
    # nova-manage db sync
    # service nova-api start
    # service nova-scheduler start
    # service nova-conductor start
    # service nova-cert start
    # service nova-consoleauth start
    # service nova-novncproxy start
  • OpenStack Networking:

    Populate the /etc/neutron/plugins/ml2/ml2_conf.ini file with the equivalent configuration for your environment. Do not edit the /etc/neutron/neutron.conf file until after the conversion steps.

    Stop services, upgrade the database, and perform the conversion from OVS to ML2.

    Replace NEUTRON_DBPASS with the password you chose for the database.

    [Warning]Warning

    We highly recommend that you perform a database backup prior to executing the following commands as the conversion script cannot roll back.

    # service neutron-server stop
    # neutron-db-manage --config-file /etc/neutron/neutron.conf \
      --config-file /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini stamp havana
    # neutron-db-manage --config-file /etc/neutron/neutron.conf \
      --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade icehouse
    # python -m neutron.db.migration.migrate_to_ml2 openvswitch \
      mysql://neutron:NEUTRON_DBPASS@controller/neutron

    Edit the /etc/neutron/neutron.conf file to use the ML2 plug-in and enable network change notifications:

    Replace SERVICE_TENANT_ID with the service tenant identifier (id) in the Identity service and NOVA_PASS with the password you chose for the nova user in the Identity service.

    [DEFAULT]
    ...
    core_plugin = ml2
    service_plugins = router
    ...
    notify_nova_on_port_status_changes = True
    notify_nova_on_port_data_changes = True
    nova_url = http://controller:8774/v2
    nova_admin_username = nova
    nova_admin_tenant_id = SERVICE_TENANT_ID
    nova_admin_password = NOVA_PASS
    nova_admin_auth_url = http://controller:35357/v2.0
    

    Start Networking services.

    # service neutron-server start
  • OpenStack Block Storage:

    Stop services, upgrade the database, and start services.

    # service cinder-api stop
    # service cinder-volume stop
    # service cinder-scheduler stop
    # cinder-manage db sync
    # service cinder-api start
    # service cinder-volume start
    # service cinder-scheduler start
  • Dashboard:

    Update the configuration for compatibility with Icehouse.

    Edit the /etc/openstack-dashboard/local_settings.py file:

    Change the OPENSTACK_KEYSTONE_DEFAULT_ROLE key from "Member" to "_member_".

    Restart Dashboard services.

    # service apache2 restart

The controller node update is complete. Now you can upgrade the remaining nodes.

 Upgrade the Network Node

Upgrade packages on the network node to Icehouse:

[Note]Note

Make sure you have removed the repository for Havana packages and added the repository for Icehouse packages.

# apt-get update
# apt-get dist-upgrade

Edit the /etc/neutron/neutron.conf file to use the ML2 plug-in:

[DEFAULT]
core_plugin = ml2
service_plugins = router

Populate the /etc/neutron/plugins/ml2/ml2_conf.ini file with the equivalent configuration for your environment.

Clean the active OVS configuration:

# service neutron-ovs-cleanup restart

Restart Networking services:

# service neutron-dhcp-agent restart
# service neutron-l3-agent restart
# service neutron-metadata-agent restart
# service neutron-plugin-openvswitch-agent restart

 Upgrade the Compute Nodes

Upgrade packages on the compute nodes to Icehouse:

[Note]Note

Make sure you have removed the repository for Havana packages and added the repository for Icehouse packages.

# apt-get update
# apt-get dist-upgrade

Edit the /etc/neutron/neutron.conf file to use the ML2 plug-in:

[DEFAULT]
core_plugin = ml2
service_plugins = router

Populate the /etc/neutron/plugins/ml2/ml2_conf.ini file with the equivalent configuration for your environment.

Clean the active OVS configuration:

# service neutron-ovs-cleanup restart

Restart Networking services:

# service neutron-plugin-openvswitch-agent restart

Restart Compute services:

# service nova-compute restart

 Upgrade the Storage Nodes

Upgrade packages on the storage nodes to Icehouse:

[Note]Note

Make sure you have removed the repository for Havana packages and added the repository for Icehouse packages.

# apt-get update
# apt-get dist-upgrade

Restart Block Storage services.

# service cinder-volume restart
Questions? Discuss on ask.openstack.org
Found an error? Report a bug against this page


loading table of contents...