Upgrades involve complex operations and can fail. This section provides guidance for rolling back to a previous release of OpenStack. Although only tested on Ubuntu, other distributions follow a similar procedure.
In this section, we consider only the most immediate case: you have taken down production management services in preparation for an upgrade, completed part of the upgrade process, discovered one or more problems not encountered during testing, and you must roll back your environment to the original "known good" state. Make sure that you did not make any state changes after attempting the upgrade process: no new instances, networks, storage volumes, and so on.
Within this scope, you must complete these steps to successfully roll back your environment:
Roll back configuration files.
Roll back databases.
Roll back packages.
The upgrade instructions provided in earlier sections ensure that you have proper backups of your databases and configuration files. Read through this section carefully and verify that you have the requisite backups to restore. Rolling back upgrades is a tricky process because distributions tend to put much more effort into testing upgrades than downgrades. Broken downgrades often take significantly more effort to troubleshoot and, hopefully, resolve than broken upgrades. Only you can weigh the risks of trying to push a failed upgrade forward versus rolling it back. Generally, consider rolling back as the very last option.
The following steps described for Ubuntu have worked on at least one production environment, but they might not work for all environments.
To perform the rollback from Havana to Grizzly
Stop all OpenStack services.
Copy contents of configuration backup directories
/etc/<service>.grizzlythat you created during the upgrade process back to/etc/<service>:Restore databases from the
grizzly-db-backup.sqlbackup file that you created with the mysqldump command during the upgrade process:# mysql -u root -p < grizzly-db-backup.sql
If you created this backup by using the
--add-drop-databaseflag as instructed, you can proceed to the next step. If you omitted this flag, MySQL reverts all tables that existed in Grizzly, but does not drop any tables created during the database migration for Havana. In this case, you must manually determine which tables to drop, and drop them to prevent issues with your next upgrade attempt.Downgrade OpenStack packages.
![[Warning]](../common/images/admon/warning.png)
Warning Downgrading packages is by far the most complicated step; it is highly dependent on the distribution and the overall administration of the system.
Determine which OpenStack packages are installed on your system. Use the dpkg --get-selections command. Filter for OpenStack packages, filter again to omit packages explicitly marked in the
deinstallstate, and save the final output to a file. For example, the following command covers a controller node with keystone, glance, nova, neutron, and cinder:# dpkg --get-selections | grep -e keystone -e glance -e nova -e neutron \ -e cinder | grep -v deinstall | tee openstack-selections cinder-api install cinder-common install cinder-scheduler install cinder-volume install glance install glance-api install glance-common install glance-registry install neutron-common install neutron-dhcp-agent install neutron-l3-agent install neutron-lbaas-agent install neutron-metadata-agent install neutron-plugin-openvswitch install neutron-plugin-openvswitch-agent install neutron-server install nova-api install nova-cert install nova-common install nova-conductor install nova-consoleauth install nova-novncproxy install nova-objectstore install nova-scheduler install python-cinder install python-cinderclient install python-glance install python-glanceclient install python-keystone install python-keystoneclient install python-neutron install python-neutronclient install python-nova install python-novaclient install
![[Note]](../common/images/admon/note.png)
Note Depending on the type of server, the contents and order of your package list might vary from this example.
You can determine the package versions available for reversion by using the apt-cache policy command. If you removed the Grizzly repositories, you must first reinstall them and run apt-get update:
# apt-cache policy nova-common nova-common: Installed: 1:2013.2-0ubuntu1~cloud0 Candidate: 1:2013.2-0ubuntu1~cloud0 Version table: *** 1:2013.2-0ubuntu1~cloud0 0 500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/havana/main amd64 Packages 100 /var/lib/dpkg/status 1:2013.1.4-0ubuntu1~cloud0 0 500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/grizzly/main amd64 Packages 2012.1.3+stable-20130423-e52e6912-0ubuntu1.2 0 500 http://us.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu/ precise-security/main amd64 Packages 2012.1-0ubuntu2 0 500 http://us.archive.ubuntu.com/ubuntu/ precise/main amd64 PackagesThis tells us the currently installed version of the package, newest candidate version, and all versions along with the repository that contains each version. Look for the appropriate Grizzly version—
1:2013.1.4-0ubuntu1~cloud0in this case. The process of manually picking through this list of packages is rather tedious and prone to errors. You should consider using the following script to help with this process:# for i in `cut -f 1 openstack-selections | sed 's/neutron/quantum/;'`; do echo -n $i ;apt-cache policy $i | grep -B 1 grizzly | grep -v Packages | awk '{print "="$1}';done | tr '\n' ' ' | tee openstack-grizzly-versions cinder-api=1:2013.1.4-0ubuntu1~cloud0 cinder-common=1:2013.1.4-0ubuntu1~cloud0 cinder-scheduler=1:2013.1.4-0ubuntu1~cloud0 cinder-volume=1:2013.1.4-0ubuntu1~cloud0 glance=1:2013.1.4-0ubuntu1~cloud0 glance-api=1:2013.1.4-0ubuntu1~cloud0 glance-common=1:2013.1.4-0ubuntu1~cloud0 glance-registry=1:2013.1.4-0ubuntu1~cloud0 quantum-common=1:2013.1.4-0ubuntu1~cloud0 quantum-dhcp-agent=1:2013.1.4-0ubuntu1~cloud0 quantum-l3-agent=1:2013.1.4-0ubuntu1~cloud0 quantum-lbaas-agent=1:2013.1.4-0ubuntu1~cloud0 quantum-metadata-agent=1:2013.1.4-0ubuntu1~cloud0 quantum-plugin-openvswitch=1:2013.1.4-0ubuntu1~cloud0 quantum-plugin-openvswitch-agent=1:2013.1.4-0ubuntu1~cloud0 quantum-server=1:2013.1.4-0ubuntu1~cloud0 nova-api=1:2013.1.4-0ubuntu1~cloud0 nova-cert=1:2013.1.4-0ubuntu1~cloud0 nova-common=1:2013.1.4-0ubuntu1~cloud0 nova-conductor=1:2013.1.4-0ubuntu1~cloud0 nova-consoleauth=1:2013.1.4-0ubuntu1~cloud0 nova-novncproxy=1:2013.1.4-0ubuntu1~cloud0 nova-objectstore=1:2013.1.4-0ubuntu1~cloud0 nova-scheduler=1:2013.1.4-0ubuntu1~cloud0 python-cinder=1:2013.1.4-0ubuntu1~cloud0 python-cinderclient=1:1.0.3-0ubuntu1~cloud0 python-glance=1:2013.1.4-0ubuntu1~cloud0 python-glanceclient=1:0.9.0-0ubuntu1.2~cloud0 python-quantum=1:2013.1.4-0ubuntu1~cloud0 python-quantumclient=1:2.2.0-0ubuntu1~cloud0 python-nova=1:2013.1.4-0ubuntu1~cloud0 python-novaclient=1:2.13.0-0ubuntu1~cloud0Note If you decide to continue this step manually, don't forget to change
neutrontoquantumwhere applicable.Use the apt-get install command to install specific versions of each package by specifying
<package-name>=<version>. The script in the previous step conveniently created a list ofpackage=versionpairs for you:# apt-get install `cat openstack-grizzly-versions`
This step completes the rollback procedure. You should remove the Havana repository and run apt-get update to prevent accidental upgrades until you solve whatever issue caused you to roll back your environment.
