The most important step is the pre-upgrade testing. If you are upgrading immediately after release of a new version, undiscovered bugs might hinder your progress. Some deployers prefer to wait until the first point release is announced. However, if you have a significant deployment, you might follow the development and testing of the release to ensure that bugs for your use cases are fixed.
Even if you have what seems to be a near-identical architecture as the one described in this guide, each OpenStack cloud is different. As a result, you must still test upgrades between versions in your environment. For this, you need an approximate clone of your environment.
However, that is not to say that it needs to be the same size or use identical hardware as the production environment—few of us have that luxury. It is important to consider the hardware and scale of the cloud that you are upgrading, but these tips can help you avoid that incredible cost:
- Use your own cloud
The simplest place to start testing the next version of OpenStack is by setting up a new environment inside your own cloud. This might seem odd—especially the double virtualization used in running compute nodes—but it's a sure way to very quickly test your configuration.
- Use a public cloud
Especially because your own cloud is unlikely to have sufficient space to scale test to the level of the entire cloud, consider using a public cloud to test the scalability limits of your cloud controller configuration. Most public clouds bill by the hour, which means it can be inexpensive to perform even a test with many nodes.
- Make another storage endpoint on the same system
If you use an external storage plug-in or shared file system with your cloud, in many cases, you can test whether it works by creating a second share or endpoint. This action enables you to test the system before entrusting the new version onto your storage.
- Watch the network
Even at smaller-scale testing, look for excess network packets to determine whether something is going horribly wrong in inter-component communication.
To set up the test environment, you can use one of several methods:
Do a full manual install by using the OpenStack Installation Guide for your platform. Review the final configuration files and installed packages.
Create a clone of your automated configuration infrastructure with changed package repository URLs.
Alter the configuration until it works.
Either approach is valid. Use the approach that matches your experience.
An upgrade pre-testing system is excellent for getting the configuration to work; however, it is important to note that the historical use of the system and differences in user interaction can affect the success of upgrades, too. We've seen experiences where database migrations encountered a bug (later fixed!) because of slight table differences between fresh Grizzly installs and those that migrated from Folsom to Grizzly.
Artificial scale testing can go only so far. After your cloud is upgraded, you must pay careful attention to the performance aspects of your cloud.
