OpenStack keystone zero downtime upgrade process newton to ocata
This blog post will show Keystone upgrade procedure from OpenStack Newton to Ocata release with zero-downtime.
In the case of doing this in production, please read release notes, ensure a proper configuration, do database backups and test the upgrade a thousand times.
Keystone upgrade will need to stop one node in order to use it as upgrade server. | In the case of a PoC this is not an issue, but in a production environment, Keystone loads may be intensive and stopping a node for a while may decrease other nodes performance more than expected. | For this reason I prefer orchestrate the upgrade from an external Docker container. With this method all nodes will be fully running almost all the time.
New container won\'t start any service, just will sync the database
schema with new Keystone version avoiding stop a node to orchestrate
the upgrade.
The Docker image is provided by OpenStack Kolla project, if already
using Kolla this upgrade won\'t be needed as kolla-ansible already
provide an upgrade method.
At the moment of writing of this blog, Ocata packages were not
released into stable repositories. For this reason I use DLRN
repositories.
If Ocata is released please do not use DLRN, use stable packages
instead.
Use stable Ocata Docker image if available with tag 4.0.x and will
avoid repository configuration and package upgrades.
NOTE: Upgrade may need more steps depending of your own
configuration, i.e, if using fernet token more steps are necessary
during the upgrade.
All Keystone nodes are behind HAproxy.
Prepare the upgrade
Start Keystone Docker container with host networking (needed to communicate with database nodes directly) and root user (needed to install packages).
Download Delorean CentOS trunk repositories
Disable Newton repository
Ensure Newton repository is not longer used by the system
Update all packages in the Docker container to bump keystone version to Ocata.
Configure keystone.conf file, this are my settings. Review you configuration and ensure all is correctly, otherwise may cause issues in the database. | An important option is default_domain_id, this value is for backward compatible with users created under default domain.
Check migrate version in the database. | As you will notice, contract/data_migrate/expand are in the same version
Before start upgrading the database schema, you will need add SUPER privileges in the database to keystone user or set log_bin_trust_function_creators to True. | In my opinion is safer set the value to True, I don\'t want keystone with SUPER privileges.
Now use Rally, tempest or some tool to test/benchmarch keystone service during upgrade. | If don\'t want to use one of those tools, just use this for command.
Start Upgrade
Check database status before upgrade using Doctor, this may raise issues in the configuration. Some of them may be ignored(Please, ensure is not an issue before ignoring). As example, I'm not using fernet tokens and errors appear about missing folder.
Remove obsoleted tokens
Now, expand the database schema to latest version, in keystone.log can see the status.
Check in the logs if some error is raised before jump to the next step.
After expand the database, migrate it to latest version.
Ensure there are not errors in Keystone logs.
Now, see migrate_version table, you will notice that expand and data_migrate are in the latest version, but contract still in the previous version.
Every Keystone node, one by one
Go to keystone nodes.
Stop Keystone services, in my case using wsgi inside Apache
Configure Ocata repositories as made in the Docker container.
Update packages, if you have Keystone sharing the node with other OpenStack service, do not update all packages as it will break other services.
Update only required packages.
Configure Keystone configuration file to the desired state. Your configuration may change.
Start Keystone service.
Finish Upgrade
After all the nodes are updated to latest version (please ensure all nodes are using latest packages, if not will fail).
Contract Keystone database schema.
Look at keystone.log for errors.
Now if we look at migrate_version table, will see that contract version is latest and match with the other version (Ensure all are in the same version).
This means the database upgrade has been successfully implemented.
Remove log_bin_trust_function_creators value.
After finish the upgrade, Rally tests should not have any error.
**If using HAproxy for load balance Keystone service, some errors may happen due a connection drop while stopping Keystone service and re-balance to other Keystone node. This can be avoided putting the node to update in Maintenance Mode in HAproxy backend.
Have to thank Keystone team in #openstack-keystone IRC channel for the help provided with a couple of issues.
Regards, Eduardo Gonzalez
Last updated
Was this helpful?