Our way to Managed OpenShift 4

Red Hat OpenShift 4

This summer Red Hat released OpenShift 4. At first glance, the new major version is a continuous development of OpenShift 3 with relatively manageable changes for the user. But if you look under the hood, you will quickly see a completely revised OpenShift. The blogpost of Benjamin Affolter on the APPUiO blog examines the changes of OpenShift 4 and describes them in detail.

With the following article we would like to take a look behind the scenes of our Managed OpenShift offering and explain what we have to do to be able to offer our Managed Service with OpenShift 4.

Advantages of OpenShift 4

Red Hat promises the following improvements with version 4 of OpenShift:

• New Installer
• Fully automated operations, maintenance and configuration using Operators
• Integration of Operator Hub
• Current versions of Kubernetes

To fully understand the benefits and also the implications, we need to take a step back and take a look at OpenShift 3.

Managed OpenShift 3 – what’s included?

For better understanding, you can find a short overview of what our Managed OpenShift 3 service includes so far (not exhaustive):

• Architecture engineering and setup of the OpenShift Cluster on almost any infrastructure (Cloud, On-Premise)
• Monitoring of all cluster-relevant components to ensure operation
• Regular backup of the cluster configuration incl. ensuring the integrity of the backup
• Weekly maintenance of all systems, application of software patches and configuration improvements on all clusters
• Automation of all work with Ansible (configuration, maintenance, updates, upgrades, installation, sanity checks and much more)
• Integration into our central customer portal for an overview of the status of the cluster and other functions
• Extensive dashboards in Grafana
• Close cooperation with Red Hat Support for solving bugs in OpenShift, among others
• Maintenance of various internal lab clusters to test changes to productive clusters
• Provision of persistent storage using Gluster
• Management and maintenance of the operating system Red Hat Enterprise Linux for the OpenShift masters and nodes
• Training of system engineers to run OpenShift

All these listed points have been developed since the very first version of OpenShift 3 and are developed daily by our VSHNeers.

Status Quo VSHN Systems

From a technical point of view, our current system landscape looks something like this (brief overview):

• Puppet for the local operating system management of all VMs (system configuration, maintenance of the defined state) and inventory of all systems and services.
• Icinga2 for monitoring all operating system parameters within the VM, but also very extensive checks of all OpenShift cluster components. Icinga2 is configured and orchestrated by Puppet.
• Ansible for installation and configuration of OpenShift, for regular maintenance and for much more
• BURP for consistent data backups incl. cluster configuration, configured and orchestrated by Puppet
• Gluster for persistent storage, managed by Ansible

Over the years, countless Ansible Playbooks have accumulated and all our knowledge and automation has gone into these Playbooks. We maintain our own fork from the official OpenShift Ansible Repository to be able to react quickly to any bugs. We regularly keep this fork up to date with upstream.
Puppet not only takes care of the local operating system configuration, but also controls many important components such as the monitoring and backup system. In addition, the PuppetDB provides us with an up-to-date inventory of all systems managed by us, including detailed version information of the installed components. This is also integrated in our customer portal and is used for automatic billing of our managed services.
The monitoring plugins we developed for Icinga2 cover almost every problem we have discovered with OpenShift and notify us if there is anything wrong with the cluster or one of its components.
Our system documentation and OpenShift operation guide include several dozen Wiki articles.

Managed OpenShift 4 – what is there to do for VSHN?

From a system engineering point of view, OpenShift 4 is a completely new product. For VSHN this means that we have to completely redevelop a large part of the above points.
A few examples:

• The installation and configuration of OpenShift 4 is no longer based on Ansible, but on a separate installer (which uses Terraform in the background) and the configuration is done by In-Cluster Operators. Our Ansible Playbooks for OpenShift 3 can for the most part no longer be used for OpenShift 4.
• The operating system is no longer Red Hat Enterprise Linux, but Red Hat CoreOS, which behaves completely different. Puppet cannot be used anymore and as described above we have to find other ways to inventory, orchestrate and bill the surrounding systems.
• Our monitoring plugins for Icinga2 are no longer compatible with OpenShift 4 and the monitoring concept with Icinga2 no longer fits the platform’s revised architecture. For us this means a new development of our monitoring concept.
• The backup system BURP can no longer be used in its current form, a new backup system has to be developed.

This is not an exhaustive list, there are many more details in our system landscape that need to be adapted.

The path to production

For us as a Managed Service Provider, stability and scalability are the most important points which are non-negotiable. This means that we have to take the necessary time to learn all the changes and peculiarities for a productive operation of OpenShift 4. The adaptation and development of the necessary tools and processes for the operation of dozens of clusters requires a lot of time and engineering effort. However, we started early and have already gained some experience with OpenShift 4. The experience gives us great confidence that OpenShift 4 can deliver on its promises of greatly simplified operation.
The current version OpenShift 4.1 also has some limitations. Here is a small selection of what we noticed:

• No support for proxies
• AWS and VMware are the only supported IaaS providers with OpenShift 4.1 (current version at the time of this article)
• Installation on unsupported and non-cloud platforms is very fragile
• Container storage only via CSI

Many IaaS providers are not yet ready for OpenShift 4, but we are in close contact with our IaaS & Cloud partners like cloudscale.ch, Exoscale, Swisscom and AWS, to ensure compatibility so that we can continue to offer a smooth operation with OpenShift 4.
OpenShift 4.1 reminds us partly of the early days of OpenShift 3, when it took some time until OpenShift 3 was ready for production.
But we are very confident that the open issues can be solved and we are looking forward to the 4th generation of Red Hat OpenShift!

More Info

Our friends from Adfinis SyGroup have described their first experiences with OpenShift 4 in their blog post “OpenShift 4 – Learnings from the first productive environment” which fits very well with our observations.
If you want to learn more about OpenShift and Kubernetes, we recommend reading our article “What is a Kubernetes Distribution and what are the differences between Kubernetes and OpenShift” or have a look at the impressions of Red Hat Forum Zurich 2019, where APPUiO was a sponsor and where we had a booth on-site.

APPUiO – Swiss Container Platform

With APPUiO.ch we have created a Swiss Container Platform based on Red Hat OpenShift on which we offer Managed Services as a PaaS solution (Platform-as-a-Service) on any infrastructure: public, dedicated, private and on-premises. Based on proven Open Source concepts like Docker and Kubernetes you develop, operate and scale an application according to your needs. With APPUiO, your applications can run on public clouds as well as in-house. The platform was originally developed in 2015 by the two IT specialists Puzzle ITC and VSHN AG for the professionalization of their internal IT. Today, APPUiO is used productively by many customers and is supported by a strong community.

How can we help?

With our experience in operating OpenShift clusters around the world, we offer managed OpenShift clusters on almost any public, private or on-premise cloud. We are happy to help with evaluation, integration and operation and support with our many years of Kubernetes experience. Contact us, subscribe to our newsletter and follow us on Twitter (@vshn_ch and @APPUiO) to keep up with the latest news and have a look at our Services.
We look forward to your feedback!