Long-Term-Supported (LTS) ReleasesOct 13, 2021
The dotCMS Long-Term Supported (LTS) release process was started in December 2020 to help us improve both the reliability of our products and our level of support to all our customers. It's been a clear success, with customers adopting LTS releases seeing a fairly dramatic reduction in both number of support calls logged and support call resolution times. In fact, the process has been so successful that even customers who haven’t adopted LTS releases have seen an improvement in service levels.
In this blog, we're going to discuss what an LTS release is, explain why we implemented them, and show you the benefits they offer - including some data to back that up.
What is an LTS Release?
Let’s start with the basics - what is an LTS release?
dotCMS has two different kinds of releases, Agile releases and Long-Term-Supported (LTS) releases. Agile releases are regular releases which are part of our CI/CD process to deliver both fixes and new features continually. Every 6-9 months, dotCMS will designate a previous Agile release as an LTS release. Once a release has been designated as an LTS releases, we perform continual updates to the release to fix known issues, improve the reliability, and enable the release to be supported for a longer period.
Each type of release has advantages and disadvantages.
Agile releases contain all of the latest fixes, new features, and enhancements. We deliver a new Agile release every month, each one has improvements and new features. However, they’re only supported for 12 months after release, and since they contain new features, they may have new undiscovered issues. Also, if you need a fix for an issue in an Agile release, then instead of patching your release you’ll need to upgrade to a newer Agile release which contains the fix for that issue.
LTS releases are built to maximize stability and reliability, and since they continually have new fixes, the reliability keeps improving over time. They receive regular patch releases (without the need for a full upgrade), and they're supported for 18-24 months from the date the release was designated as an LTS release. However, they won’t have the very latest new features and enhancements; the newest LTS release may be anywhere from 2 to 8 months behind the features in the latest Agile release.
Here's a summary of the differences between Agile and LTS releases:
|Agile Release||LTS Patch Release|
|Includes New Features||Yes||No*|
|Includes New Bug Fixes||All||Many|
|Post-Release Bug Fixes||No|
(Requires full upgrade)
(Wait for next patch release)
|Release Frequency||Monthly||6-12 weeks|
|Full Support Period||12 months||18-24 months|
|Data Migration Needed||Possible||No|
|Upgrade Time||Days-weeks||10 min/node|
* LTS patch releases may occasionally include small, low-risk enhancements
Which Release Should You Choose?
You can choose to run on either an Agile or LTS release, depending on your needs and your plans.
If you’d like to use a brand new feature now, or want to take advantage of all the latest new features and enhancements as they’re released, you’ll want to be on an Agile Release.
If you place a higher priority on consistency and reliability, or you’d like to reduce your frequency of upgrades, you’ll want to be on an LTS release.
For dotCMS Cloud customers there’s also a third option, which combines the best of both worlds: you can run you main environments on an LTS version, while maintaining a separate “evergreen” environment which is continually updated to the latest Agile release.
This enables you to receive all the advantages of LTS releases for your day-to-day work, but your development team can continually move forward with the latest Agile releases and features. So, when the time comes to upgrade to a newer LTS release, you’ve already had a chance to integrate the latest new features, and the majority of your data migration has already been performed.
When we created the LTS release process, we had one simple goal in mind: to provide a higher level of support to customers, without an increase in cost. And that’s exactly what LTS releases have accomplished.
Customers on Agile releases still receive the same level of support as they always have, including full helpdesk support with all the same terms and SLAs as before, and the possibility that hotfixes will be delivered for critical issues.
In addition, customers that move to LTS releases may receive an even higher level of support, including all of the following:
|Improved Stability and Reliability||Stability and reliability improves over the lifetime of each LTS release|
|Longer Support Period||You can run the same release without upgrading for 18-24 months|
|Proactive Prevention||A large number of issues are fixed before you experience them|
|Simplified Patching||A standardized patch process ensures fast and low-risk patches|
|Improved Upgrade Path||Upgrades between LTS releases follow a known and well-tested path.|
|Improved Support||The dotCMS support team has more time to help with deeper issues and questions|
Improved Patches and Upgrades
One benefit we'd like to highlight is the improvement LTS releases offer for patches and upgrades.
dotCMS is a platform, and every single dotCMS customer implements their sites in different ways, using different features of dotCMS, writing their own custom code, and integrating different third party applications and libraries. Because of this, no two upgrades are the same.
So, although many upgrades go relatively smoothly, upgrading from one dotCMS version to another always requires some work to ensure that your specific implementation works properly with the new version. This difference in implementations impacts hotfixes as well; even the best tested hotfixes can’t be tested against every possible implementation detail for all dotCMS versions it may apply to.
Thankfully, the LTS process enables us to improve both the patch and upgrade processes for customers on LTS releases.
Improved Patch Process
The LTS patch process has been streamlined to enable us to deliver fast, reliable, and more frequent patches for LTS releases.
Since fixes are delivered as part of a new full patch release, and since patch releases are delivered on a regular bases, it virtually eliminates the need for hotfixes for LTS releases. This also simplifies system configuration, and eliminates potential conflicts between different hotfixes.
We don’t include all possible fixes in LTS patch releases. Instead, we choose which fixes to install, based on both the impact of the issue and on the likelihood that customers will experience them. As part of this decision process, we ensure that only fixes with low impact and low risk to existing systems are included; if a fix has the potential to cause new issues, it will intentionally be excluded.
Since the patch applies to only a single version, and only low-impact fixes are included, we can perform much more extensive testing. In other words, instead of having to test wide, we can now test deep.
No data migration
In addition, we exclude any fixes that would require data migration. This means that you can install patch releases without needing to change any of your data.
Because data migration and other changes aren't needed, a patch install can be performed with a simple “in-place” upgrade for binary installs, and containerized instances can be upgraded by simply changing the version label and restarting the container.
Minimal or no downtime
All of this means that if you have a clustered instance, you can install a patch release with no downtime; each node can be upgraded and restarted separately without your environment going off-line. And even for non-clustered instances, the downtime is typically ten minutes or less.
The following compares the traditional hotfix process with the new LTS patch release process:
|Hotfixes||LTS Patch Releases|
|Fixes Included||Very few|
(critical and security fixes only)
|User Base||Very limited||Large|
|Testing||Necessarily limited||Full testing cycle|
|Documentation||Necessarily limited||Full changelogs|
|Availability||By request only||Freely downloadable or|
* LTS Patch releases are automatically applied for all dotCMS Cloud customers on LTS releases.
The most important difference is that the hotfix model is necessarily reactive; fixes are only generated and installed after a problem is encountered (and even then only for critical issues). But LTS patch releases are proactive; fixes are continually integrated, and patch releases are automatically applied for all dotCMS Cloud customers.
Improved Upgrade Process
Another benefit for customers on LTS releases is that the process of performing a full upgrade between two different LTS releases is significantly improved over upgrades between major Agile releases.
Since there are many more Agile releases, when you upgrade between two Agile releases, there may be only a handful of customers who have upgraded between those two specific releases. So, it's less likely that another customer has performed the same upgrade with the same features that your implementation uses.
However, when you upgrade between LTS releases, there's almost always a large number of customers who have upgraded between those exact two releases, so the upgrade path is well known and well tested. In addition, since most customers will upgrade to a new LTS release after some patch releases have already been released for the later LTS release, it's less likely that you'll run into any issues during the upgrade.
Ultimately, this all means that upgrades between LTS versions are both faster and lower risk.
LTS Release Limitations
LTS releases do have some limitations compared to Agile releases.
Receive latest features and enhancements less frequently
As a general rule, new features and enhancements are not included in LTS patch releases. So, if you want a new feature or enhancement, you may need to upgrade to a newer LTS version - or possibly even an Agile release - to have access to that feature.
Not all fixes are included
Just as with hotfixes, there is no guarantee that any specific fix will be included in a given LTS release. While we make every effort to include fixes for issues which affect customers, we prioritize those that have the highest impact, and affect the most customers.
In addition, since the most important goal of LTS releases is to provide stability and reliability, high impact fixes (those which have the potential to cause new issues) are intentionally excluded. This also means that sometimes even if we plan to include a fix, we may need to leave it out if, while backporting the fix, we find that it presents risk that we didn't anticipate.
No fixed schedule
Although we'll typically deliver a new patch release for each LTS version every 6-12 weeks, there is no guaranteed schedule. The schedule is determined by a combination of need and resources, and at times releases may not happen for several months. Longer patch release cycles are especially likely toward the end of an LTS version's lifecycle (when it's both very diverged from the latest Agile code, and also already very stable).
So, now that we’ve discussed the benefits, we’d like to show you how much progress we’ve made so far. As of the time of this writing:
3 releases have been designated as LTS
- 5.2.8 LTS
- 5.3.8 LTS
- 21.06 LTS
81% of dotCMS Cloud customers are on LTS releases
About half of these are currently on the 5.3.8 LTS release, and many of these are planning for migration to 21.06 LTS.
In the 18.104.22.168 LTS release alone, 103 fixes have been included
The vast majority of these fixes were delivered to customers before they ever experienced the issues themselves.
So, LTS releases have been widely adopted, and a large number of fixes have been delivered to customers - far more than would have been possible with the traditional hotfix process.
It’s easy to say that something will work better, but the only thing that really matters is - does it actually work?
And the answer is that, although our goal was to be able to improve service levels for customers who adopted LTS releases, the process has been so successful that we’ve been able to improve support for all of our customers - even those on Agile releases.
So, how do we know that? As an engineering-driven company, we believe that when it comes to showing that something works, it’s hard to beat looking at the numbers.
Number of support requests reduced 32%
If the LTS release process is really improving the stability and reliability, then we should see a reduced number of support requests. And that's exactly what we've seen. And this has helped customers on both Agile and LTS release, because the fewer support requests we receive, the more time our support staff has to help all of our customers, regardless of what releases they’re running.
The following chart shows the total number of support requests received every month for the past two years. The green vertical line shows when the LTS process was started (with the designation of the first LTS releases):
For the past two years, the number of support requests we received each month has been within a consistent range. But since the implementation of the LTS process, the number of requests has dropped to historically low levels.
One important thing to note is that these numbers are the total support requests for all customers (on both Agile and LTS releases). But the number of support requests just from customers on LTS releases has dropped by almost 50%. In other words, customers who’ve moved to LTS releases are now submitting half as many support requests as they did before they moved to an LTS release.
Support calls resolved 61% faster
If the LTS release process is improving support, the other place we expect to see a change is in the time it takes us to resolve the submitted support requests to the customers' satisfaction. So, let's take a look at that.
The following graph shows the average resolution time for support requests for the past two years:
Only a few months after starting the LTS process, the speed at which we resolved support requests was faster than at any time in the previous 18 months. And just over half a year later, we’re resolving requests more than twice as fast as we ever did in the past.
Greatly improved response times
To round out the numbers, let's take a quick look at response times. The the combination of the LTS release process and other improvements in our support processes now enables us to respond to support requests much faster than in the past. Initial response times (the time it takes us to respond after a request is first submitted) has been cut by 56%, and the time it takes for us to respond to each and every customer comment has dropped by 24%.
The Long-Term-Supported (LTS) release process has successfully allowed us to improve stability and reliability, almost double release support periods, and simplify both patching and upgrades for customers on LTS releases. In addition, it's enabled us to improve the support we provide to all of our customers.
As an engineer I tend to understate things, but it’s fairly obvious from the data that the results have been dramatic. We’re excited about how much we’ve been able to help our customers, and looking forward to sharing with you more great changes as we keep moving forward!