Blogs
John Michael Thomas
Director of Support

Long-Term-Supported (LTS) Releases

Oct 13, 2021
By: John Michael Thomas

The dotCMS Long-Term Supported (LTS) release process was started in December 2020 to help us improve both the reliability of our products and our level of support to all our customers. It's been a clear success, with customers adopting LTS releases seeing a fairly dramatic reduction in both number of support calls logged and support call resolution times. In fact, the process has been so successful that even customers who haven’t adopted LTS releases have seen an improvement in service levels.

In this blog, we're going to discuss what an LTS release is, explain why we implemented them, and show you the benefits they offer - including some data to back that up.

What is an LTS Release?

Let’s start with the basics - what is an LTS release?

dotCMS has two different kinds of releases, Agile releases and Long-Term-Supported (LTS) releases. Agile releases are regular releases which are part of our CI/CD process to deliver both fixes and new features continually. Every 6-9 months, dotCMS will designate a previous Agile release as an LTS release. Once a release has been designated as an LTS releases, we perform continual updates to the release to fix known issues, improve the reliability, and enable the release to be supported for a longer period.

Each type of release has advantages and disadvantages.

Agile releases contain all of the latest fixes, new features, and enhancements. We deliver a new Agile release every month, each one has improvements and new features. However, they’re only supported for 12 months after release, and since they contain new features, they may have new undiscovered issues. Also, if you need a fix for an issue in an Agile release, then instead of patching your release you’ll need to upgrade to a newer Agile release which contains the fix for that issue.

LTS releases are built to maximize stability and reliability, and since they continually have new fixes, the reliability keeps improving over time. They receive regular patch releases (without the need for a full upgrade), and they're supported for 18-24 months from the date the release was designated as an LTS release. However, they won’t have the very latest new features and enhancements; the newest LTS release may be anywhere from 2 to 8 months behind the features in the latest Agile release.

Here's a summary of the differences between Agile and LTS releases:

Agile ReleaseLTS Patch Release
Includes New FeaturesYesNo*
Includes New Bug FixesAllMany
Post-Release Bug FixesNo
(Requires full upgrade)
Yes
(Wait for next patch release)
Release FrequencyMonthly6-12 weeks
Full Support Period12 months18-24 months
User BaseSmallLarge
Data Migration NeededPossibleNo
Upgrade TimeDays-weeks10 min/node

* LTS patch releases may occasionally include small, low-risk enhancements

Which Release Should You Choose?

You can choose to run on either an Agile or LTS release, depending on your needs and your plans.

If you’d like to use a brand new feature now, or want to take advantage of all the latest new features and enhancements as they’re released, you’ll want to be on an Agile Release.

If you place a higher priority on consistency and reliability, or you’d like to reduce your frequency of upgrades, you’ll want to be on an LTS release.

Evergreen Environments

For dotCMS Cloud customers there’s also a third option, which combines the best of both worlds: you can run you main environments on an LTS version, while maintaining a separate “evergreen” environment which is continually updated to the latest Agile release.

This enables you to receive all the advantages of LTS releases for your day-to-day work, but your development team can continually move forward with the latest Agile releases and features. So, when the time comes to upgrade to a newer LTS release, you’ve already had a chance to integrate the latest new features, and the majority of your data migration has already been performed.

Goals

When we created the LTS release process, we had one simple goal in mind: to provide a higher level of support to customers, without an increase in cost. And that’s exactly what LTS releases have accomplished.

Customers on Agile releases still receive the same level of support as they always have, including full helpdesk support with all the same terms and SLAs as before, and the possibility that hotfixes will be delivered for critical issues.

In addition, customers that move to LTS releases may receive an even higher level of support, including all of the following:

BenefitResult
Improved Stability and ReliabilityStability and reliability improves over the lifetime of each LTS release
Longer Support PeriodYou can run the same release without upgrading for 18-24 months
Proactive PreventionA large number of issues are fixed before you experience them
Simplified PatchingA standardized patch process ensures fast and low-risk patches
Improved Upgrade PathUpgrades between LTS releases follow a known and well-tested path.
Improved SupportThe dotCMS support team has more time to help with deeper issues and questions

Improved Patches and Upgrades

One benefit we'd like to highlight is the improvement LTS releases offer for patches and upgrades.

dotCMS is a platform, and every single dotCMS customer implements their sites in different ways, using different features of dotCMS, writing their own custom code, and integrating different third party applications and libraries. Because of this, no two upgrades are the same.

So, although many upgrades go relatively smoothly, upgrading from one dotCMS version to another always requires some work to ensure that your specific implementation works properly with the new version. This difference in implementations impacts hotfixes as well; even the best tested hotfixes can’t be tested against every possible implementation detail for all dotCMS versions it may apply to.

Thankfully, the LTS process enables us to improve both the patch and upgrade processes for customers on LTS releases.

Improved Patch Process

The LTS patch process has been streamlined to enable us to deliver fast, reliable, and more frequent patches for LTS releases.

Eliminates hotfixes

Since fixes are delivered as part of a new full patch release, and since patch releases are delivered on a regular bases, it virtually eliminates the need for hotfixes for LTS releases. This also simplifies system configuration, and eliminates potential conflicts between different hotfixes.

Low-risk changes

We don’t include all possible fixes in LTS patch releases. Instead, we choose which fixes to install, based on both the impact of the issue and on the likelihood that customers will experience them. As part of this decision process, we ensure that only fixes with low impact and low risk to existing systems are included; if a fix has the potential to cause new issues, it will intentionally be excluded.

Increased testing

Since the patch applies to only a single version, and only low-impact fixes are included, we can perform much more extensive testing. In other words, instead of having to test wide, we can now test deep.

No data migration

In addition, we exclude any fixes that would require data migration. This means that you can install patch releases without needing to change any of your data.

Simple intallation

Because data migration and other changes aren't needed, a patch install can be performed with a simple “in-place” upgrade for binary installs, and containerized instances can be upgraded by simply changing the version label and restarting the container.

Minimal or no downtime

All of this means that if you have a clustered instance, you can install a patch release with no downtime; each node can be upgraded and restarted separately without your environment going off-line. And even for non-clustered instances, the downtime is typically ten minutes or less.

The following compares the traditional hotfix process with the new LTS patch release process:

HotfixesLTS Patch Releases
Fixes IncludedVery few
(critical and security fixes only)
Many
User BaseVery limitedLarge
TestingNecessarily limitedFull testing cycle
DocumentationNecessarily limitedFull changelogs
AvailabilityBy request onlyFreely downloadable or
Automatically applied*
PostureReactiveProactive

* LTS Patch releases are automatically applied for all dotCMS Cloud customers on LTS releases.


The most important difference is that the hotfix model is necessarily reactive; fixes are only generated and installed after a problem is encountered (and even then only for critical issues). But LTS patch releases are proactive; fixes are continually integrated, and patch releases are automatically applied for all dotCMS Cloud customers.

Improved Upgrade Process

Another benefit for customers on LTS releases is that the process of performing a full upgrade between two different LTS releases is significantly improved over upgrades between major Agile releases.

Since there are many more Agile releases, when you upgrade between two Agile releases, there may be only a handful of customers who have upgraded between those two specific releases. So, it's less likely that another customer has performed the same upgrade with the same features that your implementation uses.

However, when you upgrade between LTS releases, there's almost always a large number of customers who have upgraded between those exact two releases, so the upgrade path is well known and well tested. In addition, since most customers will upgrade to a new LTS release after some patch releases have already been released for the later LTS release, it's less likely that you'll run into any issues during the upgrade.

Ultimately, this all means that upgrades between LTS versions are both faster and lower risk.

LTS Release Limitations

LTS releases do have some limitations compared to Agile releases.

Receive latest features and enhancements less frequently

As a general rule, new features and enhancements are not included in LTS patch releases. So, if you want a new feature or enhancement, you may need to upgrade to a newer LTS version - or possibly even an Agile release - to have access to that feature.

Not all fixes are included

Just as with hotfixes, there is no guarantee that any specific fix will be included in a given LTS release. While we make every effort to include fixes for issues which affect customers, we prioritize those that have the highest impact, and affect the most customers.

In addition, since the most important goal of LTS releases is to provide stability and reliability, high impact fixes (those which have the potential to cause new issues) are intentionally excluded. This also means that sometimes even if we plan to include a fix, we may need to leave it out if, while backporting the fix, we find that it presents risk that we didn't anticipate.

No fixed schedule

Although we'll typically deliver a new patch release for each LTS version every 6-12 weeks, there is no guaranteed schedule. The schedule is determined by a combination of need and resources, and at times releases may not happen for several months. Longer patch release cycles are especially likely toward the end of an LTS version's lifecycle (when it's both very diverged from the latest Agile code, and also already very stable).

Progress

So, now that we’ve discussed the benefits, we’d like to show you how much progress we’ve made so far. As of the time of this writing:

3 releases have been designated as LTS

  • 5.2.8 LTS
  • 5.3.8 LTS
  • 21.06 LTS

81% of dotCMS Cloud customers are on LTS releases

About half of these are currently on the 5.3.8 LTS release, and many of these are planning for migration to 21.06 LTS.

In the 5.3.8.6 LTS release alone, 103 fixes have been included

The vast majority of these fixes were delivered to customers before they ever experienced the issues themselves.


So, LTS releases have been widely adopted, and a large number of fixes have been delivered to customers - far more than would have been possible with the traditional hotfix process.

Results

It’s easy to say that something will work better, but the only thing that really matters is - does it actually work?

And the answer is that, although our goal was to be able to improve service levels for customers who adopted LTS releases, the process has been so successful that we’ve been able to improve support for all of our customers - even those on Agile releases.

So, how do we know that? As an engineering-driven company, we believe that when it comes to showing that something works, it’s hard to beat looking at the numbers.

Number of support requests reduced 32%

If the LTS release process is really improving the stability and reliability, then we should see a reduced number of support requests. And that's exactly what we've seen. And this has helped customers on both Agile and LTS release, because the fewer support requests we receive, the more time our support staff has to help all of our customers, regardless of what releases they’re running.

The following chart shows the total number of support requests received every month for the past two years. The green vertical line shows when the LTS process was started (with the designation of the first LTS releases):

For the past two years, the number of support requests we received each month has been within a consistent range. But since the implementation of the LTS process, the number of requests has dropped to historically low levels.

One important thing to note is that these numbers are the total support requests for all customers (on both Agile and LTS releases). But the number of support requests just from customers on LTS releases has dropped by almost 50%. In other words, customers who’ve moved to LTS releases are now submitting half as many support requests as they did before they moved to an LTS release.

Support calls resolved 61% faster

If the LTS release process is improving support, the other place we expect to see a change is in the time it takes us to resolve the submitted support requests to the customers' satisfaction. So, let's take a look at that.

The following graph shows the average resolution time for support requests for the past two years:

Only a few months after starting the LTS process, the speed at which we resolved support requests was faster than at any time in the previous 18 months. And just over half a year later, we’re resolving requests more than twice as fast as we ever did in the past.

Greatly improved response times

To round out the numbers, let's take a quick look at response times. The the combination of the LTS release process and other improvements in our support processes now enables us to respond to support requests much faster than in the past. Initial response times (the time it takes us to respond after a request is first submitted) has been cut by 56%, and the time it takes for us to respond to each and every customer comment has dropped by 24%.

Conclusion

The Long-Term-Supported (LTS) release process has successfully allowed us to improve stability and reliability, almost double release support periods, and simplify both patching and upgrades for customers on LTS releases. In addition, it's enabled us to improve the support we provide to all of our customers.

As an engineer I tend to understate things, but it’s fairly obvious from the data that the results have been dramatic. We’re excited about how much we’ve been able to help our customers, and looking forward to sharing with you more great changes as we keep moving forward!

Filed Under:

Recommended Reading

Headless CMS vs Hybrid CMS: How dotCMS Goes Beyond Headless

What’s the difference between a headless CMS and a hybrid CMS, and which one is best suited for an enterprise?

Exploring TimeMachine: How dotCMS Handles Content Versioning

With dotCMS, tools like TimeMachine and Workflow Management help enterprises control content versioning, content edits, user permissions, workflows and more.

Digital Ambition Requires Ambitious Technology

When it comes to the content foundation of a digital experience platform, critical capabilities are reasonably straightforward. There are only four, really - since security, scalability and containeri...