Auto-Clustering Configurationdocumentation for the dotCMS Content Management System

The Auto-clustering feature allows you to automatically add servers to a cluster, and automatically manages the cluster configuration for you. The following configuration steps are required to set up auto-clustering.

Note: You must complete all installations steps in the Cluster Setup document before performing any of these installation steps.

Download and Apply a License Pack

Auto-clustering requires a valid Enterprise license pack. Enterprise clients can download license packs from the dotCMS Support website. Contracts that have licenses available provide the option to download license packs.

If you are an Enterprise client (or would like to move to Enterprise), please contact dotCMS Support for more information about Enterprise contracts and support portal options.

Once downloaded from the support portal, license packs for auto-clustering should be uploaded to only one server that is part of the cluster. Once the license pack is uploaded, a license can be chosen for the current server.

For more information on applying licenses, see the License Management documentation.

Add Additional Servers to the Cluster

The rest of the server nodes in the cluster do not need their own database or assets folder, and do not need separate licenses. Instead, each additional server in the cluster must only:

  • Have a symbolic link to the same asset directory as the first server.
  • Share a connection to the same database.

No other configuration is required. However the following settings found in the dotcms-config-cluster.properties file can be used to modify Auto-clustering behavior:

PropertyDefaultDescription
AUTOWIRE_CLUSTER_TRANSPORTtrueActivates the auto-wire process at cluster cache transport level.
AUTOWIRE_CLUSTER_EStrueActivates the auto-wire process at the elastic search index level.
HEARTBEAT_TIMEOUT1800
(30 minutes)
Timeout (in seconds) before a node will be considered unresponsive (please see Unresponsive Nodes, below).
Note: 30 minutes without a heart beat provides a reasonable time window to perform normal server changes without dropping the node from the cluster.
HEARTBEAT_CRON_EXPRESSION0 0/1 * * * ?
(once every minute)
The frequency to check for dead servers in the cluster.

Note: It is strongly recommended that all changes to the dotcms-config-cluster.properties file be made through a properties file extension.

As each additional node starts up, it will recognize that it shares the same database and assets directory as the first node in the cluster, and will automatically search for an available license from the license pack already uploaded to the first server. If a license is available, it will automatically be assigned, as long as there are enough valid licenses in the license pack for each node. Additional license packs can always be uploaded to the first server, allowing scaling of the cluster over time.

Unresponsive Nodes

If a node's “heartbeat” is not detected before the HEARTBEAT_TIMEOUT expires, then the node will be dropped from the cluster. If a node is dropped, dotCMS will:

  1. Free the license.
  2. Update the Elasticsearch replica count.
  3. Rewire the cache and Elasticsearch.

If a temporarily disconnected server is re-discovered within 30 minutes, the server can still re-join the cluster normally.

Activate License Icon

Elasticsearch Replica Management

By default, the Auto-clustering feature handles the replicas for Elasticsearch. The default configuration setting ES_INDEX_REPLICAS=autowire. The AUTOWIRE_CLUSTER_ES is also set to true by default.

When AUTOWIRE_CLUSTER_ES is set to true OR if ES_INDEX_REPLICAS is set to anything other than a fixed integer, then backend UI manual management of clustered elastic search replicas is disabled.

However, if AUTOWIRE_CLUSTER_ES=false AND ES_INDEX_REPLICAS is set to a fixed INTEGER (not a range), then this enables the backend UI manual management of clustered elastic search replicas

The possible values for ES_INDEX_REPLICAS are:

  • autowire: This will use the dotCMS autowire logic to set the replicas based on the current alive dotCMS instances. This should only be used with local Elasticsearch indexes. (this one lets the autowire logic set the replicas based on the alive-node count and will automatically set auto_expand_replicas=false)
  • integer: An integer value (e.g. “1,2,3,4, etc.“) - this will configure the indexes to use a static number of desired replicas. This is useful if you have a static set of external Elasticsearch servers. If there are not enough instances/replicas available to satisfy the config, the index will go into the “yellow” (unhealthy) state until additional instances/replicas are available. This setting will automatically set auto_expand_replicas=false
  • 0-all: (or any other boundary. 0-8, etc.) - A lower to upper integer boundary on the number of replicas the index can auto expand to, with the “all” keyword allowed as the upper boundary. This will allow the index replicas to auto-expand and contract based on the current Elasticsearch cluster topology. It is recommended to use this only with an external Elasticsearch cluster, and not with local Elasticsearch indexes. This option allows for flexible Elasticsearch cluster scalability, but can cause data loss in certain situations. Please discuss this with dotCMS Support before implementing. Sets auto_expand_replicas property based on the upper boundary that the replicas can “auto expand” to.

If you are using an external ElasticSearch cluster with dotCMS running in Elasticsearch client-mode, it is recommended that you disable autowire replica management and manage your replicas directly in ElasticSearch. For more information, please consult the ElasticSearch documentation or dotCMS Support for the best practices in your environment.

For more information on setting the number of replicas, please see the Manual Cluster Configuration documentation.

Auto-Cluster Configuration Checklist

The following checklist outlines the steps required to establish an auto-cluster environment:

  1. Have a valid Enterprise contract with available server licenses.
  2. Request a license pack by submitting a ticket on the dotCMS Support website.
  3. Start up the first dotCMS server (server #1).
  4. Upload the license pack to the License Manager of server #1.
  5. Select a license from the pack to be used by server #1.
  6. Configure the second dotCMS server to share the same assets and database.
  7. Start up the second server (it will automatically join the cluster).

To add additional server nodes to the cluster, repeat steps 6 & 7 and each node will automatically be joined to the cluster (as long as unused licenses are still available*).

* Warning: If multiple nodes in the cluster are on the same physical server, then the Tomcat port and shutdown ports will need to be different for each node in the cluster. Ports can be configured in the server.xml file (/dotserver/tomcat-X.xx/conf/server.xml), to avoid conflicts.

Verifying Your Cluster

After adding all nodes to your cluster, please see the Testing Your Cluster section in the Cluster Setup documentation for steps to verify the proper operation of your cluster.