Backup & Restore of Payara Server

Recently, I was working with a customer on a production deployment of Payara Server and the topic of disaster recovery and backups arose. It seemed like a useful topic to blog about and share my thoughts with the community.

Oracle GlassFish 3 had a built-in automatic backup facility using the asadmin command create-backup-config. This allowed you to specify a schedule for when to backup, where to place the backed up file, and how many backups to keep.

This is no longer present in GlassFish, since it was a commercial feature that was never open-sourced, but its effects can be reproduced without much effort using the backup-domain command and some scheduling software (crontab, for example)

Since the domain directory is where all of the user configuration of Payara Server is kept, we can consider this to be separate to the installation files that Payara Server uses to run the app server itself. We can backup the domain using the asadmin backup-domain command as follows:

asadmin backup-domain --backupDir /path/to/backup/directory myDomain

The effect of this command is to create a full backup of the domain directory, however, to complete the command the domain must be stopped. This will mean that any application deployed to the DAS is unavailable while the backup takes place.

The job of running this backup command on a given schedule and managing the resulting backed up files needs to be done by a script external to Payara Server.

To restore from a backup the asadmin restore-domain command is used:

asadmin restore-domain --filename ${name} --backupdir /path/to/backup/directory myDomain

This command will restore just the domain named in the command. Since instances are stored separately to each domain (and may be on remote hosts), the command will have no effect on instances.

Since all data in an instance directory is kept in sync with the DAS, there is no added data (besides logs, caches and some MQ data when a file-based store is used) and an instance can be completely restored simply by making sure there is an empty directory with the name of the instance inside a node folder of the correct name. For example: payara41/nodes/localhost-localdomain/myInstance

Running the start-local-instance command with a full sync, as discussed earlier, will restore the instance completely:

asadmin start-local-instance --sync=full myInstance

If OpenMQ (Payara Server’s JMS broker) is being used, the default configuration is to use EMBEDDED mode, meaning that Payara Server will fully manage the JMS broker within the same JVM as the server instance and a file-based store is used for persistent messages 

The alternative option of LOCAL allows for the use of an “enhanced” broker cluster, whereby the JMS broker instances are started as separate JVMs and use a database connection for storage. Assuming the database is highly available, this completely avoids any danger when needing to –sync=full the instance.

This process has quite a few steps when considering that this process would need to be repeated each time an upgrade was to be done. Since Payara offer a new patch release (bugfixes only) every month and a new full release (bugfixes and new features) every 3 months, this could end up being a process that gets repeated very often. If regular backups are taken, as recommended, then it may not be so major.