This is a tutorial for upgrading the Mender Server in production environments. In general updating is only supported between connected minor versions, and latest minor to next major version, and this tutorial reflects this. Both the Open Source and Enterprise editions can be upgraded using this tutorial, but both the old and the new version must be the same type of server, either both Open Source, or both Enterprise.

Prerequisites

It is assumed that the installation was performed following the steps in the Installation tutorial. That means that you currently have:

• a local git repository based on mender-integration
• a branch with your production overrides
• all configuration overrides committed to your production branch

As a good engineering practice, it is advisable to perform the upgrade on a staging environment first. This will allow you to discover potential problems and allow to exercise the procedure in a safe manner.

The installation is largely based on using git and Mender integration repository. This is the reason why the upgrade procedure follows a regular git workflow with branching, pulling remote changes and merging locally.

Backing up existing data

Before upgrading it is advisable to backup existing data and volumes. Consult the MongoDB and Docker manuals for the necessary steps.

The Backup and restore chapter provides examples and introduces example tools provided in Mender integration repository.

Cleaning up the deviceauth database after device decommissioning.

Before upgrading it is advisable to clean up any leftover devices from the deviceauth database. These can sometimes happen due to device decommissioning. You can find instructions on how to clean up the deviceauth database in the Troubleshoot chapter.

Updating your local repository

The first step is to upgrade your local repository by pulling changes from the Mender integration repository. This can be achieved by running git remote update origin.

git fetch origin --tags

Fetching origin
remote: Counting objects: 367, done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 367 (delta 134), reused 122 (delta 122), pack-reused 214
Receiving objects: 100% (367/367), 83.55 KiB | 0 bytes/s, done.
Resolving deltas: 100% (214/214), completed with 42 local objects.
From https://github.com/mendersoftware/integration
   02cd118..75b7831  3.3.2      -> origin/3.3.2
   06f3212..e9e5df4  3.3.2     -> origin/3.3.2

For each release there will be a corresponding release branch. For example, the branch named 2.0.x provides the 2.0 release setup. Stable releases are tagged, e.g. 2.0.1.

Recall from the installation tutorial that our local setup was introduced in a branch that was created from given release version. You can use git commands such as git log and git diff to review the changes introduced in upstream branch. For example:

# to list differences between current HEAD and remote branch
git log HEAD..origin/2.0.x
# to list differences between current HEAD and stable tag
git log HEAD..3.3.2

The most important thing to review is the diff between our production template version and the version present in the repository. For a patch release there should be none, or just some minor changes. However, when there is a minor/major release, one can expect the diff to be larger. Example:

# while at the root of repository
user@local$ git diff HEAD..3.3.2 -- template

Upgrading our local production branch is performed by issuing a git merge command, like this:

git merge 3.3.2

Merge made by the 'recursive' strategy.
 .travis.yml            | 16 ++++++++++++++++
 tests/run.sh           |  4 ++--
 update                 |  1 -
 verify-docker-versions | 29 ++++++++++++++++++++---------
 4 files changed, 38 insertions(+), 12 deletions(-)

Upgrading from Mender 2.6 or older

If you are upgrading from Mender 2.6 or older to Mender 2.7 or newer please follow this section. If you are already running Mender 2.7 or newer you can skip this.

Starting from version 2.7, Mender backend is not using separate service as a Storage Proxy. Instead, Mender backend uses an API Gateway based on traefik to serve both backend endpoints and artifacts downloads. Because of this you should update the gateway definition in your production yaml file to include the domain name of your old Storage Proxy. The updated entry should look like that:

    ...
    mender-api-gateway:
        ...
        networks:
            mender:
                aliases:
                    # mender-api-gateway is a proxy to storage
                    # and has to use exactly the same name as devices
                    # and the deployments service will;
                    #
                    # if devices and deployments will access storage
                    # using https://s3.acme.org:9000, then
                    # set this to https://s3.acme.org:9000
                    - set-my-alias-here.com
    ...

Starting upgraded environment

Once the changes are merged, you can recreate the containers.

First, pull in new container images:

./run pull

Pulling mender-mongo                  ... done
Pulling mender-deviceconfig           ... done
Pulling mender-useradm                ... done
Pulling mender-workflows-worker       ... done
Pulling mender-create-artifact-worker ... done
Pulling mender-workflows-server       ... done
Pulling mender-device-auth            ... done
Pulling mender-gui                    ... done
Pulling mender-inventory              ... done
Pulling mender-api-gateway            ... done
Pulling minio                         ... done
Pulling mender-deployments            ... done
Pulling mender-nats                   ... done
Pulling mender-deviceconnect          ... done

Then stop and remove existing containers:

./run stop

Stopping menderproduction_mender-api-gateway_1 ... done
Stopping menderproduction_mender-inventory_1 ... done
Stopping menderproduction_mender-deployments_1 ... done
Stopping menderproduction_mender-device-auth_1 ... done
Stopping menderproduction_mender-useradm_1 ... done
Stopping menderproduction_storage-proxy_1 ... done
Stopping menderproduction_mender-mongo_1 ... done
Stopping menderproduction_mender-gui_1 ... done
Stopping menderproduction_minio_1 ... done

./run rm

Going to remove menderproduction_mender-api-gateway_1, ...
Are you sure? [yN] y
Removing menderproduction_mender-api-gateway_1 ... done
Removing menderproduction_mender-inventory_1 ... done
Removing menderproduction_mender-deployments_1 ... done
Removing menderproduction_mender-device-auth_1 ... done
Removing menderproduction_mender-useradm_1 ... done
Removing menderproduction_storage-proxy_1 ... done
Removing menderproduction_mender-mongo_1 ... done
Removing menderproduction_mender-gui_1 ... done
Removing menderproduction_minio_1 ... done

Start the new environment:

./run up -d

Creating menderproduction_mender-nats_1  ... done
Creating menderproduction_mender-gui_1   ... done
Creating menderproduction_minio_1        ... done
Creating menderproduction_mender-mongo_1 ... done
Creating menderproduction_mender-deviceconfig_1           ... done
Creating menderproduction_mender-workflows-worker_1       ... done
Creating menderproduction_mender-create-artifact-worker_1 ... done
Creating menderproduction_mender-deviceconnect_1          ... done
Creating menderproduction_mender-useradm_1                ... done
Creating menderproduction_mender-inventory_1              ... done
Creating menderproduction_mender-workflows-server_1       ... done
Creating menderproduction_mender-device-auth_1            ... done
Creating menderproduction_mender-api-gateway_1            ... done
Creating menderproduction_mender-deployments_1            ... done

Closing notes

Since the production repository is versioned in git, it is possible to use git tools and apply typical git workflows, such as pushing, pulling, branching, etc.

Pushing and pulling to/from a company hosted git repository is a great way to share configuration between staging and production environments. A configuration can be validated in a staging environment as relevant changes are committed and pushed to the repository. Once they are validated, a production environment can pull the changes and apply them locally.