You are browsing documentation for a version other than the latest stable release. Switch to the latest stable release, 2.3.
You can save time by using hosted Mender, a secure Mender server ready to use, maintained by the Mender developers.
This is a step by step guide for deploying the Mender Server for production
environments, and will cover relevant security and reliability aspects of Mender
production installations. The Mender backend services can be deployed to
production using a skeleton provided in the production directory of the
integration
repository. Most of the steps are the same whether you are installing the Open
Source or Enterprise edition of the Mender Server, but there are some extra
steps that are covered in the Enterprise subsection.
If you have already installed an Open Source server and wish to upgrade to Enterprise, you should follow the Enterprise upgrade guide instead of this guide.
If you are setting up a Mender Enterprise server, you will also need:
It is very likely possible to use other Linux distributions and versions. However, we recommend using this exact environment for running Mender because it is known to work and you will thus avoid any issues specific to your environment if you use this reference.
The guide will use a publicly available Mender integration repository. Then proceed with setting up a local branch, preparing deployment configuration from a template, committing the changes locally. Keeping deployment configuration in a git repository ensures that the history of changes is tracked and subsequent updates can be easily applied.
At the end of this guide you will have:
docker-compose fileConsult the section on certificates and keys for details on how the certificates and keys are used in the system.
docker-compose implements a particular naming scheme for containers, volumes
and networks that prefixes each object it creates with project name. By default,
the project is named after the directory the main compose file is located in.
The production configuration template provided in the repository explicitly sets
project name to menderproduction.
docker-compose automatically assigns a
com.docker.compose.project=menderproduction label to the created containers. The
label can be used when filtering output of commands such as docker ps.
Start off by cloning the Mender integration repository into a local directory
named mender-server:
git clone -b 2.4.0b1 https://github.com/mendersoftware/integration mender-serverCloning into 'deployment'... remote: Counting objects: 1117, done. remote: Compressing objects: 100% (11/11), done. remote: Total 1117 (delta 1), reused 0 (delta 0), pack-reused 1106 Receiving objects: 100% (1117/1117), 233.85 KiB | 411.00 KiB/s, done. Resolving deltas: 100% (678/678), done. Checking connectivity... done.
Enter the directory:
cd mender-serverPrepare a branch where all deployment related changes will be kept:
git checkout -b my-production-setupEnter the production directory:
cd productionCopy the production template to its own file:
cp config/prod.yml.template config/prod.ymlls -l *-rwxr-xr-x 1 user user 725 Sep 17 14:14 run config: total 24 -rw-r--r-- 1 user user 566 Sep 17 14:14 enterprise.yml.template -rw-r--r-- 1 user user 5814 Sep 10 15:44 prod.yml -rw-r--r-- 1 user user 5752 Sep 17 14:14 prod.yml.template
The template includes a few files:
run - a convenience helper that invokes docker-compose by passing the required
compose files; arguments passed to run in command line are forwarded
directly to docker-compose
prod.yml and prod.yml.template - contains deployment-specific
configuration, builds on top of docker-compose.yml (located at the root of
integration repository)
enterprise.yml.template - configuration for running an Enterprise instance
of the Mender server. This topic is covered separately in the Enterprise
part of the Production installation guideIf an enterprise.yml file exists in the config directory, this will
automatically turn on Enterprise features in the backend service. If you
want to use the Open Source edition (not Enterprise), please remove
enterprise.yml from the config directory.
At this point all changes should be committed to the repository:
git add .git commit -m 'production: initial template'[my-production-setup 556cc2e] production: initial template 1 file changed, 54 insertions(+) create mode 100644 production/config/prod.yml
Assuming that current working directory is still production, download
necessary Docker images:
./run pullPulling mender-mongo (mongo:3.4)... 3.4: Pulling from library/mongo Digest: sha256:aff0c497cff4f116583b99b21775a8844a17bcf5c69f7f3f6028013bf0d6c00c Status: Image is up to date for mongo:3.4 ... Pulling mender-gui (mendersoftware/gui:latest)... latest: Pulling from mendersoftware/gui b7f33cc0b48e: Already exists 31e101b48355: Pull complete 307a023cd01f: Pull complete 2edcf3035646: Pull complete aeb59eb28f90: Pull complete ... Pulling mender-useradm (mendersoftware/useradm:latest)... latest: Pulling from mendersoftware/useradm Digest: sha256:3346985a2679b7edd9243363c4b1c291871481f8c6f557ccdc51af58dc6d3a1a Status: Image is up to date for mendersoftware/useradm:latest Pulling mender-device-auth (mendersoftware/deviceauth:latest)... latest: Pulling from mendersoftware/deviceauth Digest: sha256:ed47310dc9a86cca70d52c520d565ef9814f39421f2faa02d60bbe8a1dbd63c5 Status: Image is up to date for mendersoftware/deviceauth:latest Pulling mender-api-gateway (mendersoftware/api-gateway:latest)... latest: Pulling from mendersoftware/api-gateway Digest: sha256:97243de3da950e754ada73abf8c123001c0a0c8b20344256dc7db4c89c0ecd82 Status: Image is up to date for mendersoftware/api-gateway:latest
Using the run helper script may fail when local user has insufficient permissions to reach a local docker daemon. Make sure that the Docker installation was completed successfully and the user has sufficient permissions (typically the user must be a member of the docker group).
First, set the public domain name of your server (the URL your devices will reach your Mender server on):
API_GATEWAY_DOMAIN_NAME="mender.example.com" # NB! replace with your server's public domain name
STORAGE_PROXY_DOMAIN_NAME="$API_GATEWAY_DOMAIN_NAME" # change if you are hosting the Storage Proxy on a different domain name (not the default)Prepare certificates using the helper script keygen:
CERT_API_CN=$API_GATEWAY_DOMAIN_NAME CERT_STORAGE_CN=$STORAGE_PROXY_DOMAIN_NAME ../keygenGenerating a 256 bit EC private key writing new private key to 'private.key' ... All keys and certificates have been generated in directory keys-generated. Please include them in your docker compose and device builds. For more information please see https://docs.mender.io/Administration/Certificates-and-keys.
Your local directory tree should now look like this:
├── keys-generated
│ ├── certs
│ │ ├── api-gateway
│ │ │ ├── cert.crt
│ │ │ └── private.key
│ │ └── server.crt
│ │ └── storage-proxy
│ │ ├── cert.crt
│ │ └── private.key
│ └── keys
│ ├── deviceauth
│ │ └── private.key
│ └── useradm
│ └── private.key
├── config/enterprise.yml.template
├── config/prod.yml
├── config/prod.yml.template
└── runThe production template file prod.yml is already configured to load keys and
certificates from locations created by the keygen script. If you wish to use a
different set of certificates or keys, please consult the
relevant documentation.
Next, we can add and commit generated keys and certificates:
Note that the following steps will commit your keys in plaintext to the git history. Make sure to keep all keys secure. Use encryption if needed and decrypt the keys before using them in containers; see the section below on encrypting keys for some examples.
git add keys-generatedgit commit -m 'production: adding generated keys and certificates'[my-production-setup fd8a397] production: adding generated keys and certificates 6 files changed, 108 insertions(+) create mode 100644 production/keys-generated/certs/api-gateway/cert.crt create mode 100644 production/keys-generated/certs/api-gateway/private.key create mode 100644 production/keys-generated/certs/server.crt create mode 100644 production/keys-generated/certs/storage-proxy/cert.crt create mode 100644 production/keys-generated/certs/storage-proxy/private.key create mode 100644 production/keys-generated/keys/deviceauth/private.key create mode 100644 production/keys-generated/keys/useradm/private.key
The API Gateway and Storage Proxy certificates generated here need to be made available to the Mender client. Consult the section on building for production for a description on how to include the certificates in the client builds.
Only certificates need to be made available to devices or end users. Private keys should never be shared.
A simple method for encrypting keys is to
use the GNU Privacy Guard tool. Instead of committing
plaintext keys to the repository, we will tar and encrypt the entire
keys-generated directory structure, then commit it to the repository as a
single binary blob.
The entire directory structure can be encrypted with the following command:
tar c keys-generated | gpg --output keys-generated.tar.gpg --symmetric
Enter passphrase:rm -rf keys-generatedls -ltotal 20 drwxrwxr-x. 1 user group 4096 01-30 11:08 config -rw-rw-r--. 1 user group 5094 01-30 11:24 keys-generated.tar.gpg -rwxrwxr-x. 1 user group 173 01-27 14:16 run
Keys need to be decrypted before bringing the whole environment up:
gpg --decrypt keys-generated.tar.gpg | tar xvf -
gpg: AES encrypted data
Enter passphrase:When using encryption, commit keys-generated.tar.gpg instead of the
keys-generated directory structure to the repository, like this:
git add keys-generated.tar.gpggit commit -m 'production: adding generated keys and certificates'[my-production-setup 237af44] production: adding generated keys and certificates 1 file changed, 111 insertions(+) create mode 100644 production/keys-generated.tar.gpg
Persistent storage of backend services' data is implemented using named Docker volumes. The template is configured to mount the following volumes:
mender-artifacts - artifact objects storagemender-db - mender services databases dataCreate these volumes with the following commands:
docker volume create --name=mender-artifacts
docker volume create --name=mender-dbThe storage location of these volumes depends on your docker configuration. You can check the path for a specific volume by running docker volume inspect --format '{{.Mountpoint}}' mender-artifacts.
The deployment configuration in config/prod.yml now needs to be updated.
The keys MINIO_ACCESS_KEY and MINIO_SECRET_KEY control
credentials for uploading artifacts into the object store. Since Minio is a S3 API
compatible service, these settings correspond to Amazon's AWS Access Key ID and
Secret Access Key respectively.
First, generate a secret key for Minio with the pwgen utility:
On the Debian family of distributions you can install pwgen with apt-get install pwgen.
MINIO_SECRET_KEY_GENERATED=$(pwgen 16 1) && echo $MINIO_SECRET_KEY_GENERATEDahshagheeD1ooPae
Insert the access and secret keys into config/prod.yml with the following commands:
sed -i.bak "s/MINIO_ACCESS_KEY:.*/MINIO_ACCESS_KEY: mender-deployments/g" config/prod.yml
sed -i.bak "s/MINIO_SECRET_KEY:.*/MINIO_SECRET_KEY: $MINIO_SECRET_KEY_GENERATED/g" config/prod.ymlThe updated entry should look similar to this, you can verify with git diff:
...
minio:
environment:
# access key
MINIO_ACCESS_KEY: mender-deployments
# secret
MINIO_SECRET_KEY: ahshagheeD1ooPae
...
The deployments service will upload artifact objects to minio storage via storage-proxy,
see the administration overview for more details. For this reason,
access credentials DEPLOYMENTS_AWS_AUTH_KEY and DEPLOYMENTS_AWS_AUTH_SECRET
need to be updated and DEPLOYMENTS_AWS_URI must point to the domain name of your Storage proxy.
Run the following commands to set DEPLOYMENTS_AWS_AUTH_KEY and
DEPLOYMENTS_AWS_AUTH_SECRET to the values of MINIO_ACCESS_KEY
and MINIO_SECRET_KEY, respectively.
sed -i.bak "s/DEPLOYMENTS_AWS_AUTH_KEY:.*/DEPLOYMENTS_AWS_AUTH_KEY: mender-deployments/g" config/prod.yml
sed -i.bak "s/DEPLOYMENTS_AWS_AUTH_SECRET:.*/DEPLOYMENTS_AWS_AUTH_SECRET: $MINIO_SECRET_KEY_GENERATED/g" config/prod.ymlAlso, run the following command so DEPLOYMENTS_AWS_URI points to your Storage proxy (including the right port, 9000 by default):
sed -i.bak "s/https:\/\/set-my-alias-here.com/https:\/\/$STORAGE_PROXY_DOMAIN_NAME:9000/g" config/prod.ymlAfter these three commands, the updated entry should look like this (you can again verify with git diff):
...
mender-deployments:
...
environment:
DEPLOYMENTS_AWS_AUTH_KEY: mender-deployments
DEPLOYMENTS_AWS_AUTH_SECRET: ahshagheeD1ooPae
DEPLOYMENTS_AWS_URI: https://mender.example.com:9000
...The address used in DEPLOYMENTS_AWS_URI must be exactly the same as the one that will be used by devices. The deployments service generates signed URLs for accessing artifact storage. A different hostname or port will result in signature verification failure and download attempts will be rejected.
Add your storage server's DNS name under the key networks.mender.aliases key
by running the following command:
sed -i.bak "s/set-my-alias-here.com/$STORAGE_PROXY_DOMAIN_NAME/g" config/prod.ymlThe entry should now look like this:
...
storage-proxy:
networks:
mender:
aliases:
- mender.example.com
...
You can also change the values for DOWNLOAD_SPEED and MAX_CONNECTIONS.
See the section on bandwidth for more details on these
settings.
For security purposes, the API Gateway must know precisely the DNS name allocated to it,
which you'll configure via the ALLOWED_HOSTS environment variable.
Change the placeholder value my-gateway-dns-name to a valid hostname, by running
the following command:
sed -i.bak "s/my-gateway-dns-name/$API_GATEWAY_DOMAIN_NAME/g" config/prod.ymlThe updated entry should look like this:
...
mender-api-gateway:
...
environment:
ALLOWED_HOSTS: mender.example.com
...Note that if for some reason you need more than 1 DNS name pointing to the gateway, you can supply a whitespace-separated list of hostnames as follows:
...
mender-api-gateway:
...
environment:
ALLOWED_HOSTS: mender.example.com mender2.example.com
...The setup uses Docker's default json-file logging driver, which exposes two important log
rotation parameters: max-file and max-size. For selected services - consistently producing
a large amount of logs - we override these settings to ensure that at any given time, logs capture
about 1 week of a running system's operation.
The reference setup used for determining these parameters (max-file: 10, max-size: 50m)
was 200 devices, polling the server every 30 minutes. The total log volume produced was 2.4GB,
and this is the recommended amount of disk space to reserve for logging purposes alone.
These parameters can however be adjusted accordingly to a particular deployment and workload. The deciding factors determining the log volume are:
It is suggested to adjust log rotation parameters only after measuring the actual space usage for a given use case.
Once all the configuration is complete, commit all changes to the repository:
git add config/prod.ymlgit commit -m 'production: configuration'At this point your commit history should look as follows:
git log --oneline 2.4.0b1..HEAD7a4de3c production: configuration 41273f7 production: adding generated keys and certificates 5ad6528 production: initial template
This section deals specifically with setting up an Open Source server. If you are setting up an Enterprise server, please proceed to the Enterprise section.
Bring up all services up in detached mode with the following command:
./run up -dCreating network "menderproduction_mender" with the default driver Creating menderproduction_mender-mongo_1 Creating menderproduction_mender-gui_1 Creating menderproduction_minio_1 Creating menderproduction_mender-device-auth_1 Creating menderproduction_mender-inventory_1 Creating menderproduction_storage-proxy_1 Creating menderproduction_mender-useradm_1 Creating menderproduction_mender-deployments_1 Creating menderproduction_mender-api-gateway_1
Services, networks and volumes have a menderproduction prefix, see the note about docker-compose naming scheme for more details. When using docker .. commands, a complete container name must be provided (ex. menderproduction_mender-deployments_1).
Since this is a brand new installation we need to create the initial user via the CLI provided by the User Administration Service. The service's binary is embedded in a Docker container, so to execute it you will issue the exec subcommand of docker-compose, e.g.:
./run exec mender-useradm /usr/bin/useradm create-user --username=myusername@host.com --password=mysecretpasswordKeep in mind that above is executed in a command-line interpreter meaning that certain characters might need to be escaped, e.g if you are using the $ character in your password, this could interpret as a variable name unless it is escaped.
The next sections deal with installation of an Enterprise server. If you are installing an Open Source server, please proceed to verification now.
This section will go through setting up the Enterprise features of the Mender server. If you are using the Open Source edition of the Mender server, you can skip ahead to the verification.
Once the Open Source parts of the configuration have been completed, you should have a fully functional server setup, except that it has not been launched yet. It is now time to configure the Enterprise specific features before bringing the server up.
Copy the enterprise.yml.template file to its production location,
enterprise.yml:
cp config/enterprise.yml.template config/enterprise.ymlCreating the enterprise.yml file enables the Enterprise Mender server.
First log in to the Mender docker registry with your Mender Enterprise credentials:
docker login registry.mender.ioIf you have lost your credentials or need an evaluation account please email contact@mender.io.
Bring up all services up in detached mode with the following command:
./run up -dCreating menderproduction_mender-api-gateway_1 ... done Creating menderproduction_mender-create-artifact-worker_1 ... done Creating menderproduction_mender-deployments_1 ... done Creating menderproduction_mender-device-auth_1 ... done Creating menderproduction_mender-gui_1 ... done Creating menderproduction_mender-inventory_1 ... done Creating menderproduction_mender-mongo_1 ... done Creating menderproduction_mender-tenantadm_1 ... done Creating menderproduction_mender-useradm_1 ... done Creating menderproduction_mender-workflows-server_1 ... done Creating menderproduction_mender-workflows-worker_1 ... done Creating menderproduction_minio_1 ... done Creating menderproduction_storage-proxy_1 ... done
Services, networks and volumes have a menderproduction prefix, see the note about docker-compose naming scheme for more details. When using docker .. commands, a complete container name must be provided (ex. menderproduction_mender-deployments_1).
Note that even though this launches the Enterprise backend services, the Mender Enterprise server is not fully functional yet.
Multi tenancy is a feature of Mender where users can be divided into "organizations" (also sometimes called "tenants"). Each organization is confined and can not see or influence the data of other organizations. This can be used for example to give two different product teams access to the Mender service, without them seeing each others data, as well as isolating devices and users in test and production environments.
When using Mender Enterprise, multi tenancy is automatically enabled, and it cannot be turned off. Below follows a guide for setting up a single organization. For additional information on administering organizations, see the help screen from:
./run exec mender-tenantadm /usr/bin/tenantadm --helpIn either case, at least one organization must be created when first installing
the service. Execute this command in the production directory.
Replace the organization name, username and password with desired values and run this command:
TENANT_ID=$(./run exec mender-tenantadm /usr/bin/tenantadm create-org --name=MyOrganization --username=myusername@host.com --password=mysecretpassword | tr -d '\r') && (echo $TENANT_ID)Make sure to save the tenant ID that appears after calling the command (which will also be in the TENANT_ID shell variable).
This will be the identifier for the first organization. Creating additional organizations
works exactly the same way, so the above step may be repeated multiple times.
Now we need to fetch the tenant token for the new organization. This is
available in the JSON output from the get-tenant command, in the
tenant_token field. To avoid manually parsing raw JSON, we can use the jq
tool:
On the Debian family of distributions you can install jq with apt-get install jq.
TENANT_TOKEN=$(./run exec mender-tenantadm /usr/bin/tenantadm get-tenant --id $TENANT_ID | jq -r .tenant_token) && (echo $TENANT_TOKEN)where $TENANT_ID is the ID that was printed by the previous command.
The output tenant token should be a long string like this:
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJtZW5kZXIudGVuYW50IjoiNWQ4MzM1MzJkMTMwNTgwMDI4NDhmZmRmIiwia
XNzIjoiTWVuZGVyIiwic3ViIjoiNWQ4MzM1MzJkMTMwNTgwMDI4NDhmZmRmIn0.HJDGHzqZqbosAYyJpSIEeL0W4HMiOmb15ETnu
ChxE0i7ahW49dZZlQNJBKLkLzuESDxXnmQbQwsSGP6t32pGkeHVXTPjrSjhtMPC80NiibNG3f-QATw9I8YgG2SBd5xaKl17qdta1
nGi80T2UKrwlzqLHR7wNed10ss3NgJDIDvrm89XO0Rg6jpFZsHCPyyK1c8-Vn8zZjW5azZLNSgtgSLSmFQguQLRRXL2x12VEcmez
tFY0kJnhGtN07CD3XXxcz0BpWbevDYOPOEUusGd2KpLK2Y4QU8RdngqNtRe7SppG0fn6m6tKiifrPDAv_THCEG6dvpMHyIM77vGI
PwvV4ABGhZKRAlDe1R4csJQIbhVcTWMGcoZ4bKH-zDK0900_wWM53i9rkgNFDM470i6-d1oqfaCPcbiniKsq1HcJRZsIVNJ1edDo
vhQ6IbffPRCw-Au_GlnPTn_czovJqSa3bgwrJguYRIKJGWhHgx0e3j795oJ08ks2Mp3Rshbv4da
Make sure that the string does not include any spaces or newlines when you copy it from the terminal. The tenant token will be used in the following sections.
You can repeat the steps if you would like to isolate devices into multiple organizations.
The tenant token needs to be included in the client configuration of each device that is going to be managed by Mender. Exactly how to include the token depends on which integration method is used with the client. Please refer to one of these sections:
Once the Enterprise configuration is complete, commit all changes to the repository:
git add config/enterprise.ymlgit commit -m 'production: Enterprise configuration'At this point your commit history should look as follows:
git log --oneline 2.4.0b1..HEAD76b3d00 production: Enterprise configuration 7a4de3c production: configuration 41273f7 production: adding generated keys and certificates 5ad6528 production: initial template
To verify that the services are running, execute the following command and verify that the state of all services is "Up":
./run psBelow you can see typical output for the Enterprise server. The Open Source server will be similar, but will have fewer services running.
Name Command State Ports ------------------------------------------------------------------------------------------------------------------------------- menderproduction_mender-api-gateway_1 /entrypoint.sh Up 0.0.0.0:443->443/tcp, 80/tcp menderproduction_mender-create-artifact-worker_1 /usr/bin/workflows --confi ... Up 8080/tcp menderproduction_mender-deployments_1 /entrypoint.sh --config /e ... Up 8080/tcp menderproduction_mender-device-auth_1 /usr/bin/deviceauth --conf ... Up 8080/tcp menderproduction_mender-gui_1 /entrypoint.sh nginx Up (healthy) 80/tcp menderproduction_mender-inventory_1 /usr/bin/inventory --confi ... Up 8080/tcp menderproduction_mender-mongo_1 docker-entrypoint.sh mongod Up 27017/tcp menderproduction_mender-useradm_1 /usr/bin/useradm --config ... Up 8080/tcp menderproduction_mender-workflows-server_1 /usr/bin/workflows --confi ... Up 8080/tcp menderproduction_mender-workflows-worker_1 /usr/bin/workflows --confi ... Up menderproduction_minio_1 /usr/bin/docker-entrypoint ... Up (healthy) 9000/tcp menderproduction_storage-proxy_1 /usr/local/openresty/bin/o ... Up 0.0.0.0:9000->9000/tcp
At this point you can try to log into the Web UI using the URL of your server, and the username and password that was created earlier.
It is advised to use the UI to change the password of all users that were added in this guide, since the shell may keep a record of the password in plaintext.
If you encounter any issues while starting or running your Mender Server, you can take a look at the section for troubleshooting Mender Server.