You are browsing documentation for a version other than the latest stable release. Switch to the latest stable release, 2.2.
You can save time by using Hosted Mender; a secure Mender server ready to use, maintained by the Mender developers.
This is a step by step guide for deploying the Mender Server for production environments,
and will cover relevant security and reliability aspects of Mender production installations.
The Mender backend services can be deployed to production using a skeleton provided
in the template
directory
of the integration
repository.
mender.example.com
and s3.example.com
) that resolve to the public
IP of current host on the devices.It is very likely possible to use other Linux distributions and versions. However, we recommend using this exact environment for running Mender because it is known to work and you will thus avoid any issues specific to your environment if you use this reference.
The guide will use a publicly available Mender integration repository. Then proceed with setting up a local branch, preparing deployment configuration from a template, committing the changes locally. Keeping deployment configuration in a git repository ensures that the history of changes is tracked and subsequent updates can be easily applied.
At the end of this guide you will have:
docker-compose
fileConsult the section on certificates and keys for details on how the certificates and keys are used in the system.
docker-compose
implements a particular naming scheme for containers, volumes
and networks that prefixes each object it creates with project name. By default,
the project is named after the directory the main compose file is located in.
The production configuration template provided in the repository explicitly sets
project name to menderproduction
.
docker-compose
automatically assigns a
com.docker.compose.project=menderproduction
label to the created containers. The
label can be used when filtering output of commands such as docker ps
.
Start off by cloning the Mender integration repository into a local directory
named mender-server
:
git clone -b 2.0.1 https://github.com/mendersoftware/integration mender-server
Cloning into 'deployment'... remote: Counting objects: 1117, done. remote: Compressing objects: 100% (11/11), done. remote: Total 1117 (delta 1), reused 0 (delta 0), pack-reused 1106 Receiving objects: 100% (1117/1117), 233.85 KiB | 411.00 KiB/s, done. Resolving deltas: 100% (678/678), done. Checking connectivity... done.
Enter the directory:
cd mender-server
Prepare a branch where all deployment related changes will be kept:
git checkout -b my-production-setup
Copy deployment configuration template to a new directory named production
:
cp -a template production
Enter the directory:
cd production
ls -l
total 12 -rw-rw-r--. 1 user user 4101 01-26 14:06 prod.yml -rwxrwxr-x. 1 user user 161 01-26 14:06 run
The template includes 2 files:
prod.yml
- contains deployment-specific configuration, builds on top of
docker-compose.yml
(located at the root of integration repository)
run
- a convenience helper that invokes docker-compose
by passing the required
compose files; arguments passed to run
in command line are forwarded
directly to docker-compose
First we need to update paths that are mounted from the root of integration repository:
sed -i -e 's#/template/#/production/#g' prod.yml
At this point all changes should be committed to the repository:
git add .
git commit -m 'production: initial template'
[my-production-setup 556cc2e] production: initial template 2 files changed, 110 insertions(+) create mode 100644 production/prod.yml create mode 100755 production/run
Assuming that current working directory is still production
, download
necessary Docker images:
./run pull
Pulling mender-mongo (mongo:3.4)... 3.4: Pulling from library/mongo Digest: sha256:aff0c497cff4f116583b99b21775a8844a17bcf5c69f7f3f6028013bf0d6c00c Status: Image is up to date for mongo:3.4 ... Pulling mender-gui (mendersoftware/gui:latest)... latest: Pulling from mendersoftware/gui b7f33cc0b48e: Already exists 31e101b48355: Pull complete 307a023cd01f: Pull complete 2edcf3035646: Pull complete aeb59eb28f90: Pull complete ... Pulling mender-useradm (mendersoftware/useradm:latest)... latest: Pulling from mendersoftware/useradm Digest: sha256:3346985a2679b7edd9243363c4b1c291871481f8c6f557ccdc51af58dc6d3a1a Status: Image is up to date for mendersoftware/useradm:latest Pulling mender-device-auth (mendersoftware/deviceauth:latest)... latest: Pulling from mendersoftware/deviceauth Digest: sha256:ed47310dc9a86cca70d52c520d565ef9814f39421f2faa02d60bbe8a1dbd63c5 Status: Image is up to date for mendersoftware/deviceauth:latest Pulling mender-api-gateway (mendersoftware/api-gateway:latest)... latest: Pulling from mendersoftware/api-gateway Digest: sha256:97243de3da950e754ada73abf8c123001c0a0c8b20344256dc7db4c89c0ecd82 Status: Image is up to date for mendersoftware/api-gateway:latest
Using the run
helper script may fail when local user has insufficient permissions to reach a local docker daemon. Make sure that the Docker installation was completed successfully and the user has sufficient permissions (typically the user must be a member of the docker
group).
Prepare certificates using the helper script keygen
(replacing mender.example.com
and s3.example.com
with your DNS names):
CERT_API_CN=mender.example.com CERT_STORAGE_CN=s3.example.com ../keygen
Generating a 256 bit EC private key writing new private key to 'private.key' ... All keys and certificates have been generated in directory keys-generated. Please include them in your docker compose and device builds. For more information please see https://docs.mender.io/Administration/Certificates-and-keys.
Your local directory tree should now look like this:
├── keys-generated
│ ├── certs
│ │ ├── api-gateway
│ │ │ ├── cert.crt
│ │ │ └── private.key
│ │ └── server.crt
│ │ └── storage-proxy
│ │ ├── cert.crt
│ │ └── private.key
│ └── keys
│ ├── deviceauth
│ │ └── private.key
│ └── useradm
│ └── private.key
├── prod.yml
└── run
The production template file prod.yml
is already configured to load keys and
certificates from locations created by the keygen
script. If you wish to use a
different set of certificates or keys, please consult the
relevant documentation.
Next, we can add and commit generated keys and certificates:
Note that the following steps will commit your keys in plaintext to the git history. Make sure to keep all keys secure. Use encryption if needed and decrypt the keys before using them in containers; see the section below on encrypting keys for some examples.
git add keys-generated
git commit -m 'production: adding generated keys and certificates'
[my-production-setup fd8a397] production: adding generated keys and certificates 6 files changed, 108 insertions(+) create mode 100644 production/keys-generated/certs/api-gateway/cert.crt create mode 100644 production/keys-generated/certs/api-gateway/private.key create mode 100644 production/keys-generated/certs/server.crt create mode 100644 production/keys-generated/certs/storage-proxy/cert.crt create mode 100644 production/keys-generated/certs/storage-proxy/private.key create mode 100644 production/keys-generated/keys/deviceauth/private.key create mode 100644 production/keys-generated/keys/useradm/private.key
The API Gateway and Storage Proxy certificates generated here need to be made available to the Mender client. Consult the section on building for production for a description on how to include the certificates in the client builds.
Only certificates need to be made available to devices or end users. Private keys should never be shared.
A simple method for encrypting keys is to
use the GNU Privacy Guard tool. Instead of committing
plaintext keys to the repository, we will tar
and encrypt the entire
keys-generated
directory structure, then commit it to the repository as a
single binary blob.
The entire directory structure can be encrypted with the following command:
tar c keys-generated | gpg --output keys-generated.tar.gpg --symmetric
Enter passphrase:
rm -rf keys-generated
ls -l
total 20 -rw-rw-r--. 1 user group 5094 01-30 11:24 keys-generated.tar.gpg -rw-rw-r--. 1 user group 5519 01-30 11:08 prod.yml -rwxrwxr-x. 1 user group 173 01-27 14:16 run
Keys need to be decrypted before bringing the whole environment up:
gpg --decrypt keys-generated.tar.gpg | tar xvf -
gpg: AES encrypted data
Enter passphrase:
When using encryption, commit keys-generated.tar.gpg
instead of the
keys-generated
directory structure to the repository, like this:
git add keys-generated.tar.gpg
git commit -m 'production: adding generated keys and certificates'
[my-production-setup 237af44] production: adding generated keys and certificates 1 file changed, 111 insertions(+) create mode 100644 production/keys-generated.tar.gpg
Persistent storage of backend services' data is implemented using named Docker volumes. The template is configured to mount the following volumes:
mender-artifacts
- artifact objects storagemender-db
- mender services databases datamender-elasticsearch-db
- elasticsearch service database datamender-redis-db
- redis service database dataEach of these volumes need to be created manually with the following commands:
docker volume create --name=mender-artifacts
docker volume create --name=mender-db
docker volume create --name=mender-elasticsearch-db
docker volume create --name=mender-redis-db
Since we are using local driver for volumes, each volume is based on a host local directory. It is possible to access files in this directory once a local path is known. To find the local path for a specific volume run the following command:
docker volume inspect --format '{{.Mountpoint}}' mender-artifacts
/var/lib/docker/volumes/mender-artifacts/_data
The path depends on local docker configuration and may vary between installations.
The deployment configuration still needs some final touches. Open prod.yml
in
your favorite editor.
Locate the storage-proxy
service and add s3.example.com
(or your DNS name)
under networks.mender.aliases
key. The entry should look like this:
...
storage-proxy:
networks:
mender:
aliases:
- s3.example.com
...
You can also change the values for DOWNLOAD_SPEED
and MAX_CONNECTIONS
.
See the section on bandwidth for more details on these
settings.
Locate the minio
service. The keys MINIO_ACCESS_KEY
and MINIO_SECRET_KEY
control
credentials for uploading artifacts into the object store. Since Minio is a S3 API
compatible service, these settings correspond to Amazon's AWS Access Key ID and
Secret Access Key respectively.
Set MINIO_ACCESS_KEY
to mender-deployments
. MINIO_SECRET_KEY
should be set
to a value that can not easily be guessed. We recommend using the pwgen
utility
for generating the secret. Run the following command to generate a 16-character long secret:
pwgen 16 1
ahshagheeD1ooPae
The updated entry should look similar to this:
...
minio:
environment:
# access key
MINIO_ACCESS_KEY: mender-deployments
# secret
MINIO_SECRET_KEY: ahshagheeD1ooPaeT8lut0Shaezeipoo
...
Locate the mender-deployments
service. The deployments service will upload
artifact objects to minio
storage via storage-proxy
,
see the administration overview for more details. For this reason,
access credentials DEPLOYMENTS_AWS_AUTH_KEY
and DEPLOYMENTS_AWS_AUTH_SECRET
need to be updated and DEPLOYMENTS_AWS_URI
must point to s3.example.com
(or
your DNS name).
The address used in DEPLOYMENTS_AWS_URI
must be exactly the same as the one that will be used by devices. The deployments service generates signed URLs for accessing artifact storage. A different hostname or port will result in signature verification failure and download attempts will be rejected.
Set DEPLOYMENTS_AWS_AUTH_KEY
and DEPLOYMENTS_AWS_AUTH_SECRET
to the values
of MINIO_ACCESS_KEY
and MINIO_SECRET_KEY
respectively. Set
DEPLOYMENTS_AWS_URI
to point to https://s3.example.com:9000
.
The updated entry should look like this:
...
mender-deployments:
...
environment:
DEPLOYMENTS_AWS_AUTH_KEY: mender-deployments
DEPLOYMENTS_AWS_AUTH_SECRET: ahshagheeD1ooPaeT8lut0Shaezeipoo
DEPLOYMENTS_AWS_URI: https://s3.example.com:9000
...
Locate the mender-api-gateway
service. For security purposes, the gateway
must know precisely the DNS name allocated to it, which you'll configure via
the ALLOWED_HOSTS
environment variable.
Change the placeholder value my-gateway-dns-name
to a valid hostname; assuming it's
mender.example.com
, the updated entry should look like this:
...
mender-api-gateway:
...
environment:
ALLOWED_HOSTS: mender.example.com
...
Note that if for some reason you need more than 1 DNS name pointing to the gateway, you can supply a whitespace-separated list of hostnames as follows:
...
mender-api-gateway:
...
environment:
ALLOWED_HOSTS: mender1.example.com mender2.example.com
...
Locate the mender-device-auth
service.
Add the environment
section if it is absent. In the environment
section add the DEVICEAUTH_MAX_DEVICES_LIMIT_DEFAULT
variable with an integer value. 0
represents no limit
and is the default.
The updated entry should look like this:
...
mender-device-auth:
...
environment:
DEVICEAUTH_MAX_DEVICES_LIMIT_DEFAULT: 15
...
The setup uses Docker's default json-file
logging driver, which exposes two important log
rotation parameters: max-file
and max-size
. For selected services - consistently producing
a large amount of logs - we override these settings to ensure that at any given time, logs capture
about 1 week of a running system's operation.
The reference setup used for determining these parameters (max-file: 10
, max-size: 50m
)
was 200 devices, polling the server every 30 minutes. The total log volume produced was 2.4GB,
and this is the recommended amount of disk space to reserve for logging purposes alone.
These parameters can however be adjusted accordingly to a particular deployment and workload. The deciding factors determining the log volume are:
It is suggested to adjust log rotation parameters only after measuring the actual space usage for a given use case.
Once all the configuration is complete, commit all changes to the repository:
git add prod.yml
git commit -m 'production: final configuration'
At this point your commit history should look as follows:
git log --oneline origin/2.0.1..HEAD
7a4de3c production: final configuration 41273f7 production: adding generated keys and certificates 5ad6528 production: initial template
Bring up all services up in detached mode with the following command:
./run up -d
Creating network "menderproduction_mender" with the default driver Creating menderproduction_mender-mongo_1 Creating menderproduction_mender-gui_1 Creating menderproduction_minio_1 Creating menderproduction_mender-device-auth_1 Creating menderproduction_mender-inventory_1 Creating menderproduction_storage-proxy_1 Creating menderproduction_mender-useradm_1 Creating menderproduction_mender-deployments_1 Creating menderproduction_mender-api-gateway_1
Services, networks and volumes have a menderproduction
prefix, see the note about docker-compose naming scheme for more details. When using docker ..
commands, a complete container name must be provided (ex. menderproduction_mender-deployments_1
).
To verify that the services are running, you can issue the following command:
./run ps
Name Command State Ports
-------------------------------------------------------------------------------------------------------------
menderproduction_mender-api-gateway_1 /entrypoint.sh Up 0.0.0.0:443->443/tcp
menderproduction_mender-deployments_1 /entrypoint.sh Up 8080/tcp
menderproduction_mender-device-auth_1 /usr/bin/deviceauth -confi ... Up 8080/tcp
menderproduction_mender-gui_1 /entrypoint.sh Up
menderproduction_mender-inventory_1 /usr/bin/inventory -config ... Up 8080/tcp
menderproduction_mender-mongo_1 /entrypoint.sh mongod Up 27017/tcp
menderproduction_mender-useradm_1 /usr/bin/useradm -config / ... Up 8080/tcp
menderproduction_minio_1 minio server /export Up 9000/tcp
menderproduction_storage-proxy_1 /usr/local/openresty/bin/o ... Up 0.0.0.0:9000->9000/tcp
Furthermore, since this is a brand new installation it should be possible to create the initial user via the CLI provided by the User Administration Service. The service's binary is embedded in a Docker container, so to execute it you will issue the exec subcommand of docker-compose, e.g.:
./run exec mender-useradm /usr/bin/useradm create-user --username=myusername@host.com --password=mysecretpassword
Keep in mind that above is executed in a command-line interpreter meaning that certain characters might need to be escaped, e.g if you are using the $
character in your password, this could interpret as a variable name unless it is escaped.
At this point you should be able to access https://mender.example.com with your web browser.
If you encounter any issues while starting or running your Mender Server, you can take a look at the section for troubleshooting Mender Server.