3. Sync Appliance install¶
The Sync Appliance runs on x86-64 Linux systems atop the Docker Engine user-space isolation layer so as to ensure binary compatibility across Linux distributions, allow atomic upgrades of all its components, and decrease the attack surface by executing them in restricted environments.
The installation script has been tested on the following platforms:
- Debian 8, Debian 9
- Ubuntu 12.04 LTS, 14.04 LTS, 16.04 LTS
- Centos 7
The installation script will:
- install Docker Engine and its dependencies
- launch Docker Engine and configure it to start at boot time
- download and install the Æoncase Sync Appliance launcher/updater
- start the basic configuration program, which is only accessible on localhost
The Æoncase Sync Appliance will appear as 2 Docker containers, corresponding to:
- the launcher/updater and configuration system, responsible for downloading the actual appliance and keeping it up-to-date, and allowing the initial configuration
- the sync appliance, containing the sync server, the embedded webserver, and other internal services
The configuration system will bind to a localhost port, indicated by the installation script.
The sync appliance binds by default to the standard HTTP (80) and HTTPS (443) ports. UDP ports in the range 13010 to 13013 are used to allow clients in the local area network to connect directly to the server.
The sync protocol and regular HTTPS access are automatically multiplexed over the same port using server name indication (SNI).
On successful install, the appliance will run as self-contained docker containers, with a few additional changes to the host environment:
a aeoncase user will be created in order to drop root privileges when running appliance services
appliance state will be saved in /var/lib/aeoncase:
local storage backend, used as fallback when the main storage backend is down. Its data will be moved to the main storage backend when it becomes available.
support DBs of smaller size
If file data is to be stored on a remote filesystem, /var/lib/aeoncase/data should be mounted before the installation procedure is started.
/etc/aeoncase will hold files containing appliance configuration information
temporary data will be saved (and cleaned up automatically on a periodic basis) in the following directories:
Sync Appliance package data, downloaded by the updater
data in transfer or recently transferred
files being uploaded to the web server
locally available appliance instances (containers) will be stored under /var/lib/docker. A Sync Appliance container requires around 300 MB, and a few are kept by the updater so as to allow easy up/downgrades.
File data under /var/lib/aeoncase/data is stored under 256 directories named 00 to ff (i.e., 0-255 in hexadecimal). It is possible to use a crude form of tiered storage by mounting different disks on them, but it is not recommended.
3.1.1. Storage backend¶
File data is kept in the main storage backend under /var/lib/aeoncase/data. By mounting a distributed or remote file system (e.g. NFS) on this directory, it is trivially possible to extend storage beyond the local disk capacity.
The storage appliance makes mimimal assumptions about the underlying file system: no lock operations are required, and in practice the only hard requirement is atomic renames (i.e., standard POSIX rename semantics). The storage backend is unaffected by directory entry caching: read operations for data known to exist (as indicated in the internal database) will be retried until the read succeeds.
The Sync Appliance is designed to withstand temporary storage backend outages, refer to Resilience against storage backend unavailability.
Special care is taken to ensure storage backend unavailability does not affect negatively the system:
potentially blocking operations (read/write to the backend) are performed in separate threads so as to avoid blocking the whole application
when an operation times out (after 3 seconds), the backend is marked as unavailable and:
- pending write operations are diverted to the fallback backend
- pending read operations against the main storage backend wait until it becomes available again
further operations are directly diverted or stalled similarly until the Sync Appliance detects the storage backend is available
the Sync Appliance tolerates file system unavailability being reported by the OS both as I/O errors (e.g. NFS soft mounts) and as hanging operations (e.g. NFS hard mounts). In the latter case, care is taken to put a bound on the number of ongoing operations so as to minimize file descriptor usage and the number of threads blocked in kernel space (only up to 2 I/O operations are allowed to be blocked in a system call; other operations are queued internally at very little cost).
3.1.2. Network connectivity and outgoing connections¶
In normal operation, the appliance will occasionally connect to aeoncase.com‘s servers in these circumstances:
- to relay notifications to users
- to allow sync clients to locate and link to the appliance
- to look for updates periodically and download them when available
- in order to allow operation without public IP address, domain name or SSL certificate
(1) allows notifications to work “out of the box” without any need for outgoing email configuration (SMTP or other mechanism).
(2) uses aeoncase.com‘s servers as a central registry where clients can look for connection parameters used to reach and authenticate against the Sync Appliance. Such parameters (which include the self-signed certificate generated locally on install and used by the appliance’s sync server) are encrypted using the link code presented to the user on the client download page, and associated in the central registry to the result of applying a one-way (hash) function to such code.
In order to launch the installation procedure, run the following as root:
# curl -sSL https://get.aeoncase.com/repos/install.sh | bash
If you normally run commands as root using sudo, you can become root with:
$ sudo -i
Alternatively, download the installation script, inspect it at your leisure and execute it manually:
# wget https://get.aeoncase.com/repos/install.sh # less install.sh # bash install.sh
You might need to install curl and/or wget if they are not available, using e.g. yum install curl or apt-get install curl.
When the installation script completes, it will indicate a localhost URL, as well as instructions to access it via a SSH tunnel if the server is headless.
The basic configuration is a 3-step procedure to:
- validate the license key
- create the administrator account
- provide basic information required by the web and sync servers:
- the hostname the clients (web browsers and sync clients) can use to reach the server
- (optional) the X509 certificate used to secure TLS/SSL connections to the web service
- the maximum allowed upload file size
The administrator account password and the basic setup can be changed at any time.
The ports used for the appliance HTTP and HTTPS/sync services can only be changed via the configurator during the initial setup. Refer to Changing service ports to modify them once the initial setup is complete.
Changing the setup will restart the sync and web services, which will result in a sub-second service interruption. Sync clients will reconnect automatically, and browsers showing a self-updating page (like the dashboard or a project explorer) will reload it.
The supplied hostname will be used by the sync clients and serve as the domain name used by the embedded web server. If the services are to be reachable from the public Internet, proper DNS domains or public IPs are required.
If no X509 certificate is provided, the embedded web server will use a sample certificate (issued to the localhost CN) for deployment testing. In the final deployment, a proper X509 certificate signed by a CA recognized by the end user’s browsers is required. The configuration program will verify that the provided certificate matches the hostname.
3.3.1. Administrator account password recovery¶
The administrator account is also, as far as the appliance is concerned, a regular account (with sync abilities), so its password can be changed using the normal in-app password modification procedure.
If the credentials are lost, however, the administrator account can be changed by the root user in the appliance host as follows:
reset the admin account credentials on the host:
# rm -rf /etc/aeoncase/config/multi/AEONCASE_ADMIN_CREDENTIALS
go to the configurator and enter the new credentials
It is recommended to use the same username as in the original setup and to merely change the password. If a different one is provided, a new admin account will be created, and the old one will be left as-is (making it still possible to log into the system with the associated credentials).
3.4. Sync Appliance launch¶
The Sync Appliance is downloaded and installed by the launcher while the basic configuration is performed. Once complete, the service will be launched and should become available at the address (hostname) given during setup.
The launcher will detect when the appliance has started and show a link to the web service.
Depending on the firewall setup of the host, additional work might be needed to open the ports used by the sync appliance (e.g. HTTP 80 and HTTP 443).
3.4.1. WebDAV service¶
WebDAV access is enabled by default. It can be disabled explicitly with the aeoncasectl tool (see aeoncasectl tool).
Users will be able to access their projects (including the current state as well as all their snapshots) at
by default, or, if an alternative HTTPS port was specified,
using their regular login credentials.
Updates are performed automatically by the launch/update system. The update system follows The Update Framework‘s (TUF) design to provide secure updates using a comprehensive security framework that addresses all known attacks on the software update process and minimizes the impact of key compromise via well-defined roles with multiple keys.
In particular, it protects against these attacks:
- Rollback attacks
- Attackers should not be able to trick clients into installing software that is older than that which the client previously knew to be available.
- Indefinite freeze attacks
- Attackers should not be able to respond to client requests with the same, outdated metadata without the client being aware of the problem.
- Endless data attacks
- Attackers should not be able to respond to client requests with huge amounts of data (extremely large files) that interfere with the client’s system.
- Slow retrieval attacks
- Attackers should not be able to prevent clients from being aware of interference with receiving updates by responding to client requests so slowly that automated updates never complete.
- Extraneous dependencies attacks
- Attackers should not be able to cause clients to download or install software dependencies that are not the intended dependencies.
- Mix-and-match attacks
- Attackers should not be able to trick clients into using a combination of metadata that never existed together on the repository at the same time.
- Attacks on mirrors
- Malicious repository mirrors should not be able to prevent updates from good mirrors.
3.5.1. Update procedure¶
Updates are automatic and involve as of 1.19.1 a short unavailability window (usually a couple seconds at most).
Old appliance versions (their images and instances, in other words, containers) will be garbage collected automatically.
The update system keeps a small number of versions around for easy downgrading in case of need.
The versions available locally can be listed using the aeoncasectl tool, see aeoncasectl tool.
Older apppliance (and updater) versions will also appear as docker images and can be listed with:
# docker images
Direct manipulation using docker commands is not recommended: the aeoncasectl tool is both easier and safer for common usages.
3.6. Load testing¶
The Æoncase Sync Appliance includes a load generator tool that allows to analyze system performance when subject to thousands of simultaneous connections (users/devices), both inactive and engaged in active file sync.
3.6.1. Workload generation¶
The aeoncase-loadtest load generator is available as /opt/aeoncase/aeoncase-loadtest in the server where the appliance was installed. For more reliable results, a separate machine should be used to generate the system load. aeoncase-loadtest is self-contained and can be copied to and used in any x86-64 Linux system.
aeoncase-loadtest uses the same sync engine as a regular client, and as far as the server is concerned it behaves like a large number of clients connecting at once. It requires about the same resources as the regular client, which, while limited, quickly add up when several thousand clients are simulated:
- around 1 MB of RAM at startup
- 1 inotify instance
- up to 20 file descriptors per sync process
The server, on the other hand, will only require 100 to 200 KB of RAM per connection.
3.6.2. Test procedure¶
Correct testing requires some tuning in the load generator machine to ensure the resource limits indicated via kernel parameters are high enough.
On the load generator machine, as root:
ensure the fs.file-max system-wide limit is high enough:
sysctl -w fs.file-max=500000
This setting can be made permanent by specifying it in /etc/sysctl.conf.
edit /etc/security/limits.conf to raise the file descriptor limits for the user that will run the tests (loadtester in the following) by appending a line like the following:
loadtester hard nofile 400000
ensure the inotify resource limits are high enough:
# sysctl -w fs.inotify.max_user_instances=100000 # sysctl -w fs.inotify.max_user_watches=500000
These limits can be made persistent across system reboots by specifying them in /etc/sysctl.conf.
on RHEL/CentOS: make sure /etc/pam.d/login includes the following line:
session required pam_limits.so
On the Sync Appliance server, as root:
get the certificate used by the sync service with
/opt/aeoncase/aeoncasectl cert > ca.pem
transfer the PEM certificate (ca.pem in the following) to the load generator machine
The load test will require a high number of accounts so that aeoncase-loadtest can initiate the corresponding sync processes. They are generated on the Sync Appliance server as follows:
# /opt/aeoncase/aeoncasectl make-test-users --password SOMEPASSWORD \ --prefix loadtest01 2000
The above will create 2000 accounts named loadtest0100000 to loadtest0101999, all of them using the specified password (SOMEPASSWORD). You can verify that you can log into the web app using any of them (note however that such “test users” are not listed in the admin page).
On the load generator machine, as the system user that will run the tests (e.g. loadtester), make the new file descriptor resource limit effective:
$ ulimit -n 500000
If the user session started before the limits were raised in /etc/security/limits.conf, it will be necessary to log out and log in again before they can be made effective with ulimit.
Finally, on the directory holding the certificate retrieved above (ca.pem) and the aeoncase-loadtest executable, load generation is started as follows:
188.8.131.52.1. Inactive connections¶
Launch 2000 sync processes that will sync the AEoncase directories created under loadtest01/USERNAME/:
$ ./aeoncase-loadtest --prefix loadtest01 --ca ca.pem \ --server SERVER_ADDRESS --password SOMEPASSWORD --dir loadtest01 2000
No user activity is generated (making this ideal for measuring stationary memory needs and CPU usage), but can be created manually by placing data under loadtest01/USERNAME/AEoncase/.
Connections will be ramped up until reaching the target. On the first run, the process will be fairly slow (a few per second) since client-side initialization and key generation is required. Subsequent runs will be faster.
184.108.40.206.2. Active connections¶
Workload corresponding to user activity following a Poisson model for N active users (out of e.g. 2000 connected ones) can be generated with the --active N parameter:
$ ./aeoncase-loadtest --active N --prefix loadtest01 --ca ca.pem \ --server SERVER_ADDRESS --password SOMEPASSWORD --dir loadtest01 2000
It is possible to measure raw sync speed (change propagation, transaction processing, etc.) independently from the I/O load imposed on the storage subsystem with the --reuse-data option. This allows to assess the maximum throughput (limited by CPU and database I/O) of a single-node server when it is using local storage (instead of e.g. a remote NFS mount) during initial load testing. Conversely, the extra I/O load when not using --reuse-data can be attributed to the work required to save each synced file on disk. This component of the workload can be scaled trivially by using more performant remotely mounted storage such as network-attached storage (NAS).
Refer to aeoncase-loadtest‘s documentation for further options:
$ ./aeoncase-loadtest --help
NAME aeoncase-loadtest - generate sync load test SYNOPSIS aeoncase-loadtest --server SERVER --password PASSWORD [OPTION]... USERS DESCRIPTION Establish USERS simultaneous connections to the SERVER and generate sync activity corresponding to the specified number of active users. USER MODEL The generated user activity follows a Poisson model, with each user uploading on average 5 files (of 10 KB average size) every 10 minutes. RESOURCE NEEDS The load generator uses the same sync engine as the regular client. The requirements per user/connection are: up to 20 file descriptors, one inotify instance, around 1 MB of RAM. Make sure the file descriptor limit is high enough (e.g. by editing /etc/security/limits.conf as root, logging in as the test user, and setting ulimit -n) and the inotify limits are set correctly (e.g. sysctl -w fs.inotify.max_user_instances=100000 and sysctl -w fs.inotify.max_user_watches=200000 as root). ARGUMENTS --ca=FILE file holding the CA certificate (PEM format; use aeoncasectl cert to get it) -p PASSWORD, --password=PASSWORD password used to authenticate users -s SERVER, --server=SERVER server to connect to USERS total number of connections (= users) to establish OPTIONS --active=N (absent=0) number of active users -d PATH, --dir=PATH directory to create synced dirs under (default: current dir); a directory named after the corresponding user will be created for each connection, under the specified dir. --help[=FMT] (default=pager) Show this help in format FMT (pager, plain or groff). --passphrase=PASSPHRASE passphrase used for client secrets (client-side encryption keys, user key) (default: password repeated) --port=PORT port to connect to (default 443) --prefix=PREFIX (absent=test) prefix for user names (the user names will be of the form PREFIX00001) --reuse-data use limited number of data patterns (allows to factor out file storage I/O) --version Show version information.