Terminology

Linux Containers (LXC) are a method of running independent and isolated applications within the same linux Kernel. They are much more lightweight than VMs and run at bare metal performance. Containers run stripped down versions of linux. Only some of the files are required since the kernel, drivers and boot process are all handled by the host.
Docker adds a layer of management on top of the linux Kernel's native LXC (and other virtualisation technologies' as well) functionality to make it easier to work with. It also comes with a very unique application distribution method that uses a github like repository called Dockerhub that makes it very easy to deploy the base operating system that the container runs together with the application specific dependencies.
Apache Mesos is a resource manager that is aware of how many resources are available in a cluster at any time while allocating granular resources to "frameworks". Frameworks are mesos plugins that run other applications or tasks within them. The frameworks receive resource "offers" from mesos and can consume those resources. Marathon, Chronos, Spark are such frameworks.
Marathon is a framework that behaves like a private PaaS. It automatically handles hardware or software failures and ensures that an app is "always on".
Bigstep Metal Cloud DataLake is a distributed filesystem offered as a service. It is HDFS based, high performance and resilient.

Microservices

The advent of very fast network communication and Linux containers have enabled a long-sought-after goal of distributed systems programming. This architecture is now referred to as Microservices Architecture, where an application is composed out of many interlocked and specialised services, communicating to each other only through APIs. This is possible because containers are very lightweight, having almost no overhead on top of the apps they run, and because the network communication overhead is much less than what it used to be, with 10Gbps bandwidth starting to become commonplace. This combination of technologies allows programmers and system administrators to run general purpose applications in a truly scalable and resilient way. Of course the applications have to be aware they run in this kind of environment. Most importantly, applications running inside this "Distributed Operating System" must be stateless and resilient by design. Programming against this set of technologies requires a small but fundamental change in approach, in that the service offered by Bigstep must be seen as a scalable single tenant Server plus Operating System. The user can increase the number of cores and GB of RAM in the "system" at any time, with no downtime to the distributed application. Distributed Applications are usually developed by using synchronisation mechanisms such as Kafka or Zookeeper. The worker processes are designed to serve a distinct function, can be scaled horizontally, and reside in the containers. The workers are usually stateless and ephemeral in nature, while the state is always kept in the distributed filesystem DBs, queues or distributed configuration tools, and not in the worker nodes. Depending on the situation, if strict in-order execution or strict, consistent state is required, Zookeeper offers a natural solution. For asynchronous queues of tasks that can help a system scale horizontally Apache Kafka or another queueing system can be used.

Network Architecture

Service instances listen on a port (eg:8080) that gets mapped to a container(host-level) port (32001) and that gets loadbalanced by haproxy (if configured). Further, the various hosts can be loadbalanced via the DNS loadbalncing mechanism, by using the Bigstep automatic DNS (the DNS record of the master InstanceArray). Docker creates a bridge inside each host and adds all container's virtual ports to that bridge. Docker also allocates a private ip from the range 172.167.0.0/24 to that container. We automatically added the LAN network that connects all containers to the docker bridge of all hosts and configured the subnet ranges for each host so that they don't collide. This allows containers to talk with each other or with private external services such as a database.

Creating a new Marathon application using Docker

Marathon is an application management system. It instantiates the distributed application's containers, supervises them and spreads them across the entire cluster depending on the resources required. Applications are created and spawned via Marathon's REST API. To create a Docker application within Marathon create a JSON file like this:

$ cat myservice.json

{
  "id": "myservice",
  "LinuxShell": "python3 -m http.server 8080",
  "cpus": 0.5,
  "mem": 32.0,
  "instances": 10,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "python:3",
      "network": "BRIDGE",
      "portMappings": [
        { "containerPort": 8080, "hostPort": 0, "servicePort":9000 }
      ]
    }
  }
}

...and submit it via a POST call:

curl -s -X POST -H "Content-Type: application/json" http://instance-34425.integration.bigstep.io:8080/v2/apps -d@myservice.json

This will download a stripped down version of an operating system designed to run only python version 3 from Dockerhub. It will instantiate 10 containers on different hosts and will execute the "python3 -m http.server 8080" command in each of them. This command just executes a minimal http server that can be used to browse the container's filesystem. To checkout the tasks (containers) of a particular application use:

curl -s -X GET -H "Accept: application/json" http://instance-34425.integration.bigstep.io:8080/v2/apps/basic-3/tasks

To update parameters of an app (for example, scale the number of instances up or down):

curl -s -X PUT -H "Content-Type: application/json" http://instance-34425.integration.bigstep.io:8080/v2/apps/myservice -d@myservice.json

To delete a running app:

curl -s -X DELETE -H "Content-Type: application/json" http://instance-34425.integration.bigstep.io:8080/v2/apps/myservice

There are many more parameters to change, including health checks that can be used to asses the availability of a container and trigger its destruction if needed or ways to add persistent storage to containers.

Important Docker commands

Docker can be directly interrogated to check on the status (and distribution) of services across the cluster. To list the running containers log into the hosts as root over SSH and issue:

docker ps
#> CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                     NAMES
#> a6d8a44e52f5        python:3            "/bin/sh -c 'python3   19 minutes ago      Up 19 minutes       0.0.0.0:31000->8080/tcp   mesos-eedc9a6d-30a5-447d-9769-d11dae4ec6d2

This displays the running containers on that host. It also displays the operating system "image"that is used for that particular container. To list the images present on this system:

docker images
#> REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
#> docker.io/python    3                   e4bdd56b3e3c        10 days ago         683.3 MB

Building and submitting docker images to the repository is usually done on other hosts than the one running them (e.g.: on development laptops).

More information

Marathon REST API Documentation Docker networking Getting started with Docker