Docker - Build, Ship, and Run any App, Anywhere

1. get the container ID :
   docker ps

   CONTAINER ID	IMAGE				
   c031860c837e	sonatype/nexus:2.14.4-03	

2. inspect it :
   docker inspect c031860c837e

   (returns a wealth of JSON data)

3. for details about the host-side storage, search these keys :
   • MergedDir
   • LowerDir
   • UpperDir

"LowerDir": "/var/lib/docker/overlay2/__ID__-init/diff
             /var/lib/docker/overlay2/__ID__/diff
             /var/lib/docker/overlay2/__ID__/diff
             /var/lib/docker/overlay2/__ID__/diff
             /var/lib/docker/overlay2/__ID__/diff
             /var/lib/docker/overlay2/__ID__/diff",
"MergedDir": "/var/lib/docker/overlay2/__ID__/merged",
"UpperDir": "/var/lib/docker/overlay2/__ID__/diff",

• Overview
• What are these : bind mounts, volumes, --mount and --volume ?
  • What do bind mounts and volumes have in common ?
  • What differentiates bind mounts and volumes ?
  • What's the difference between --mount and --volume ?
• How to declare a bind mount ? A volume ?

Overview (source)

What are these : bind mounts, volumes, --mount and --volume ?

How to declare a bind mount ? A volume ?

version: '3.7'

services:
  db:
    volumes:
      - ./db:/var/lib/mysql

  wp:
    volumes:
      - ./wpb:/var/www/html

version: '3.1'

services:

  drupal:
    image: drupal:8.2-apache
    ports:
      - 8080:80
    volumes:
      - /var/www/html/modules
      - 

version: "3"

services:
  db:
    image: db
    volumes:			#1 use the named volume
      - myvol:/var/lib/db	myvol created with #3

  backup:
    image: backup-service
    volumes:			#2 use the named volume
      - myvol:/var/lib/backupmyvol created with #3

volumes:			#3 create the named
  myvol:			volume myvol

build

configuration options that are applied at build time

version: '3.8'
services:
  myService:
    build:
      context: './path/to/myService-7.3.1/'
    image: myService:7.3.1
    

other example

When using both build and image: imageName:imageTag, imageName:imageTag refers to the image built by Compose, not to the base image.

env_file (details : 1, 2)

• in docker-compose.yml :

    elastic:
      image: docker.elastic.co/elasticsearch/elasticsearch:7.14.0
      stop_grace_period: '5m'
      ports:
        - '9207:9200'
      volumes:
        - 'elastic_data:/usr/share/elasticsearch/data:rw'
        - './logs/elastic:/usr/share/elasticsearch/logs:rw'
      env_file:
        - './env/elastic.env'

• in ./env/elastic.env :
  The expected format is : NAME=value.

  cluster.name='talend7pp'
  ingest.geoip.downloader.enabled=false
  discovery.type='single-node'
  _JAVA_OPTIONS='-Xmx4g -Xms4g'

• which gives a file tree like :
  . project root
  docker-compose.yml
  .env
  env/
  service_1.env
  service_2.env
  service_n.env
  Whatever the location, environment variables will be loaded when running docker-compose up (details).

  difference between .env and env/serviceName.env files (source) :

  • .env :
    • default file to store environment variables + values
    • found in the same directory than docker-compose.yml
  • env/serviceName.env :
    • path must be specified :
      • in docker-compose.yml : with env_file
      • in CLI : with --env-file
    • path + file name env/serviceName.env is purely conventional
    • allows storing the environment file anywhere and naming it freely

environment

• dictionary format :

  environment:
    RACK_ENV: development
    SHOW: 'true'		quotes are necessary around boolean-likes in YAML files because of this and that
    SESSION_SECRET:

• array format :

  environment:
    - RACK_ENV=development
    - SHOW=true
    - SESSION_SECRET

ports

expose ports :

ports:
  - "hostPort:containerPort"

volumes

• short syntax :

  source:target:mode

  • source :
    • volume name
    • path on the host side
      • absolute path : (explicit)
      • relative path : relative to the directory containing the docker-compose.yml file
  • target : path in the container (i.e. mount point)
  • mode :
    • ro
    • rw (default)
• long syntax
• driver : you may also see code like :

  volumes:
    my_named_volume:
      driver: local

  • local :
    • means the corresponding volume will be created on the same host where the container is running. This is the default setting and can be omitted in most cases (sources : 1, 2).
    • accepts options similar to the mount command when running on GNU/Linux (details)

• ADD
• ARG
• COPY
• EXPOSE
• FROM
• RUN
• USER
• VOLUME
• WORKDIR

ADD

ADD [--chown=kevin:developers] source destination

• copy source into the filesystem of the image at the destination path
• destination can either be :
  • an absolute path
  • a path relative to WORKDIR
• About .zip files (sources : 1, 2 ) :
  • looks like there is no —and never will be— a solution to copy+extract .zip files with ADD, like it actually works for .tar.gz & al
  • workarounds :
    • unpack + repack the .zip archive into .tar.gz format, then ADD it
    • COPY file.zip + RUN unzip file.zip + RUN rm file.zip
• ADD or COPY ?

ARG

• defines a variable that users can pass at build-time (example) :
  • Dockerfile :

    
    ARG myVariable
    

  • build command :
    docker build --build-arg myVariable=value
• it is possible to define an optional default value, which will be used when no value is provided at build-time :
  ARG MYVARIABLE=42
• the variable name is often written uppercase, but I've found no rule enforcing this.
• the defined variable lives (details) :
  • from the line it is defined (it can override an existing variable)
  • until the end of the build stage (end of Dockerfile or subsequent FROM)
• since every build stage :
  • starts with FROM
  • ends with
    • the next FROM (if any)
    • the end of the Dockerfile
  ARG must not appear before FROM
• the value of a variable can be used with the $ prefix (like shell variables, source) :

  ARG myVariable=42
  RUN echo $myVariable

COPY

When (trying to) COPY files into an existing directory :

COPY someFile /path/to/destination

• will not result in /path/to/destination/someFile
• will result in /path/to/destination being a regular file : this is someFile copied there + renamed into destination

Workaround :

COPY someFile /path/to/destination/

EXPOSE

• informs Docker that the container listens on the specified network ports at runtime. More explicitly, EXPOSE portNumber states that something inside the container is listening on port portNumber.
• How this internal port will be plugged to the external world will be defined when actually running the container, with docker run flags (details) :
  • -P
  • -p
• you can specify whether the port listens on TCP (default) or UDP
• EXPOSE :
  • does not enable communication on the specified port(s) (aka it does not publish the port(s))
  • is just a kind of "documentation" between the person who builds the image and the person who runs the container about which ports are intended to be published

FROM

• initializes a new build stage on top of baseImage for extra instructions
• ARG is the only instruction that may precede FROM (Understand how ARG and FROM interact)
• AS newName can be specified to refer to this new build stage in a subsequent FROM
• examples :
  • FROM baseImage [AS newName]
  • FROM baseImage:tag [AS newName]

RUN

USER

sets the user name (or UID) and optionally the user group (or GID) to use

• when running the image
• and for any RUN, CMD and ENTRYPOINT instructions that follow it in the Dockerfile

VOLUME

• there are 2 syntaxes :
  • "string" syntax :
    • VOLUME /data
    • VOLUME /volume1 /volume2
  • YAML "array" syntax :
    • VOLUME [ "/data" ]
    • VOLUME [ "/volume1", "/volume2" ]
• whatever syntax is used, specifying 2 directories must not be mistaken with some kind of "mount" operation : VOLUME /dir1 /dir2 :
  • does not "mount" /dir1 into /dir2 (or vice-versa)
  • is just a list of 2 directories (inside the container) to be used as 2 anonymous volumes
• paths specified here are all inside the container
• since WORKDIR defaults to /, VOLUME /data and VOLUME data may/may not refer to the same thing depending on circumstances

WORKDIR

• sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow it in the Dockerfile
• If the WORKDIR doesn't exist, it will be created even if it's not used in any subsequent Dockerfile instruction.

• Docker is very convenient to experiment all sort of things on containers as if we were provisioning VMs + playing with them + destroying them when the test is over (this is actually what I've tried to achieve here). It is especially nice to play with software without breaking things.
• AppArmor is a security framework that prevents applications from turning evil.

Looks like both would fit pretty well with each other. So what's the problem ?

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application's services. Then, with a single command, you create and start all the services from your configuration.

Setup :

Let's build things with it (source) :

version: '3.7'
services:

  ubuntu_web1:
    hostname: web1
    build: .				this service uses the image built from the Dockerfile found in this directory (details)
    container_name: c_ubuntu_web1
    tty: true

  ubuntu_web2:
    hostname: web2
    build: .
    container_name: c_ubuntu_web2
    tty: true

  ubuntu_db1:
    hostname: db1
    build: .
    container_name: c_ubuntu_db1
    tty: true

  ubuntu_db2:
    hostname: db2
    build: .
    container_name: c_ubuntu_db2
    tty: true

Building ubuntu_db2
Step 1/11 : FROM ubuntu:16.04
 ---> 5e13f8dd4c1a
Step 2/11 : ARG userName='ansible'
 ---> Using cache
 ---> 7f163145edfd
Step 3/11 : ARG homeDir="/home/$userName"
 ---> Using cache
 ---> 9025b899c1df
Step 4/11 : ARG sshDir="$homeDir/.ssh"
 ---> Using cache
 ---> b32d4b7d1abf
Step 5/11 : ARG authorizedKeysFile="$sshDir/authorized_keys"
 ---> Using cache
 ---> 8477ab5bcac4
Step 6/11 : ARG publicSshKey='./ansible.pub'
 ---> Using cache
 ---> 5b2235eb9fbf
Step 7/11 : RUN apt-get update &&       apt-get install -y iproute2 iputils-ping openssh-server &&      apt-get clean &&        useradd -d "$homeDir" -s /bin/bash -m "$userName" &&    mkdir -p "$sshDir"
 ---> Using cache
 ---> 41cfeaaa7dda
Step 8/11 : COPY "$publicSshKey" "$authorizedKeysFile"		this step doesn't like it when source files are symlinks
 ---> Using cache
 ---> 3dcab2010a2b
Step 9/11 : RUN chown -R "$userName":"$userName" "$sshDir" &&   chmod 700 "$sshDir" &&  chmod 600 "$authorizedKeysFile"
 ---> Using cache
 ---> 1f511354cd72
Step 10/11 : EXPOSE 22
 ---> Using cache
 ---> deb5199624a4
Step 11/11 : CMD [ "sh", "-c", "service ssh start; bash"]
 ---> Using cache
 ---> 3bb266505c4c

Successfully built 3bb266505c4c
Successfully tagged dockercompose_ubuntu_db2:latest
WARNING: Image for service ubuntu_db2 was built because it did not already exist. To rebuild this image you must use docker-compose build or docker-compose up --build.

Creating c_ubuntu_web1 ... done
Creating c_ubuntu_db1  ... done
Creating c_ubuntu_web2 ... done
Creating c_ubuntu_db2  ... done
Attaching to c_ubuntu_db1, c_ubuntu_web1, c_ubuntu_web2, c_ubuntu_db2
c_ubuntu_db1   |  * Starting OpenBSD Secure Shell server sshd            [ OK ]
c_ubuntu_web1  |  * Starting OpenBSD Secure Shell server sshd            [ OK ]
c_ubuntu_web2  |  * Starting OpenBSD Secure Shell server sshd            [ OK ]
c_ubuntu_db2   |  * Starting OpenBSD Secure Shell server sshd            [ OK ]

CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS               NAMES
76f413236d70        dockercompose_ubuntu_db2    "sh -c 'service ssh …"   3 minutes ago      Up 3 minutes       22/tcp              c_ubuntu_db2
240e1505313e        dockercompose_ubuntu_web2   "sh -c 'service ssh …"   3 minutes ago      Up 3 minutes       22/tcp              c_ubuntu_web2
b5b6887d2b02        dockercompose_ubuntu_db1    "sh -c 'service ssh …"   3 minutes ago      Up 3 minutes       22/tcp              c_ubuntu_db1
95f988a02ff2        dockercompose_ubuntu_web1   "sh -c 'service ssh …"   3 minutes ago      Up 3 minutes       22/tcp              c_ubuntu_web1

REPOSITORY                  TAG                 IMAGE ID            CREATED            SIZE
dockercompose_ubuntu_db1    latest              3bb266505c4c        4 minutes ago      205MB
dockercompose_ubuntu_db2    latest              3bb266505c4c        4 minutes ago      205MB
dockercompose_ubuntu_web1   latest              3bb266505c4c        4 minutes ago      205MB
dockercompose_ubuntu_web2   latest              3bb266505c4c        4 minutes ago      205MB

Now let's use this "virtualized infrastructure" :

Procedure (source) :

Post-install steps :

container (source)

• runtime instance of an image (i.e. what the image becomes in memory when actually executed).
• It runs completely isolated from the host environment by default, only accessing host files and ports if configured to do so via a Dockerfile.
• Containers run apps natively on the host machine's kernel. They have better performance characteristics than virtual machines that only get virtual access to host resources through a hypervisor. Containers can get native access, each one running in a discrete process, taking no more memory than any other executable.
• Container != VM

Dockerfile (source, directives reference)

text file containing all the commands you would normally execute manually in order to build a Docker image :

• what's inside / outside of the image
• from what + how the image is built
• how the app inside the image interacts with the rest of the world (i.e. access to resources such as network, storage, etc)
• some environment variables
• what to do when the image launches
• ...

image

An image is static and lives only on disk (source).
• An image is an executable package that includes everything needed to run an application (source) :
  • the code of the application
  • a runtime
  • libraries
  • environment variables
  • config files
• An image typically contains a union of layered filesystems stacked on top of each other. An image does not have state and it never changes.

Union file system (source)

• Docker images are stored as series of read-only layers. When we start a container, Docker takes the read-only image and adds a read-write layer on top. If the running container modifies an existing file, the file is copied out of the underlying read-only layer and into the top-most read-write layer where the changes are applied. The version in the read-write layer hides the underlying file, but does not destroy it : it still exists in the underlying layer.
• When a Docker container is deleted, relaunching the image will start a fresh container without any of the changes made in the previously running container. Those changes are lost.
• Docker calls this combination of read-only layers with a read-write layer on top a Union File System. (more about union mount)

volume

• a special directory within one or more containers that bypasses the Union File System
• a volume :
  • is designed to persist data, independent of the container's life cycle
  • is a directory of the host filesystem that is mounted into a container
  • may be anonymous or named
  details
• since :
  • the host path may not exist on all systems
  • Docker images are expected to be portable
  the host path for a volume cannot be specified from a Dockerfile. Instead, it can be specified :
  • as a CLI argument when playing with Docker alone
  • within docker-compose.yml otherwise
• context and keywords :
  • while building an image : VOLUME in Dockerfile
  • while running an image : volumes in docker-compose.yml
• More explanations :
  • https://docs.docker.com/storage/volumes/
  • Understanding Volumes in Docker
  • Understanding docker -v command
• Let's play with volumes :
  • List the running containers :
    docker ps

    CONTAINER ID	IMAGE			COMMAND			CREATED		STATUS		PORTS					NAMES
    
    f64391d2ae1b	atlassian_confluence	"./entrypoint.sh"	5 days ago	Up 4 days	0.0.0.0:8090-8091->8090-8091/tcp	atlassian_confluence_1
    
    

  • Get details about their volumes :
    docker inspect -f "" atlassian_confluence_1 | grep -A7 '"Type": "volume"'

    
    	"Type": "volume",
    	"Name": "atlassian_confluence_tmp",
    	"Source": "/var/lib/docker/volumes/atlassian_confluence_tmp/_data",
    	"Destination": "/tmp",
    	"Driver": "local",
    	"Mode": "rw",
    	"RW": true,
    	"Propagation": ""
    --
    	"Type": "volume",
    	"Name": "atlassian_confluence_data",
    	"Source": "/var/lib/docker/volumes/atlassian_confluence_data/_data",	the Source is inside the container (see Mountpoint below)
    	"Destination": "/var/atlassian/application-data/confluence",		the Destination is on the host
    	"Driver": "local",
    	"Mode": "rw",
    	"RW": true,
    	"Propagation": ""

  • More details about a single volume :
    docker volume inspect atlassian_confluence_data

    [
    	{
    		"CreatedAt": "2020-10-08T23:09:51+02:00",
    		"Driver": "local",
    		"Labels": {
    			"com.docker.compose.project": "atlassian",
    			"com.docker.compose.version": "1.21.2",
    			"com.docker.compose.volume": "confluence_data"
    		},
    		"Mountpoint": "/var/lib/docker/volumes/atlassian_confluence_data/_data",
    		"Name": "atlassian_confluence_data",
    		"Options": null,
    		"Scope": "local"
    	}
    ]

Container != VM (source) :

The underlying kernel is shared by all containers on the same host. The container-specific stuff is implemented via kernel namespaces. Each container has its own :

• process tree (pid namespace)
• filesystem (mnt namespace)
• network namespace
• RAM allocation + CPU time

Everything else is shared with the host : in general the host machine sees / contains everything inside the container from file system to processes etc. You can issue a ps on the host and see processes running inside the container (source).

Docker containers are not VMs hence everything is actually running natively on the host and is using the host kernel directly. Each container has its own user namespace (like good ol' chroot jails). There are tools / features which make sure containers only see their own processes, have they own file-system layered onto host file system and a networking stack which pipes to the host networking stack.

About Docker itself :

Handle images :

Handle containers :

Miscellaneous :