Debian/Ubuntu Package Developing with Docker, Continued

Last time I blogged about that Docker being the best tool for developing Debian/Ubuntu packages. That was in principle generally speaking, and let’s get practical this time.


Docker advantages

To recap,

  • The reason Docker being the best tool for developing Debian/Ubuntu packages is that, it gives you isolated pure & pristine environments to build packages, and it allows you to build layers on top of each other, while giving you the freedom to play in any layer at any given time. Moreover, when it comes to Debian packages developing, the best part is that all your changes to the layer can be discarded at your will, so you can always start from the pristine stage of your layer.
  • We talked about constructing the first layer to be the bare-minimum installation system that contains not much more than a deb-bootstrap, which will be used to test our final package installation, to verify that the package install well on a bare-minimum system.
  • We also talked about strategically constructing layers on top of each other so as to best facilitate packages developing. Further than that, sharing resources among docker images is also briefly touched upon.

Now let’s see how thing are done.


Building base docker images

To build the fundamental bare-minimum installation system is very simple. For example, to build a Debian Sid base system, you just do:

mkimage-debootstrap.sh sfxpt/debian sid

the sfxpt is just a username that docker recommends you to use. I chose sfxpt for myself. You can replace it all with your chosen username and make it yours.

To build an Ubuntu docker image, likewise:

mkimage-debootstrap.sh sfxpt/ubuntu saucy

Of course, you need debootstrap and mkimage-debootstrap.sh installed first. For details, including more on the docker username, check out how to build base boxes for Docker.


The Dockerfiles

Now we have out first layer. How to build further layers on top of each other? Simple as well, just script it in the Dockerfile. Here is my Ubuntu Saucy Dockerfile:

# DevEnv:saucy
# Dockerfile for build environment for ubuntu saucy

FROM sfxpt/ubuntu:saucy

# Set the env variables to non-interactive
ENV DEBIAN_FRONTEND noninteractive
ENV DEBIAN_PRIORITY critical
ENV DEBCONF_NOWARNINGS yes

# Upstart and DBus have issues inside docker. Work around it
#RUN dpkg-divert --local --rename --add /sbin/initctl && ln -s /bin/true /sbin/initctl

RUN apt-get update

# Installing the build environment
RUN apt-get install -y build-essential devscripts fakeroot quilt dh-make automake libdistro-info-perl less nano

Notes,

  • This environment is based on my sfxpt/ubuntu:saucy base system.
  • I like to base it on my own base system instead of pulling someone else’s from the internet because
    • this way I can guarantee that my base system is up to date by the time I need it. Otherwise I might need to override many things on top of that if the base is outdated.
    • it’s extremely simple to build a base system of my own, as we’ve seen.
  • The DBus issue mentioned everywhere else has already been taken care of by the mkimage-debootstrap.sh script, so we don’t need to do it again here.
  • This will be my packages developing environment, and so far the package list seems to be sufficient.
  • I chose not to install pbuilder within the docker image, because I can always start from a pristine stage.

I put the Dockerfile file under /export/build/docker/DevEnv-saucy. For my Debian Sid developing environment, it’s even more simpler:

cd /export/build/docker/DevEnv-sid
sed 's/ubuntu/debian/g; s/saucy/sid/g' ../DevEnv-saucy/Dockerfile > Dockerfile

Building more docker layers

Now that we have the Dockerfile files, how to turn them into docker images? Simple as well:

sudo docker build -rm -t sfxpt/DevEnv:sid /export/build/docker/DevEnv-sid

This is actually what I will do:

# docker image tag
dit=saucy
sudo docker build -rm -t sfxpt/DevEnv:$dit /export/build/docker/DevEnv-$dit
dit=sid
sudo docker build -rm -t sfxpt/DevEnv:$dit /export/build/docker/DevEnv-$dit

Notes,

  • The -rm switch will have the intermediate containers removed, to save space, speed things up, and to be more robust.
  • The -t will tag the docker image, which you can refer to later, just like the sfxpt/ubuntu:saucy that we are referring to in the Dockerfile file.

More docker layer examples

Here is another example, to show how to further build docker layers on top of each other.

The Ubuntu launchpad uses the Bzr as their version control system. I don’t use Bzr much, but I do need to use it from time to time, so instead of building Bzr in to my Ubuntu develop environment, I build a Bzr layer on top of my Ubuntu develop environment:

FROM sfxpt/DevEnv:saucy

RUN apt-get update

RUN apt-get install -y bzr-builder

and to build it,

sudo docker build -rm -t sfxpt/Bzr /export/build/docker/Bzr

Docker layer showcase

Here are the docker images that we covered so far:

$ sudo docker images
REPOSITORY    TAG     ID             CREATED         SIZE
sfxpt/Bzr     latest  376e8a4b7deb   11 minutes ago  16.88 kB (virtual 591.3 MB)
sfxpt/DevEnv  sid     33d6320a8b04   13 minutes ago  382.4 MB (virtual 574.2 MB)
sfxpt/DevEnv  saucy   89989d257023   26 minutes ago  287 MB (virtual 485.4 MB)
sfxpt/ubuntu  13.10   45248c37ac6d   3 hours ago     98.64 MB (virtual 98.64 MB)
sfxpt/ubuntu  saucy   45248c37ac6d   3 hours ago     98.64 MB (virtual 98.64 MB)
sfxpt/debian  sid     b02adec6141f   4 hours ago     125.3 MB (virtual 125.3 MB)

We can see that being able to build docker layers one on top of another is a big saver for both space and time. For example, for only less that 17kB, we can have a Bzr version control system with the fully functioning Ubuntu developing environment. It will be over 590MB in size if not build this way.


More on sharing

This is that part that I’m most fond of. As we can see, the docker images/layers are extremely easy to build, but when it comes to packages developing, how to put our source code into each system and how to maintain their synchronization could be a big challenge/headache.

As it turns out there is a very simple solution — all our building systems can share the same code base, because docker allows us to map part of the host file system into the docker sub system. Here is how:

sudo docker run -v /export:/export -i -t sfxpt/DevEnv:sid /bin/bash

It allows me to get into my Debian Sid developing environment with the /export folder from my host mapped as the /export directory within my Sid developing system. For more on this, check out Docker Advanced Volumes, and

Can I specify host directory to mount from Dockerfile
http://stackoverflow.com/questions/18873474/can-i-specify-host-directory-to-mount-from-dockerfile

How to deal with persistent storage (e.g. databases) in docker
http://stackoverflow.com/questions/18496940/how-to-deal-with-persistent-storage-e-g-databases-in-docker

So, as long as I take good care of my /export developing directory, the docker images/layers are just throwaways that I can tear-down/rebuild at any time. And as we have seen, these pristine environments can be built very easily.


What’s inside the host system

As explained in previous section, the codebase is in the host system. What else are better in the host system?

  • I don’t have pbuilder within the host system either because I am sure that I will always be building from the pristine environment.
  • I keep gpg and dput in my host system so there will be less things to install in the sub system, and I only need to cache my gpg password to gpg-agent only once.
  • I use emacs to edit my source. I only install emacs in my host system.
  • I only install git in my host system as well.
  • And of course, other reliable backup/restore/version-control tools as well

Takeaway

So the takeaway is that the docker is very flexible, super easy and simple, and extremely cool for Debian/Ubuntu package developing.

Advertisements

4 thoughts on “Debian/Ubuntu Package Developing with Docker, Continued

  1. Thanks for your personal marvelous posting! I quite enjoyed reading it, you will be a great author.I will be sure to bookmark your blog and will come back someday. I want to encourage you to definitely continue your great work, have a nice holiday weekend!

  2. Warning: ‘-rm’ is deprecated, it will be replaced by ‘–rm’ soon. See usage.
    2014/07/06 15:03:28 Invalid repository name (DevEnv), only [a-z0-9-_.] are allowed

    The new docker can’t have repository names with capital letters, so s/DevEnv/devenv/g, and use ‘–rm’ instead of ‘-rm’.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s