Tupperware: Mozilla apps in Docker!

Announcing Tupperware, a setup for Mozilla apps in Docker! Tupperware is portable, reusable, and containerized. But unlike typical tupperware, please do not put it in the Microwave.

Tupperware

Why?

This is a project born out of a need to lower the barriers to entry for new contributors to Release Engineering (RelEng) maintained apps and services. Historically, RelEng has had greater difficulty attracting community contributors than other parts of Mozilla, due in large part to how much knowledge is needed to get going in the first place. For a new contributor, it can be quite overwhelming to jump into any number of the code bases that RelEng maintains and often leads to quickly losing that new contributor out of exaspiration. Beyond new contributors, Tupperware is great for experienced contributors as well to assist in keeping an unpolluted development environment and testing patches.

What?

Currently Tupperware includes the following Mozilla apps:

BuildAPI – a Pylons project used by RelEng to surface information collected from two databases updated through our buildbot masters as they run jobs.

BuildBot – a job (read: builds and tests) scheduling system to queue/execute jobs when the required resources are available, and reporting the results.

Dependency apps currently included:

RabbitMQ – a messaging queue used by RelEng apps and services

MySQL – Forked from orchardup/mysql

How?

Vagrant is used as a quick and easy way to provision the docker apps and make the setup truly plug n’ play. The current setup only has a single Vagrantfile which launches BuildAPI and BuildBot, with their dependency apps RabbitMQ and MySQL.

How to run:

– Install Vagrant 1.6.3

– hg clone https://hg.mozilla.org/build/tupperware/ && cd tupperware && vagrant up (takes >10 minutes the first time)

Where to see apps:

– BuildAPI: http://127.0.0.1:8888/

– BuildBot: http://127.0.0.1:8000/

– RabbitMQ Management: http://127.0.0.1:15672/

Troubleshooting tips are available in the Tupperware README.

What’s Next?

Now that Tupperware is out there, it’s open to contributors! The setup does not need to stay solely usable for RelEng apps and services. So please submit bugs to add new ones! There are a few ideas for adding functionality to Tupperware already:

  • Bug 1027410 - Add Treeherder docker container to Tupperware
  • Bug 1027412 - Add multiple vagrant setups to Tupperware to customize setup
  • Bug 1027417 - Have MySQL docker app in Tupperware load database schemas

Have ideas? Submit a bug!

 

Deployed BuildAPI bug fix, L2 Access, Tupperware

A bunch of new stuff!

New Things

New Bugs

What's next?

  • Test Arbitrary Builds for bug Bug 1009565 – Triggering arbitrary jobs gets branch wrong
  • Make initial commit to hg.m.o/build/tupperware
  • Troubleshoot buildbot web interface on localhost:8501 
  • Make multiple Vagrantfiles to choose from based on required setup needs
  • Publish docker images to new Docker Index for Mozilla repo 
  • Create a wiki doc for Tupperware  
  • Create mysql-app that can load database schemas

That is all for now!

 

BuildAPI, Buildbot, RabbitMQ and MySQL containers are all up! Some testing left…

 BuildAPI, Buildbot, RabbitMQ and MySQL containers are all up now! To run pull http://hg.mozilla.org/users/jozeller_mozilla.com/vagrant-docker-setup and run 'vagrant up' from the vagrant-docker-setup/ directory.

The vagrant up command will take several minutes to run the first time because it needs to pull the docker images from the Docker Index at docker.io. More to come tomorrow on this. NOTE: Buildbot seems to be running, but I have not been able to test *full* functionality just yet. However, the buildapi-app, rabbitmq-app and orchardup/mysql containers run together just fine.

To view

  • BuildAPI: localhost:8888
  • RabbitMQ: localhost:15672
  • Buildbot: localhost:8501 – NOT YET

Keep checking back!

New

  • Added specific app users to mysql with passwords
  • Added version row with value 6 to schedulerdb
  • Showed that an added job from buildapi will show up in mysql on buildbot
  • The malformed url error was caused by the fact that the URL was not importing the environment variable
  • Once the env var was imported, I was still getting the malformed url, but this time it was because I had not created a password for the user. I remember when I was setting up my local buildbot instance that I ran into this same problem. There is a regex that is checking to see that the url is not malformed and it does not take kindly to the absense of passwords, regardless of the fact that mysql is okay with not having a password for a user at all.
  • Uploaded images for johnlzeller/rabbitmq, johnlzeller/buildapi and johnlzeller/buildbot to Docker Index
  • Verified that entire setup can be run in Vagrant

What's next?

  • Create repo on hg.mozilla.org/build for holidng Vagrantfile and Dockerfiles for images and update the new hg.m.o/build repo with Vagrantfile and Dockerfiles for images
  • Troubleshoot why the buildbot web interface is not showing up on localhost:8501
  • Publish setup to blog

After initial release

  • Have 1 of 2 things should happen:

     

    1. Have mysql-app setup to load its own schemas and users
    2. Have individual apps only load schemas and users if they do not already exist… this ensures persistence of the databases
  • Look into using the VOLUME docker command to setup an easy way to share a host directory for editing purposes. The goal here is to make it easy to make changes to the running dev setup and to test that setup. Currently, the docker setup just runs the tip of each repository for buildapi and buildbot

Questions

  • Why/how does schedulerdb.version get propogated with a version number int like 6. Buildbot-app was failing on the fact that there was no row in version. I just added 6 into it, since that is what my local schedulerdb dump had, but is there a more appropriate way to do this? Does this check need to be changed? The assert can be found on line 35 of /usr/local/lib/python2.7/dist-packages/buildbot-0.8.2_hg_f6d9311d9246_production_0.8-py2.7.egg/buildbot/db/schema/manager.py
 

Vagrant can now run BuildAPI and RabbitMQ apps

Continuing on from my previous post, I verified that buildapi and selfserve-agent are truly connected and able to exchange over the amqp, and that the entire buildapi application is running well by running similar procedures that work in my local setup.

Once I did that I updated the Vagrantfile to forward the vagrant port 8888 to the host port 8888, and to build and start the rabbitmq-app and buildapi-app. In the wild, the Vagrantfile will not be having docker build the docker images, but rather it will pull them from Mozilla's docker repository, which will be a much faster process. As it stands, running vagrant up from scratch the first time will take about 10-15 minutes to launch.

Here's how you can NOW run a fully functional BuildAPI app locally with a single command :)

  1. hg clone http://hg.mozilla.org/users/jozeller_mozilla.com/vagrant-docker-setup
  2. cd vagrant-docker-setup
  3. vagrant up
 

Building BuildAPI Docker app

I am working through building up the BuildAPI Docker app that is going to live and work in Vagrant with the other apps. Currently I have things to the stage where buildapi is fully installed, and running but am having problems with kombu.

I have run into these issues before, when I was configuring kombu for the first time on my local machine. When I run docker build -t buildapi-app . and then docker run -i -p 5000:5000 -t buildapi-app the app starts up fine, but I see the following message about kombu:

localhost:buildapi-app jzeller$ docker run -i -p 5000:5000 -t buildapi-app
Starting subprocess with file monitor
Running reloading file monitor
Starting server in PID 7.
2014-04-16 05:51:00,389 WARNI [buildapi.lib.mq] [Dummy-2] Kombu connection failed ([Errno 111] Connection refused); waiting 2s
serving on 0.0.0.0:5000 view at http://127.0.0.1:5000
2014-04-16 05:51:02,396 WARNI [buildapi.lib.mq] [Dummy-2] Kombu connection failed ([Errno 111] Connection refused); waiting 4s
2014-04-16 05:51:06,407 WARNI [buildapi.lib.mq] [Dummy-2] Kombu connection failed ([Errno 111] Connection refused); waiting 6s
2014-04-16 05:51:12,419 WARNI [buildapi.lib.mq] [Dummy-2] Kombu connection failed ([Errno 111] Connection refused); waiting 8s

Here is the current Dockerfile for buildapi-app:

FROM ubuntu:precise
MAINTAINER Chirs Atlee <catlee@mozilla.com>
MAINTAINER John Zeller <johnlzeller@gmail.com>

# Install base dependencies
RUN apt-get -q update
RUN apt-get -y -q install python-dev mercurial sudo python-pip subversion wget curl make python-virtualenv libmysqlclient-dev sqlite3

# Install Python Google Visualizations 1.7.1
RUN pip install -e svn+http://google-visualization-python.googlecode.com/svn/trunk@18#egg=gviz_api.py-1.7.1-py2.7-dev_r18

# Install BuildAPI
RUN hg clone http://hg.mozilla.org/build/buildapi

# Install pip requirements with pip install
RUN curl http://python-distribute.org/distribute_setup.py | python; rm *.tar.gz;
RUN pip install -r buildapi/rtfd-requirements.txt
RUN pip install kombu==3.0.12 MySQL-python==1.2.3 python-memcached pytz==2011h

# Setup app
RUN cd buildapi; python setup.py develop; paster make-config buildapi config.ini
RUN cd buildapi; sed 's/buildapi.cache = redis:HOSTNAME:PORT/buildapi.cache = memcached:127.0.0.1:5001/' config.ini > tmp; rm config.ini; mv tmp config.ini

# Setup sqlite dbs
#RUN cd buildapi; sqlite3 statusdb.sqlite3 < statusdb_schema.sql
#RUN cd buildapi; sqlite3 schedulerdb.sqlite3 < schedulerdb_schema.sql
#RUN cd buildapi; sqlite3 buildapi.sqlite3 < buildapi_schema.sql

# Setup Kombu
RUN cd buildapi; sed 's/mq.kombu_url =/mq.kombu_url = localhost/' config.ini > tmp; rm config.ini; mv tmp config.ini

EXPOSE 5000

CMD [“paster”, “serve”, “–reload”, “-v”, “buildapi/config.ini”]

 

Further Steps with Vagrant

Today I spent a bunch of time looking into how to setup a vagrant instance, how to pack it with docker, how to run multiple docker containers in that vagrant instance and how to link them all together to play nicely. Throughout this process I have narrowed down that vagrant will be running hashicorp/precise64, and docker will have 5 apps, one each for buildapi, mysql , redis, rabbitmq and buildbot. Once I have linked all of this together properly, there will be a directory structure that looks a little like this:

./Vagrantfile

./redis-app/Dockerfile

./rabbitmq-app/Dockerfile

./mysql-app/Dockerfile

./buildbot-app/Dockerfile

./buildapi-app/Dockerfile

The Vagrantfile will load a base image of Ubuntu 12.04 Precise 64-bit, install docker (tip), build the images for buildapi (tip), mysql (tip), redis (v2.4.5), rabbitmq (tip) and buildbot (tip), expose all the necessary ports for each and link them properly, and then run each container as a daemon. Once this is all setup the user should be able to simply pull this repo (I am assuming this directory will become a repository on hg.mozilla.org/build), run "vagrant up" and then they will be able to see buildapi at 127.0.0.1:5000 and buildbot at 127.0.0.1:8501. The awesome added benefit to this is that files can be shared between your normal development environment and the VM that vagrant started up, simply by placing and editing files in the base directory where Vagrantfile exists! This will make development of these apps a lot smoother for our community members, with the super simple push button command line interface that vagrant provides for getting these VMs started up. Additionally, once this works it can easily be expanded with more and more apps as we see fit to add them to this Vagrant instance! :)

 

Vagrant + Docker = Happy Fun Times!

I am working with Vagrant and Docker for the first time, and it is awesome! We are working towards capturing all of our "how to install" knowledge for buildbot, buildapi and redis in code by making a set of docker containers that can play well together inside of Vagrant. In this setup, we are assuming that the user has Ubuntu (or equivalent linux) in a virtualbox running. Once that 1 VM exists we are to have separate docker instances per app (buildbot, buildapi, redis, etc.). The benefit of connecting these docker instances inside vagrant to play together, is that it lets any combo of buildbot, buildapi standalone, or "system" to be run. This also allows more modules to be added later. 

So far I have installed Vagrant, Docker and Boot2Docker, worked my way through the Boot2Docker/Docker tutorials (including a sweet hello world app example from David Dias), and I have begun working my way through the Vagrant tutorials. Other than all that prep, I have also begun writing up the Dockerfiles needed to setup each of our apps. I think we will need one each for:

I got most of the way through the BuildAPI and RabbitMQ Dockerfile's before I got to a point where I needed to get Vagrant installed and figured out so that I could also then create a Vagrantfile to go along with this entire setup.

Once I have completed this entire setup, you will be able to get a buildapi, buildbot, redis, rabbitmq, and selfserve system up and running simply by downloading the Vagrantfile and running 'vagrant up'. How cool is that?!

More updates on this next week!

Useful links: