Scoping Bug 826753 – release automation should update ship it at certain points

I spent Tuesday scoping Bug 826753 - release automation should update ship it at certain points, and I have come up with a set of clear deliverables. This list could still be modified going forward, but here is the rough layout.

I learned:

  • Buildbot publishes updates, in addition to emails to release@, for the progress of releases to pulse.m.o
  • Message format for Buildbot updates is found on pulse.m.o
  • I must look for build.release-*.finished in the [‘payload’][‘buildername’] part of the pulse.m.o messages

Am doing:

  • Setup a logger on dev-master1 to scrap all the messages from pulse.m.o overnight to catch all messages pertaining to the just launched 31.0b4 release, which will give me a live current sample to test with

Deliverables (no particular order):


    • /status (AND /status?var1=1&var2=10)

      • Lists all available releases with status in JSON format and can be queried with parameters
    • /status.html

      • pretty gui view of the info shown in /status
    • /status/<release-name>

      • GET: dumps release status info in JSON
      • POST: updates a new table in update.db with new status info about <release-name>
    • /status/<release-name>.html

      • pretty gui view of the info shown in /status/<release-name> GET and will likely include a nice timeline with the A, B, C steps referred to in comment 3 of bug 826753
  • Long running standalone script that listens to pulse.m.o for updates about releases and then uses the /status/<release-name> REST API entry point to update a new table in update.db with new status info about <release-name>
  • New status table in ubdate.db

More tomorrow (Tuesday)!


Tupperware: Mozilla apps in Docker!

Announcing Tupperware, a setup for Mozilla apps in Docker! Tupperware is portable, reusable, and containerized. But unlike typical tupperware, please do not put it in the Microwave.



This is a project born out of a need to lower the barriers to entry for new contributors to Release Engineering (RelEng) maintained apps and services. Historically, RelEng has had greater difficulty attracting community contributors than other parts of Mozilla, due in large part to how much knowledge is needed to get going in the first place. For a new contributor, it can be quite overwhelming to jump into any number of the code bases that RelEng maintains and often leads to quickly losing that new contributor out of exaspiration. Beyond new contributors, Tupperware is great for experienced contributors as well to assist in keeping an unpolluted development environment and testing patches.


Currently Tupperware includes the following Mozilla apps:

BuildAPI – a Pylons project used by RelEng to surface information collected from two databases updated through our buildbot masters as they run jobs.

BuildBot – a job (read: builds and tests) scheduling system to queue/execute jobs when the required resources are available, and reporting the results.

Dependency apps currently included:

RabbitMQ – a messaging queue used by RelEng apps and services

MySQL – Forked from orchardup/mysql


Vagrant is used as a quick and easy way to provision the docker apps and make the setup truly plug n’ play. The current setup only has a single Vagrantfile which launches BuildAPI and BuildBot, with their dependency apps RabbitMQ and MySQL.

How to run:

– Install Vagrant 1.6.3

– hg clone && cd tupperware && vagrant up (takes >10 minutes the first time)

Where to see apps:

– BuildAPI:

– BuildBot:

– RabbitMQ Management:

Troubleshooting tips are available in the Tupperware README.

What’s Next?

Now that Tupperware is out there, it’s open to contributors! The setup does not need to stay solely usable for RelEng apps and services. So please submit bugs to add new ones! There are a few ideas for adding functionality to Tupperware already:

  • Bug 1027410 - Add Treeherder docker container to Tupperware
  • Bug 1027412 - Add multiple vagrant setups to Tupperware to customize setup
  • Bug 1027417 - Have MySQL docker app in Tupperware load database schemas

Have ideas? Submit a bug!


Manual Testing of Arbitrary Builds

When a new selfserve-agent change is pushed to production, it's necessary to verify functionality with some maual testing. Here are some steps to basic testing:

  1. If no new try job to mess with, then submit one, see ReleaseEngineering/TryServer


    • hg clone
    • cd mozilla-central
    • echo "THING" >> README.txt
    • hg qnew test-patch
    • hg qref –message "try: -b o -p linux64 -u none -t none"
    • hg push -f ssh://
  2. In my case you can see the try job running here:


    • If the push is successful it'll give you your own link
  3. Submit a blank arbitrary job request to x86-64 try build/3a5e6ca198d8 using
  4. python –buildername "Linux x86-64 try build" –branch try –rev 3a5e6ca198d8


    • Leaving –file out so that files = []
  5. See running job here
  6. Check for pending job at
  7. Also check
  8. Check buildbot status can be found by finding the appropriate master on the buildapi page

Deployed BuildAPI bug fix, L2 Access, Tupperware

A bunch of new stuff!

New Things

New Bugs

What's next?

  • Test Arbitrary Builds for bug Bug 1009565 – Triggering arbitrary jobs gets branch wrong
  • Make initial commit to hg.m.o/build/tupperware
  • Troubleshoot buildbot web interface on localhost:8501 
  • Make multiple Vagrantfiles to choose from based on required setup needs
  • Publish docker images to new Docker Index for Mozilla repo 
  • Create a wiki doc for Tupperware  
  • Create mysql-app that can load database schemas

That is all for now!