Buildbot is up and running!

Buildbot is now up and running on my local Mac! :)

Picking up from where I left off in my previous post, this is what was necessary to make this happen:

  1. Firstly, attempting to run 'make checkconfig' from ~/Buildbot/buildbot/ was completely misguided. I was under the impression that the master/ directory that I was looking for was ~/Buildbot/buildbot/master/. However, this was incorrect as the true master directory I needed was ~/Buildbot/build-master/.
  2. cd BASEDIR/build-master/
  3. Modify line 2 of Makefile from BUILDBOT=$(PWD)/bin/buildbot to be BUILDBOT=$(PWD)/../bin/buildbot
  4. make check-config

    • This fails with the Traceback:
    • (Buildbot)localhost:build-master jzeller$ make checkconfig
      cd master && /Users/jzeller/Buildbot/build-master/../bin/buildbot  checkconfig
      Traceback (most recent call last):
        File "/Users/jzeller/Buildbot/lib/python2.6/site-packages/buildbot-0.8.2-py2.6.egg/buildbot/scripts/", line 1042, in doCheckConfig
        File "/Users/jzeller/Buildbot/lib/python2.6/site-packages/buildbot-0.8.2-py2.6.egg/buildbot/scripts/", line 31, in __init__
          self.loadConfig(configFile, check_synchronously_only=True)
        File "/Users/jzeller/Buildbot/lib/python2.6/site-packages/buildbot-0.8.2-py2.6.egg/buildbot/", line 652, in loadConfig
          exec f in localDict
        File "/Users/jzeller/Buildbot/build-master/master.cfg", line 129, in <module>
          execfile(releaseConfigFile, releaseBranchConfig, releaseBranchConfig)
      IOError: [Errno 2] No such file or directory: ''
      make: *** [checkconfig] Error 1

  5. Modify line 11 of master_config.json so that 'release_branches' contains an empty list. It did contain [“mozilla-1.9.2″,  “mozilla-beta”] and they are unnecessary for my local setup.

  6. Again: make checkconfig

    • SUCCESS: Config file is good!

  7. make start

    • It will appear as if start has failed, but fear not! You'll see the message below, but this is simply because the buildbot process is launched as a dameon and buildmaster waits to give the all configuration is complete signal. However, Mozilla's configuration files are way to complicated to be dealt with in under 10 seconds, so it times out and you see the message below:

    • (Buildbot)localhost:build-master jzeller$ make start
      cd master && /Users/jzeller/Buildbot/build-master/../bin/buildbot  start $PWD
      Following twistd.log until startup finished..
      2014-01-22 14:47:30-0800 [-] Log opened.
      2014-01-22 14:47:30-0800 [-] twistd 12.0.0 (/Users/jzeller/Buildbot/bin/python 2.6.7) starting up.
      2014-01-22 14:47:30-0800 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
      2014-01-22 14:47:30-0800 [-] monkeypatch_twisted_cbLogin applied
      2014-01-22 14:47:30-0800 [-] Creating BuildMaster — buildbot.version: 0.8.2
      2014-01-22 14:47:30-0800 [-] loading configuration from /Users/jzeller/Buildbot/build-master/master.cfg
      2014-01-22 14:47:30-0800 [-] unable to import dnotify, so Maildir will use polling instead
      2014-01-22 14:47:30-0800 [-] nextAWSSlave: start
      2014-01-22 14:47:30-0800 [-] nextAWSSlave: start
      2014-01-22 14:47:37-0800 [-] nextAWSSlave: start
      2014-01-22 14:47:37-0800 [-] nextAWSSlave: start

      The buildmaster took more than 10 seconds to start, so we were unable to
      confirm that it started correctly. Please 'tail twistd.log' and look for a
      line that says 'configuration update complete' to verify correct startup.

      make: *** [start] Error 1

  8. To doublecheck that the configuration was successful, just follow the directions and type: less twistd.log | grep "configuration update complete". As long as you see it pop up then you're good!

  9. Go to http://localhost:8501/

  10. If you a page with "Welcome to the Buildbot for the Firefox project!" then you have been successful!

  11. You can now check out what buildbot has pending by going to http://localhost:8501/waterfall

Now buildbot is all setup! Hooray!

The next steps are:

  1. Update all the documentation to reflect this new found path!
  2. Update and upload the scripts that can make this process much easier, a buildbot-on-laptop if you will
  3. Run a database dump on my personal schedulerdb and statusdb that represents a smaller subset of builds/tests… Ideally something small enough to send around to others.

    • Send this to Armen for his BuildAPI setup
  4. Replace my current schedulerdb and statusdb with these smaller subsets to claim back more harddrive space – currently 100gb+!
  5. Backup new schedulerdb and statusdb
  6. Link up buildbot to the proper database
  7. Test that running an existing command from BuildAPI has the expected results in buildbot by looking at the http://localhost:8501/waterfall UI
  8. Finish writing up all of the manual unit tests that need to be used for testing Bug 793989's patch
  9. Modify BuildAPI to add the new functionality for a Bug 793989 patch
  10. Test/Review the patch!

Here's to shipping this patch!


Setting Up a Development Master

Setting up buildbot on my local Mac has proved to be more difficult than I first realized. hwine has helped me out by modifying and sharing a script that helps to setup a new buildbot project called, and combining that with the create-staging-master script that can be found on the How To Setup a Personal Development Master, I think buildbot will be all setup on in my local dev environment pretty soon! However, I have run into some issues, which I already emailed hwine about, but I will rephrase them here.

Here is how I set things up, complete with how I fixed a few issues:
  1. Downloaded and create-staging-master to ~/
  2.  ./ Buildbot
  3. cd Buildbot; source bin/activate
  4. cd buildbot-configs; ./

    • All tests passed
  5. cd ~
  6. Modify create-staging-master so that "my $basedir = '/builds/buildbot';" becomes "my $basedir = '/Users';" on line 39
  7. sudo perl create-staging-master –username=jzeller –master-kind=build –config-only

    • This creates a new directory at ~/build1 which contains config.json

      • More on this below, but I think the config.json file needs to be at ~/Buildbot/buildbot instead. However, when you run the script with the parameter –master-dir set to either Buildbot/ or Buildbot/buildbot, it complains that the directory already exists.
  8. Now that I want to see if it all works, I am trying to follow the Does It Work? section in the How To Setup a Personal Development Master wiki.
  9. cd $BASEDIR/master; ln -sf ../buildbot-configs/mozilla/universal_master_sqlite.cfg; cd –

    • Since $BASEDIR is not set by the 2 scripts you sent over, and the script in the Create a Build Master section sets it to /builds/buildbot/$USERNAME/build1, I go ahead and set it to BASEDIR=/Users/jzeller/Buildbot/buildbot
  10. Again: cd $BASEDIR/master; ln -sf ../buildbot-configs/mozilla/universal_master_sqlite.cfg; cd –

    • Successful… tried with BASEDIR=/Users/jzeller/Buildbot/ and it failed because there was no master directory in ~/Buildbot
  11. make checkconfig

    • When attempting from ~/Buildbot/buildbot, this fails with: make: *** No rule to make target `checkconfig'.  Stop.
    • After speaking with aki (check scrollback on mozbuild), it seems as though the Makefile I had in ~/Buildbot/buildbot was just dev utils
    • aki pointed me to the Makefile.master in buildbot-configs and I cp'd it to ~/Buildbot/buildbot and modified BUILDBOT=$(PWD)/bin/buildbot to be BUILDBOT=$(PWD)/../bin/buildbot
  12. Again: make checkconfig

    • Fails in a different way:

      • (Buildbot)localhost:buildbot jzeller$ make checkconfig
        cd master && /Users/jzeller/Buildbot/buildbot/../bin/buildbot checkconfig
        Traceback (most recent call last):
          File "/Users/jzeller/Buildbot/lib/python2.6/site-packages/buildbot-0.8.2-py2.6.egg/buildbot/scripts/", line 1042, in doCheckConfig
          File "/Users/jzeller/Buildbot/lib/python2.6/site-packages/buildbot-0.8.2-py2.6.egg/buildbot/scripts/", line 18, in __init__
            copy(configFileName, tempdir)
          File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/", line 84, in copy
            copyfile(src, dst)
          File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/", line 50, in copyfile
            with open(src, 'rb') as fsrc:
        IOError: [Errno 2] No such file or directory: '/Users/jzeller/Buildbot/buildbot/master/master.cfg'
        make: *** [checkconfig] Error 1
    • I realized that the master.cfg found in ~/Buildbot/buildbot may be a faulty sym link, so I modified the command from step 10 and reran it with an absolute path: cd $BASEDIR/master; ln -sf /Users/jzeller/Buildbot/buildbot-configs/mozilla/universal_master_sqlite.cfg; cd –

      • This gets the same Traceback as above

I am stuck at the make checkconfig step here, SO CLOSE. Here are a few other questions off the bat:

  • config.json surely goes somewhere within the Buildbot base directory, but I am not sure where this would be appropriate. I attempted to set the –master-dir parameter to either Buildbot/ or Buildbot/buildbot, it complains that the directory already exists.
  • aki is saying that I may have an easier time recreating the master, rather than modifying configs, however I am not sure how that relates to using these 2 scripts, and create-staging-master in tandum.

So for the moment this is where I am stuck, and I will be attempting to tackle this more on Friday.


Happy New Year!

Happy New Year everyone! Now that it is 2014, it is time to get back up to speed. Here is where I am at currently, ie as of mid-December

  • We decided to go ahead and manually test the patch needed for bug 793989, since it is needed in semi-short order. After the patch is of a fairly decent 1st iteration, then we will go back and diagnose the nosetests issue and determine how to restore our unit testing ability with pylons. This is extra important given that more and more people are becoming interested in BuildAPI and so we need the stability that comes with a solid testing framework.
  • I met with Armen last year and we are going to be working side-by-side on tackling the rest of this patch, he pointed me to his blog post on setting up a local buidbot instance so that I can use it for testing my patch.

Here is what I have in front of me now

  • Setup a running buildbot master instance on my local machine and verify that it can receive and "start" jobs from BuildAPI commands already present in the UI
  • Document the necessary manual unit tests needed for this patch in the wiki
  • Write up the additions to catlees patch necessary to get us to phase 1, which should include the ability to make an http POST request with the necessary parameters and having that launch an individual test on buildbot specified by those parameters in the POST request.

More updates to come as this list of things is knocked out.