Progress on shipit bug 826753

Last Monday I scoped the steps for Bug 826753, and spent the week prototyping everything to a functional state. I have now created a number of bugs to help with the steps needed to ship this functionality. Here are the steps as I have scoped them:

  1. Bug 1032975 - Add new table(s) to shipit database
  2. Bug 1032978 - Add a standalone process that listens to pulse for release related buildbot messages
  3. Bug 1032985 - Add REST API entry point to shipit that allows shipit-agent to enter release data into shipit database
  4. Bug 826753 - release automation should update ship it at certain points. Add the following endpoints to the app

    • /status – Lists all available releases, that have status data, in JSON format and can be queried with parameters (ie /status?var1=1&var2=10). Analogous to /releases
    • /statuses.html – Pretty GUI view of the info shown in /status. Analogous to /releases.html
    • /status/<release-name>

      • GET: Lists release status info in JSON. Analogous to /releases/<release-name>
      • POST: covered in Bug 1032985
    • /status/<release-name>.html – Pretty GUI view of the info shown in /status/ GET, possibly including a visual timeline of the 6 steps: Tagging completed, All builds/repacks completed, Updates on betatest, Ready for releasetest, Ready for release and Postrelease. (status steps taken from Bug 826753 comment 0)

 

Currently I have a set of outstanding questions, which I will summerize here by the bug they are posted in.

  • Bug 1032975

    • Should I add 2 tables to the shipit database, or just 1? (release_data and release_status, or just release_status)
    • How is the schema for release_status? Does it need to be more explicit and less abstract? Does it need more fields?
  • Bug 1032978

    • What should I use other than .netrc to authenticate when sending a POST request to the shipit entry point at /status?
  • Bug 1032985

    • I'm using the routing_key field of pulse.m.o messages to find buildbot release related messages. Specifically I am sorting for *release*, but there are multiple prefixes that show up including build.release-*, unittest.release-*, talos.release-*, etc, etc. Should I be keeping or ignoring these messages (ie are they useful to me in determining the 6 status steps mentioned in Bug 826753 comment 0)
    •  I am currently using the pulse.m.o message fields product + version + build_number to assemble the name to store release data with (ie Firefox-31.b05-build1). This is necessary because releases are stored in the shipit database with names of this format as their primary key. The problem is that many pulse.m.o messages have None in product and version. Additionally, some seem to deviate from the product (ie I have seen "mobile", "firefox", and "xulrunner", when I thought I should just be seeing "firefox"… I could be mistaken about what I should expect). Should I ignore the messages that hold None in product/version? Is there some other field/fields that I should be filtering based on?
    • Specifically what sort of messages pertain to each of the 6 steps: Tagging completed, All builds/repacks completed, Updates on betatest, Ready for releasetest, Ready for release and Postrelease? (status steps taken from Bug 826753 comment 0)

Useful things at this stage:

  • 640 collected pulse.m.o messages relating to Firefox-31.b05-build1 release, stored in a database with each message inhabiting a single row

    • Should contain most, if not all, buidlbot release messages from the Firefox-31.b05-build1 release, but it's possible that I missed some
    • To load and use simply run "cat releasedata.sql | sqlite releasedata.sqlite" and use the SQLite Manager Firefox Add-On to sort and use the database to gain a context for the sorts of messages I am sorting through.
  • I also have 22,000+ pulse.m.o messages from the period of time that the Firefox-31.b05-build1 release was running, that I simply scrapped for messages where routing_key="%release%"

If you have anything at all to add, please reply to the questions I have posted in the bugs that they pertain to :)

More to come on this!

Building BuildAPI Docker app

I am working through building up the BuildAPI Docker app that is going to live and work in Vagrant with the other apps. Currently I have things to the stage where buildapi is fully installed, and running but am having problems with kombu.

I have run into these issues before, when I was configuring kombu for the first time on my local machine. When I run docker build -t buildapi-app . and then docker run -i -p 5000:5000 -t buildapi-app the app starts up fine, but I see the following message about kombu:

localhost:buildapi-app jzeller$ docker run -i -p 5000:5000 -t buildapi-app
Starting subprocess with file monitor
Running reloading file monitor
Starting server in PID 7.
2014-04-16 05:51:00,389 WARNI [buildapi.lib.mq] [Dummy-2] Kombu connection failed ([Errno 111] Connection refused); waiting 2s
serving on 0.0.0.0:5000 view at http://127.0.0.1:5000
2014-04-16 05:51:02,396 WARNI [buildapi.lib.mq] [Dummy-2] Kombu connection failed ([Errno 111] Connection refused); waiting 4s
2014-04-16 05:51:06,407 WARNI [buildapi.lib.mq] [Dummy-2] Kombu connection failed ([Errno 111] Connection refused); waiting 6s
2014-04-16 05:51:12,419 WARNI [buildapi.lib.mq] [Dummy-2] Kombu connection failed ([Errno 111] Connection refused); waiting 8s

Here is the current Dockerfile for buildapi-app:

FROM ubuntu:precise
MAINTAINER Chirs Atlee <catlee@mozilla.com>
MAINTAINER John Zeller <johnlzeller@gmail.com>

# Install base dependencies
RUN apt-get -q update
RUN apt-get -y -q install python-dev mercurial sudo python-pip subversion wget curl make python-virtualenv libmysqlclient-dev sqlite3

# Install Python Google Visualizations 1.7.1
RUN pip install -e svn+http://google-visualization-python.googlecode.com/svn/trunk@18#egg=gviz_api.py-1.7.1-py2.7-dev_r18

# Install BuildAPI
RUN hg clone http://hg.mozilla.org/build/buildapi

# Install pip requirements with pip install
RUN curl http://python-distribute.org/distribute_setup.py | python; rm *.tar.gz;
RUN pip install -r buildapi/rtfd-requirements.txt
RUN pip install kombu==3.0.12 MySQL-python==1.2.3 python-memcached pytz==2011h

# Setup app
RUN cd buildapi; python setup.py develop; paster make-config buildapi config.ini
RUN cd buildapi; sed 's/buildapi.cache = redis:HOSTNAME:PORT/buildapi.cache = memcached:127.0.0.1:5001/' config.ini > tmp; rm config.ini; mv tmp config.ini

# Setup sqlite dbs
#RUN cd buildapi; sqlite3 statusdb.sqlite3 < statusdb_schema.sql
#RUN cd buildapi; sqlite3 schedulerdb.sqlite3 < schedulerdb_schema.sql
#RUN cd buildapi; sqlite3 buildapi.sqlite3 < buildapi_schema.sql

# Setup Kombu
RUN cd buildapi; sed 's/mq.kombu_url =/mq.kombu_url = localhost/' config.ini > tmp; rm config.ini; mv tmp config.ini

EXPOSE 5000

CMD ["paster", "serve", "--reload", "-v", "buildapi/config.ini"]