Last Monday I scoped the steps for Bug 826753, and spent the week prototyping everything to a functional state. I have now created a number of bugs to help with the steps needed to ship this functionality. Here are the steps as I have scoped them:
- Bug 1032975 - Add new table(s) to shipit database
- Bug 1032978 - Add a standalone process that listens to pulse for release related buildbot messages
- Bug 1032985 - Add REST API entry point to shipit that allows shipit-agent to enter release data into shipit database
Bug 826753 - release automation should update ship it at certain points. Add the following endpoints to the app
/status – Lists all available releases, that have status data, in JSON format and can be queried with parameters (ie /status?var1=1&var2=10). Analogous to /releases
- /statuses.html – Pretty GUI view of the info shown in /status. Analogous to /releases.html
GET: Lists release status info in JSON. Analogous to /releases/<release-name>
POST: covered in Bug 1032985
) /status/<release-name> .html – Pretty GUI view of the info shown in /status/ GET, possibly including a visual timeline of the 6 steps: Tagging completed, All builds/repacks completed, Updates on betatest, Ready for releasetest, Ready for release and Postrelease. (status steps taken from Bug 826753 comment 0
- /status – Lists all available releases, that have status data, in JSON format and can be queried with parameters (ie /status?var1=1&var2=10). Analogous to /releases
Currently I have a set of outstanding questions, which I will summerize here by the bug they are posted in.
- Should I add 2 tables to the shipit database, or just 1? (release_data and release_status, or just release_status)
- How is the schema for release_status? Does it need to be more explicit and less abstract? Does it need more fields?
- What should I use other than .netrc to authenticate when sending a POST request to the shipit entry point at /status?
- I'm using the routing_key field of pulse.m.o messages to find buildbot release related messages. Specifically I am sorting for *release*, but there are multiple prefixes that show up including build.release-*, unittest.release-*, talos.release-*, etc, etc. Should I be keeping or ignoring these messages (ie are they useful to me in determining the 6 status steps mentioned in Bug 826753 comment 0)
- I am currently using the pulse.m.o message fields product + version + build_number to assemble the name to store release data with (ie Firefox-31.b05-build1). This is necessary because releases are stored in the shipit database with names of this format as their primary key. The problem is that many pulse.m.o messages have None in product and version. Additionally, some seem to deviate from the product (ie I have seen "mobile", "firefox", and "xulrunner", when I thought I should just be seeing "firefox"… I could be mistaken about what I should expect). Should I ignore the messages that hold None in product/version? Is there some other field/fields that I should be filtering based on?
- Specifically what sort of messages pertain to each of the 6 steps: Tagging completed, All builds/repacks completed, Updates on betatest, Ready for releasetest, Ready for release and Postrelease? (status steps taken from Bug 826753 comment 0)
Useful things at this stage:
640 collected pulse.m.o messages relating to Firefox-31.b05-build1 release, stored in a database with each message inhabiting a single row
- Should contain most, if not all, buidlbot release messages from the Firefox-31.b05-build1 release, but it's possible that I missed some
- To load and use simply run "cat releasedata.sql | sqlite releasedata.sqlite" and use the SQLite Manager Firefox Add-On to sort and use the database to gain a context for the sorts of messages I am sorting through.
- I also have 22,000+ pulse.m.o messages from the period of time that the Firefox-31.b05-build1 release was running, that I simply scrapped for messages where routing_key="%release%"
If you have anything at all to add, please reply to the questions I have posted in the bugs that they pertain to
More to come on this!