By trying to automate tests for pull requests on Bitbucket, we've faced some interesting issues we'd like to share with you. Not least because our Jenkins instance is not publicly reachable.

The easiest solution to automate pull requests is to be notified by Bitbucket about the creation and update of pull requests. Bitbucket provides a feature called Webhooks which can be used to achieve that, but since our Jenkins instance isn't reachable from the public internet, it's not possible for us to use Webhooks. As our team grows and tests become more and more important to ensure stability of our work, we decided to evaluate some Jenkins plugins, but they either didn't work or didn't do what we expected.

Eventually, we decided to write a small Python script which was supposed to solve this problem. A script that simply pulls the information about the current pull requests and decides which one to test next. Luckily, the Bitbucket API provides all information about pull requests and it's also easy to use as it's a RESTful API. After an initial collection of the current information, they can be compared with future requests. The result is a list of pull request IDs which have been changed, added or removed. Then the script would utilize the Jenkins API to start the job and retrieve the status of the build, once the job has finished.

The Problem

As I was implementing this change, I faced a problem with the Jenkins API I was talking to. To be able to track builds, you naturally have to have the correct build number. Otherwise you won't be able to retrieve the status of the tests after they've completed. The Jenkins API provides you with a way to retrieve the build number the next build is going to get assigned to. There's no way to start a build and get the build number in the response. This is probably due to the implementation of the queue. A queued task may be aborted and the estimated build number is then freed to be used by other queued tasks. This is a problem if more than one job is going to be triggered, because the Jenkins API will always return the next build number. That is the number the next build will get assigned to if it's started, meaning the next job in the queue, but not the next job which is assigned to the queue by calling build_job of the Jenkins API, because that one might just not be the next job to be processed. Simply this has been enough to be unable to create the script as it was planned, because it will be necessary to queue more than two jobs at once and store their build numbers right after they have been queued, so that the result of those jobs can be evaluated. Surely, it might have been possible to make an educated guess about the next correct next build number, but we decided that this is too unsafe for several reasons. One of these reasons was that we would have to ensure that noone would cancel an already queued job. Otherwise the whole system would become inconsistent.

Our Solution

There are couple of solutions for this, but every single solution has its advantages and disadvantages. There's one we decided to implement, though. To ensure that we'll always have the correct build number, the script was moved to be part of the Jenkins job. Instead of coordinating everything from the outside and speaking to both, the Bitbucket and Jenkins API, it's only necessary to speak to the API from Bitbucket. This simplified the implementation.

To be able to retrieve information about the pull request to be tested, the Python script needed to be adapted. It had to return the information in a way which can be consumed by Bash, because this is the Shell we use to start the tests. This is also the reason why the script has become a CLI application, which is now used by the Jenkins job.

Usage

Here's the usage of our Bitbucket CLI Client as of writing this document. It will provide you with an overview of its functionality:

bb.py (pr|pull_request) next
bb.py (pr|pull_request) info <pr_id>
bb.py (pr|pull_request) approve <pr_id>
bb.py (pr|pull_request) unapprove (updated|all|<pr_id>)
bb.py (pr|pull_request) comment add <pr_id> <comment>
bb.py commit set tested <hash>
bb.py commit set build_status <username> <repo> <revision> <state> <job_url> <build_key>

The Jenkins job would have to do at least the following things in following order:

Unapprove pull requests which have been approved but changed since their approval.
Find the oldest pull request available and return it.

To be able to prioritize the testing of pull requests, we've introduced an option (see config.yml.examle) which allows you to move pull requests with specific tags at the end of the list of pull requests to be tested.

In our case there's the [wip] (work-in-progress) tag which will be appended at the end of the list. Those pull requests tend to change more often and thus will always be checked last. This ensures that the other pull requests, which are ready to be reviewed, are tested first.
Test the returned pull request and approve it on success.

You may also add comments with the revision which has been tested as well as the name and build number of the job. This information is available through the environment variables in the shell of the Jenkins job.

We decided to leave a comment with the status of the test in both cases, failures and successes. On a failure, the comment will also contain the name of the Jenkins job in charge as well as its build number.
Mark the revision of the pull request as tested.

This prevents the Jenkins job from testing pull requests over and over again, especially those which have already been tested (and since then haven't been altered).

Note that the tested pull requests are marked by their latest revision. This revision will change if a commit is added, the branch is rebased or by anything which changes the code.

How the Jenkins job may look like can be seen in the README.md of the Bitbucket CLI Client. We've shared the repository on Github. It is licensed under the MIT license.

The Problem

Our Solution

Usage

Comments