Building Custom Builds for Stickler CI

Recently I shipped a new feature to Stickler CI that enables users to extend Javascript and Python builds with additional packages. Maintaining review tool dependencies can be a drain on your team’s time. Stickler CI helps solve this problem, but used to come with a tradeoff of not being able to fully customize your style rules. While our default images come with many popular packages pre-installed, it is impossible to have every package for flake8 and eslint pre-installed.

Enter Custom Builds

After fielding a few package requests, I knew there had to be a better way. Stickler does all of its reviews inside docker containers with temporary filesystems. The solution seems simple enough. Before running each job find and install the additional packages required by the current repository. Then once the review is complete, docker could discard the container filesystem, leaving a pristine state for the next review.

This initially seemed simple enough. However, in addition to reviews, Stickler CI can also automatically apply any fixes from your linting tools. In order to avoid wasting a significant amount of time downloading and installing packages multiple times, I needed a way to share state between the fixer and review processes.

To solve this performance issue I chose to create temporary images. When a named container, docker will retain the container state. This state can then be turned into an image with docker commit. With a custom per-job image, I would be able to offer custom builds without wasting CPU and network. I chose to start with eslint as all of the additional package requests were for eslint.

Installing Additional Packages

Installing additional packages should be simple, but the real world is messy. As a GitHub App, Stickler is unable to access private NPM registries or pull dependencies from other git repositories. Furthermore, installing all of an application’s dependencies could be slow and expensive time wise. To workaround this only, the package installer searches for packages matching eslint-(config|preset). Only packages matching this pattern are installed. Thankfully most ESLint configuration packages are open-source and small enough that they don’t take long to download.

The end flow roughly looks like:

1. Create a named container that uses a tool specific install script.
2. The install script uses regexp to find package names & versions.
3. Each package is then installed into the image.
4. The resulting container filesystem is committed into a new image.
5. The fixer command is run in our new image, and any changed files are committed.
6. The check command is run and errors are reported to GitHub.
7. The image created in step 3 is deleted.

Once I had this flow implemented for eslint I started beta testing it with customers. When a customer contacted me about additional eslint packages, I would give them instruction on how to use the plugin installer. I used this beta group to gain confidence in the approach, monitor how custom images operated, and fix a lingering bug or two.

Once I was happy with the solution, I implemented a similar flow for the flake8 integration. Finding which python packages should be installed is even more complicated than in Javascript. There are far too many places, and ways a Python application can store its dependencies. Because of this I chose to make the flake8 plugin installer require users to provide a list of plugins they use. Having an explicit list requires a bit more work from the user, but ensure that reviews are always deterministic and work the first time.

Future improvements

I’m quite happy with the solution I arrived at. It has been operating quite well throughout its beta and public releases. In the future, I would like to improve a few things. First, would be to keep a pool of frequently used custom containers around to help further improve review times. I’d also like to expand the number of linters that include custom builds. Ruby and PHPCS are the next likely candidates.