Evolution of Container Usage in Stickler CI

Stickler CI is a software as a service application that automates a tedious part of code review; enforcing consistent style and preventing lint errors. By integrating with GitHub, Stickler checks each pull request for style errors and post review comments when an error is found. This helps your team align on coding standards and provide more valuable feedback. Stickler is free for public repositories; private repositories require a paid plan.

Containerless prototype

When I started building Stickler, the first implementation ran all linting tools on the host filesystem as a child process.

This design required having all linting tools installed on the host operating system. Getting all the linters to play together on the host operating system was challenging as a couple of them had esoteric and hard to build dependencies. This design could lead to secret information being accessed if someone managed to exploit a linter. A few supported linters allow users to provide their own rules. These linters posed the greatest risk and needed to have their custom rule capabilities disabled. I knew this was not an acceptable long term solution as it hobbled the product, and I wasn’t confident that all threats were mitigated. However, I wanted to make sure the product would be used before I invested in more complex solutions.

Kitchen sink container

After acquiring some users and gaining confidence that the product was useful, it became time to fix the prototype. In the next iteration, linter execution was moved into a docker container. Moving the linters simplified the reviewer dependencies. The new image included both parts of the Stickler application code and a sandbox to run linting tools in. The image used for this container was a ‘kitchen sink’ that contained all 13 linter tools Stickler supported. Getting the various dependencies installed resulted in a long Dockerfile. This approach resolved the obvious risks I knew about and allowed custom lint rules for PHPCS and ESLint to be enabled.

As I continued to develop with this image some shortcomings became apparent.

• The long Dockerfile was creating a massive image. 3.5GB in total. This made updates to certain tools expensive as they required rebuilding multiple layers.
• Deploying new review job code also required building new images. Thankfully the application code was the last layer, but needing to build images complicated the deploy process more than I’d like.
• The massive image resulted in long startup times for containers.
• Because the container contained application code and linters, I needed to drop privileges to mitigate exposing sensitive data stored in /proc. This added complexity to the code, and I wasn’t sure I had prevented every potential threat.

Tiny Containers

As the workflow issues with the massive image became more annoying, I began to think of solutions. Given that many issues stemmed from the size and complexity of the Dockerfile and its resulting image, I wanted to try smaller images. These images would only contain the linter tools and no application code. The application code would be responsible for cloning the repository and using docker volumes to mount the cloned code into the containers. Each container would encapsulate only a one tool chain. For example, all the python2 tools.

My hypothesis was that single tool chain images would be smaller, easier to maintain, start faster and be safer to operate. In pursuit of smaller image sizes, I switched from Ubuntu to Alpine Linux shaving more megabytes off of each image. The architecture now looks like:

Adopting smaller images meant that I had to do changes to how reviews were run. Instead of running tools as simple sub-processes, each tool was run in a separate container. Having tools run in containers that only have access to the repository code fully isolated linter execution from the Stickler code and host operating system. Switching to smaller images was a huge change that required rebuilding all path handling throughout the review execution. With it complete, I’m very happy with the results. I’ve been able to reap a number of upsides from the adopting smaller images:

• Stickler is now able to offer both python 2 and python 3 linting
• Each image is light, simple and contains no proprietary code.
• Containers start up fast.
• Application deployments don’t involve building and pushing docker containers anymore.

As Stickler continues to grow, I’m sure there will be more opportunities to revisit this solution and improve it further.