I’ve recently been working quite a bit on a worker application in python. This worker picks up tasks over a gRPC service executes those tasks, and publishes the results via gRPC. Because of python’s Global Interpreter Lock (GIL), threads don’t help for CPU intensive workloads. Instead, I’m using multiprocessing to run additional processes per worker and get better CPU utilization at the cost of using a ton of memory. When a python process opens a child process it can also share state with the child process. The most interesting of these are synchronization and communication primitives like Event, Queue, Pipe, and Lock.

Recently, I’ve been working on a system that needs to run tasks scheduled by either web requests, or other tasks. These tasks are trusted but sometimes problematic code. While developers generally write good code, we all make mistakes and the occasional slow job that can run for multiple hours gets written. These slow tasks can create problems if the volume of them goes up as they can consume all available resources.

Recently I shipped a new feature to Stickler CI that enables users to extend Javascript and Python builds with additional packages. Maintaining review tool dependencies can be a drain on your team’s time. Stickler CI helps solve this problem, but used to come with a tradeoff of not being able to fully customize your style rules.

I try to keep fairly busy. Between work, being a dad, and working on existing open source projects, I found time to work on two new ones.

Xhgui2

Profiling is a very interesting topic for me. I love spending time sifting through results trying to find ways to make code run faster or use less memory. XHProf is a C-extension created by Facebook.

I’ve recently been working a fair bit on the new documentation for CakePHP and while sphinx is amazing, it doesn’t come with a built-in domain for generating PHP documentation.

In a previous article I covered how CakePHP would potentially be moving to using sphinx for the 2.0 documentation. Myself and some of the other CakePHP developers have been working on this option, and seeing if it has any legs. Turns out that sphinx is actually a pretty great tool.

In the release announcement for 1.3.7, it was tentatively announced that CakePHP would be moving its documentation over to ReST, Git and sphinx. Having documentation in a git repo, and using sphinx to generate documentation has a few nice wins, that would be difficult to achieve with the current book application.