Building a new migrations backend for CakePHP

During the development of CakePHP, we also wanted to update migrations to be compatible. However, we had stumbled into a tedious dependency graph. Currently phinx relies on cakephp/database. While cakephp/migrations depends on phinx. In order to do a major release on cakephp/migrations (to provide CakePHP 5.0 compatibility), I would also need to do a major release to phinx, which could only be done after cakephp/database was published alongside with the framework.

Late into the release process on the night of the 5.0.0 release, all of this came to a head as I didn’t want to leave cakephp/app in a broken state, so I chose to do a major release and tagged 2.0. My rationale was that I couldn’t just drop the 5.x database package onto the existing user base of phinx in a minor release. cakephp/database was a core dependency of phinx and it contained breaking changes. I didn’t really enjoy the tangled web we had created. However, I also wanted migrations to work better with the ORM. I saw this as a good reason to start investigating how easy it would for migrations to adopt enough of phinx’s API shape that backwards compatibility could be maintained. Migrations is an often neglected project because the maintenance is complicated. The test suite is fragile and tedious to work with so few people do it. Perhaps I could contribute to improving this as well, as migrations are a vital part of an application’s life-cycle.

Inline phinx into migrations

I decided that I would ‘inline’ or merge phinx into migrations. Migrations would continue to provide the same user facing API as it did before. The only impact to applications should be renaming Phinx classes into Migrations provided versions with rector. One of my constraints was that there couldn’t be any frivolous userland API breakage. I would validate this approach by providing an opt-in flow for teams to be able to get stability and be able to adopt the new migrations backend incrementally. If your migrations work with phinx but are broken with the new backend, it is likely a bug in the plugin.

Current architecture and goals

At a high level, migrations and phinx structured like:

The blue boxes in these diagrams are ‘public’ interfaces that end users interact with frequently. They also drag in other public surface but that is too complex to illustrate here. White components either wrap phinx, or are coming from phinx. Red boxes will indicate migrations native components. In future states, the boxes will change color. The change in color denotes the availability of a migrations built-in version. The phinx versions are still available. Starting with the most important public API – Migrations/AbstractMigration and Migrations/AbstractSeed. These classes are special for a few reasons:

1. They are migrations namespaced classes but are also tightly coupled to phinx through inheritance.
2. They are the public API of migrations and cannot change in a breaking way for common usage.

My plan was to gradually reduce the number of components that interact with phinx until it was the minimal set. Phinx is a well designed piece of software, and its interfaces gave me enough control to replace its internals with a shim adapter.

Building the new Db package

I chose to start with the bottom of the tree, and began by importing the Db package from phinx. I incrementally imported all of the code in Phinx/Db until I had a fully Migrations namespaced solution. I revised the connection management to use CakePHP’s Connection class directly, reducing complexity. Having a full Migrations/Db package let me ‘build up’ to the interfaces that migrations and seeds rely on. I also needed to implement a translation layer that would implement Phinx interfaces but be Migrations code under the hood. The PhinxAdapter implements this concept via simple call forwarding methods like:

Show Plain Text
  1. public function insert(PhinxTable $table, array $row): void
  2. {
  3.     $this->adapter->insert($this->convertTable($table), $row);
  4. }

As well as a translation layer to convert both parameters and return values from Phinx to Migrations and the reverse to cover both parameter and return values scenarios. These conversion methods are all simple mapping operations that look like:

Show Plain Text
  1. /**
  2.  * Convert a phinx table to a migrations one
  3.  *
  4.  * @param \Phinx\Db\Table\Table $phinxTable The table to convert.
  5.  * @return \Migrations\Db\Table\Table
  6.  */
  7. protected function convertTable(PhinxTable $phinxTable): Table
  8. {
  9.     $table = new Table(
  10.         $phinxTable->getName(),
  11.         $phinxTable->getOptions(),
  12.     );
  13.  
  14.     return $table;
  15. }

I now had a bi-directional translation layer that implemented the `phinx` interfaces, but was also compatible with the builtin implementation. With the entire Db package migrated to migrations, the component conversion looked like:

Manager, Environment, and Commands

Following the Db package I migrated the Environment, and Manager classes into migrations. I now had a parallel implementation of the internals of phinx. I chose to make using the migrations based implementation an opt-in switch controlled by Migrations.backend. This option would allow the plugin to choose which implementation to use. The optimal place to put the fork in logic was in MigrationsPlugin::commands(). If I had ‘builtin backend’ commands separate from the phinx based ones I could greatly reduce the risk of breaking applications. I could also benefit from simpler internals quicker, and make my future cleanup work easier. With the Manager, Environment and Commands compatible with the builtin backend our conversion state looked like:

The feature flag hook logic was pretty simple too:

Show Plain Text
  1. public function console(CommandCollection $commands): CommandCollection
  2. {
  3.     if (Configure::read('Migrations.backend') == 'builtin') {
  4.         $classes = [
  5.             DumpCommand::class,
  6.             EntryCommand::class,
  7.             MarkMigratedCommand::class,
  8.             MigrateCommand::class,
  9.             RollbackCommand::class,
  10.             SeedCommand::class,
  11.             StatusCommand::class,
  12.         ];
  13.         // Simplified for brevity
  14.         $found = [];
  15.         foreach ($classes as $class) {
  16.             $name = $class::defaultName();
  17.             $found[$name] = $class;
  18.             $found['migrations.' . $name] = $class;
  19.         }
  20.  
  21.         $commands->addMany($classes);
  22.  
  23.         return $commands;
  24.     } else {
  25.         // Connect the phinx based commands
  26.         return $commands;
  27.     }

At the plugin setup level, migrations reads application configuration and changes its behavior. Compatibility with the CLI tool interfaces was done by re-implementing the same options set with the same semantics.

So far, I have loved this approach. It allowed me to not disrupt the community, while also letting me build in public. New and experimental features behind opt-in feature flags. It has been great to see community involvement in reporting issues in this feature as it develops. It is greatly appreciated.

What’s next

Up until this point, I’ve been covering my process of rebuilding migrations thus far. There is still more to be done on this project though. My next milestone will be to replicate the base migration class logic and update migration generation to use that base class when the builtin backend is selected. This means that users can opt into moving their application onto the builtin backend without compatibility for phinx. I’m hoping this provides a gradual enough upgrade that will only require minor changes to fully convert your code base, and those should be doable with rector.

If you use migrations, I’d appreciate you trying out the new backend

Comments

There are no comments, be the first!

Have your say: