CakePHP and PHPUnit

In the recent bakery article concerning the ongoing development of CakePHP 2.0. The already underway migration from SimpleTest to PHPUnit was introduced. I wanted to go into some of the reasons and motivations for that decision as well as explain some of the long term benefits.

Why the change?

Currently there are over 15,000 assertions and 650 files in the CakePHP test suite. A large application can easily have that many tests as well. Changing the underlying test suite framework for a code base of that size is not a small undertaking. Moving CakePHP away from SimpleTest is not something that was lightly decided upon. But there were several contributing reasons that we decided to move to PHPUnit:

  • SimpleTest isn’t E_STRICT. This impairs our ability to create a code base that follows E_STRICT. Having to wade through 1000’s of warnings to find the relevant ones is an arduous task. Moving to PHPUnit makes fixing the E_STRICT warnings in CakePHP easier as there are fewer of them to read.
  • PHPUnit boasts a plethora of ancillary tools. From Continuous integration to build automation. Tooling support has gathered behind PHPUnit, and a switch will allow Cake developers to leverage the great tool support.
  • PHPUnit is the de facto standard testing framework now. Currently Zend Framework and Yii as well as many other projects use PHPUnit. Symfony and Doctrine will be moving to PHPUnit with their next releases. The mindshare and market share is clearly behind PHPUnit. Moving CakePHP to PHPUnit helps ease the developer experience, as new developers may already be familiar with how to use and write tests.

But all my tests are now broken!

Switching unit test frameworks can be a painful process. There are many differences between PHPUnit and SimpleTest internally. We are very aware of this pain as we migrate all of CakePHP’s tests to use PHPUnit. Our aim is that if your test cases extend CakeTestCase you might not even really notice the change. We will be working hard to try and ensure that the various differences between SimpleTest and PHPUnit’s assertion methods are wrapped and just work wherever we can. jose_zap has already ported fixtures over and there should be no difference to you from before. The largest pain point for us so far has been the Mock class SimpleTest provides. Mock objects are implemented very differently in PHPUnit, and there is no easy way to hide the differences. With that said there will be a section in the migration covering how to update your tests.

Coverage improvements

The move to PHPUnit has given code coverage report generation a huge boost. Using the features PHPUnit provides has made code coverage generation faster, and provide more information. Code coverage with PHPUnit now reports on all files loaded during a test case. This gives you a more complete view of what files/objects are interconnected. The PHPUnit test suite also features the ability to find out which test methods covered a specific line of code. Hovering your mouse over any covered line to find out which tests ran through that line. This of course requires a browser that correctly handles title attributes.

I think this is a pretty neat feature, and I hope it shows you some of the improvements we’ll be getting from the switch to PHPUnit. I totally understand that this could be a pain point in upgrading your applications. However, I think in the long run it will be a beneficial change. The ability to access better tooling and a more uniform testing environment with other projects is something that I feel will pay off in the long term.

Comments

So what can a CakePHP developer do today to get ready for the change in the future?

I’ve been neck-deep in tests for the past several days, and after reading your article, I am hesitant to continue writing tests for SimpleTest.

Is there a ‘recommended’ way to get started with PHPUnit in my application now? I realize it might not be clear yet how the CakeTestCase class will change, but if I’m going to write tests, I’d rather write tests I can use in the future.

Thanks!

Ben McClure on 17/5/10

Ben: Well the goal with CakeTestCase is that your test cases shouldn’t have to change very much once the switch happens. The differences between the different sets of assertXX methods will be covered by CakeTestCase. Since 2.0 is a ways off I don’t think there is any immediate cause for concern right now.

mark story on 17/5/10

Great—I haven’t made extensive use of Mock objects or anything too advanced, so hopefully it will translate fairly easily to the new system.

I’m pretty new to testing, or at least using it extensively, but have just started really getting into test-based development, and since that means I’ll end up with a lot of tests, I want to make sure I’m doing it “right.”

Maybe not a good question, but can CakePHP ever be test-framework agnostic, or support pluggable test platforms? CakeTestCase seems like a good step in that direction. It would be great if CakeTestCase could be easily extended to support custom test platforms, while keeping the API consistent on the front-end (as you’re trying to do moving to PHPUnit). As most pluggable interfaces, it would be more work up-front, but could potentially allow the Cake core (and everything people build with Cake) to keep the same test cases and implement their test framework of choice, or switch to a new primary test framework like you’re doing for CakePHP 2.0.

Anyway, it’s probably not a feature that a substantial percentage of users would care about, but Cake is so modular and extendable that it feels like a logical next-step to go along with this change to PHPUnit.

Thanks for your awesome work on Cake, and on the AssetCompress plugin. I was struggling to find the most cake-centric way to combine and minify JS and CSS—I got the minification down, but was not overly happy with any other existing solutions for combining and caching the files.

Ben McClure on 17/5/10

About time you as a community made the move towards the PHPUnit framework – I just have to ponder why it has taken you so long?

Simple Test as a testing framework is way out of date for today’s PHP developer in many areas – it’s unclean and unkept IMHO and it shows… some would say it’s starting to stink?

Glad to hear the move is on the books and the community overall longterm will be thankful of it. Good on you for (finally) taking the bull by the horns!

Les on 19/5/10

Ben: Abstracting away the test framework could require additional code that would also need to be tested :) I also don’t think that its something people would often use, so I’m reluctant to do it.

Les: Part of the delay is due to php4 support. As CakePHP1.2 and 1.3 were php4 compatible, by necessity the test suite also needed to be. Glad to hear you’re looking forward to the changes being made :)

mark story on 20/5/10

I am new to unit testing. guess i will need to work on understanding this better

Harsha M V on 22/5/10

We’re just starting to integrate testing into our development process, is it possible to integrate PHPunit with 1.2 or 1.3 right now? Since I’m learning how to write tests now, might as well learn what we’ll eventually be using if possible. That way I don’t have to do the painful migrations later. I would really like to get a CI setup going too, which sounds easier with PHPunit.

Nathan on 23/5/10

@Nathan:

It sounds like you should just continue down the path of writing tests that extend CakeTestCase, as the functionality you’re using now should continue to work as long as it is wrapped by that class.

We’ll just need to build in the extra functionality PHPUnit offers after they tell us how it will be implemented :)

Ben McClure on 23/5/10

Nathan: Of course its possible, its just a lot of work :) You are by no means forced to use SimpleTest. Its just the test framework that current releases of CakePHP use and have integrated support for. You could however use PHPUnit now to write app tests and simply not use any of the simpletest or any of the related class in CakePHP.

mark story on 24/5/10

Since I’m new to testing anyway, I’m sure I’ll be staying within the realm of CakeTestCase. Thanks Mark and Ben for the info

Nathan on 25/5/10

Glad to hear about this move, even if it is still some way off. The support for phpUnit within build and deployment tools is a major (if under-appreciated) argument for switching. Thanks to the dev team for working out ways to minimise the pain in userland too!

Ian on 28/5/10

this is fantastic news. i am a newbie in cake so currently i only have 4 test files for 4 models that are extensively written.

i am going to start phpunit later this week and rewrite those 4 tests into phpunit.

my strategy is that i am going to concentrate solely on writing test cases for models only and in phpunit.

and use selenium for testing the views and controllers(indirectly)

what do you think? is that a good plan to have test driven development for a cakephp app?

keisimone on 2/6/10

Hi Mark,

what about existing view tests and the support of selenium in PHPUnit ?
Will existing view tests also be portable to PHPUnits SeleniumTestCase ?
Where would you see problems in porting view tests and what should one keep in mind now when writing tests ?

0x20h on 11/8/10

0×20h: By view tests I assume you mean using WebTestCase. There is no planned wrappers to make moving from WebTestCase to PHPUnit’s Selenium integration. They are extremely different solutions, and no automated wrapper would work well. Test cases will be portable insofar of manually adapting them.

I’ve never used WebTestCase, primarily because it lacks the ability to do any testing of interfaces that use Javascript. Selenium has always been my choice tool for doing automated UI testing. Even if there was a way to migrate from one to the other, I wouldn’t know about it.

mark story on 11/8/10

How can I use the feature to find out which test methods covered a specific line of code? It works in eclipse? I using phpunit for a while and never saw that.

Thanks!

Eric Saboia on 26/10/10

Eric: As mentioned in the article, with the test suite in CakePHP 2.0, you’ll be able to hover over lines on the coverage reports from the web-runner. Doing so will let you see which tests covered each line. This content wouldn’t be available in eclipse, unless you ran the web-runner from inside eclipse.

mark story on 29/10/10

Mark,

I am looking at integrating PHPUnit with our existing cakePhp 1.2.3 framework by replacing simpletest. Do you have any notes on how you ported PHPUnit to cake 2.0? If so, can you share them with me? Google wasn’t helping me much here.

Karthik Ramakrishnan on 4/11/10

Karthik: You might want to look at the commits related to the changes done from 1.2->1.3 as well as the changes done between 1.3->2.0. Inside cake, there are only a handful of files related to the testsuite, and they are pretty much all in cake/tests/lib. As for specific instructions, unfortunately I don’t have any beyond the commit messages.

mark story on 8/11/10

For anyone who wants to install/update PHPUnit the easy way, Hyra and I wrote a Plugin that works with Cake2:
https://github.com/dereuromark/PHPUnit-Cake2

Just run `cake Phpunit.Phpunit install` from your command line to download, unzip and move all files automatically to the Vendor folder specified.

mark on 21/2/12

Have your say:

*
* You can use Textile markup, but be reasonable