PrEV
Thoughts from a NeXTStep Guy on Cocoa Development

Unit Testing with Xcode 3.0

Dec 29, 2007 by Bill Dudney

Coming from the Java world after a long absence from the Cocoa/ObjC world had gotten me hooked on unit testing. So over the last year or so as I've built various Cocoa applications I've always felt a bit out of place without some sort of automated developer tests. I have built a few but the SenTestKit was always such a pain to setup and stuff that I've not been very diligent.

Recently I've been developing a physics engine (a really simple one) for a simple game I'm writing. Well 3D geometry and physics simulation is not something I feel comfortable with building without some sort of automated developer tests to make sure I'm not hosing stuff as I go along and add features and fix bugs. So I have gotten serious about making unit testing work. While I still don't have all the details figured out I do have something working and I thought I'd share what I have on my blog so that I will always have a place to come back to when I forget the rather long and complex path to getting unit tests setup. And yes I have filed bugs for both the complexity and the lack of docs.

And no I won't try to sell you on the virtues of unit testing. It has worked well for me for years in the Java world and generally it just makes sense to me that I should use software to take care of boring repetitive tasks for me.

A quick procedural note, I refer to the images frequently in the text. They are grouped, so when you click on any one in the group you can move between them with next and previous buttons. The idea is that you can read the text then click on the images to follow along on your app.

A real quick and dirty intro to unit testing ObjC. The SenTest project started as an open source JUnit contemporary for testing ObjC. Apple liked it so much they added support for it to Xcode and eventually assumed responsibility for the whole project. The basic idea behind SenTest is that you write your tests just as you would any other code, they live in the project they test. The test code however is not part of the actual code base, instead the tests are compiled into a bundle that can be loaded dynamically if requested. That way our tests can live next to the code they test but won't go out with the release build of our apps.

I have created a really simple example app for you to grab if you want to see the finished product. The project is available here.

Before we get started lets do a quick list of steps.

  • Create a new Target for our unit tests to live in.
  • Configure Application Executable for Test Bundle Injection.
  • Configure the Test Bundle for injection.
  • Configure GDB to run the app internally.
  • Add tests to your Test Bundle.
  • Run and debug the tests.
  • Live Happily Ever After.

Create a New Target

So lets get started making tests. First step is to add a new target. This new target is responsible for creating the unit test bundle that will be loaded dynamically when the tests are run. The image group walks you through these steps. You should control click (or right click) on the Target group and select Add->New Target... In the wizard that pops up choose Cocoa->Unit Test Bundle (be careful to choose Cocoa instead of Carbon) then click next. Name the target (I named it UnitTests and will refer to it like that in the rest of the tutorial) and click finish.

And the final step is to add the application as a direct dependency of our unit test bundle. That will make sure the app is built even when we have our unit test bundle set as the active target.

Step 1

Configure Application for Test Bundle Injection

Recall that our test bundle must be injected into our app at run time to get the tests to run (we could set up an alternate executable to run the tests but this way they will be built all the time instead of when we think about doing it). The first step to doing that is to open the inspector for your executable. Choose the executable and right click and choose Get Info or use Command-I. Next we want to add the arguments and environment variables that will cause our tests to be run. First add the argument '-SenTest All'. That causes all our tests to be run, we can also specify specific individual tests to by specifying the test class name or the test class name followed by '/' and the method name, like this MyTestClass or MyTestClass/testIt.

Next we need to specify which bundle to dynamically load, that would be our UnitTests.octest bundle. When we used the wizard to add our testing bundle it setup the extension to 'octest' for us. So all we have to do here is specify it. We add an environment variable called XCInjectBundle and make its value UnitTests.octest (or whatever you named your test bundle).

Next we specify the injection bundle called DevToolsBundleInjection.framework. We do this via the DYLD_INSERT_LIBRARIES environment variable to $(DEVELOPER_LIBRARY_DIR)/PrivateFrameworks/DevToolsBundleInjection.framework/DevToolsBundleInjection. This is a flag used by the dynamic linker. When set (as a colon separated list of libraries) it tells the linker to load these libraries before any referred to by the program that is being loaded. This framework will take care of performing the injection of our test bundle for us.

And finally we set the DYLD_FALLBACK_FRAMEWORK_PATH to $(DEVELOPER_LIBRARY_DIR)/Frameworks so that the linker can find the SenTestingKit framework. This is part of the process that I don't fully understand. From reading the docs it looks like this gives the linker clues about where to find libraries that are not in their installed directory. The SenTestingKit.framework is compiled as a relative framework with the @rpath as its initial root directory. This flag apparently tells the dynamic linker where to look for @rpath stuff. If anyone has any better insight into this please comment so I can clean this paragraph up. Thanks!

Step 1

Configure Test Bundle for Injection

For loading the bundle we need to set up the BUNDLE_LOADER and TEST_HOST build properties for the tests bundle. Recall that our application is going to be the bundle loader so we just need to specify the application executable as the BUNDLE_LOADER. Select the unit testing target and open the inspector (Comman-I, or right click Get Info) and then choose the Build properties tab. Then type bundle into the search field. Double click on the Bundle Loader field and set its value to $(BUILT_PRODUCTS_DIR)/YourAppName.app/Contents/MacOS/YourAppName. Then go back to the build properties list and double click on Test Host and set its value to $(BUNDLE_LOADER). These two build flags tell the test machinery the place that the testing bundle should be injected.

Step 1

Add Tests to the Test Bundle

And finally we can add a test. I usually add a Test group so I can organize my tests but it won't matter if you do that or not for the tests. Right click on the group you want to add the test to then choose Add->New File... When the wizard comes up choose Cocoa->Objective-C test case class and hit the Next button. Then name the test whatever you'd like. Make sure to set its target to the unit testing bundle instead of the default of your app.

Step 1

Add Code to the Tests

Now lets write a test. In this code we are doing nothing special, the setUp method creates the object we are going to test and the testAdd test to make sure 4 plus 5 really does come put to 9. Any method that starts with 'test' will get executed.

- (void)setUp {

    // create the object(s) that we want to test

    [super setUp]

    testable = [[Testable alloc] init];

}


-(void)tearDown {

    // clean up any stuff like open files or whatever

    [testable release];

    [super tearDown];

}


- (void)testAdd {

    STAssertEquals([testable add4:5], (NSUInteger)9, @"should add 4");

}

Configure GDB

And finally one last thing. You need gdb to run the exe internally instead of launching a separate shell. To do that edit the file ~/.gdbinit and add set start-with-shell 0 to the file.

Now switch the active target to you unit testing bundle and hit build and go. If everything is setup correctly your app should come up and the test (that we have not written yet) executed and then your app should shutdown.

Run and Debug the Tests

And finally we can also debug our tests. Set a breakpoint in a test (say one that is failing) and then you can step though the code and figure out what is wrong. Sure beats log statements :) The trick is to use the debugger directly instead of using Build and Go. Use Alt-Command-Y or just hit the Go button from the console.

Step 1

And there we go, debugable unit tests in our Cocoa apps!

This tutorial tries to be exhaustive and show you ever single step to getting going. If you follow along with your own code and something does not work, please let me know so I can fix it. Hope you enjoyed it and that it helps you get your code tested.

And finally much thanks to Chris Hanson without whose blog entries and help on the Xcode users list I'd have never figured this out.



Comments:

Thanks for the plug! I just wanted to send a couple of corrections your way...

  1. OCUnit wasn't a clone of JUnit, it was Sen:te's reimplementation of Kent Beck's original Smalltalk Testing framework. So it's really more of a contemporary to JUnit rather than a clone of it.

  2. You don't need to add SenTestingKit.framework to a Cocoa unit test bundle target, it's specified in "Other Linker Flags" along with Cocoa.framework already. If you want, though, you can remove the customization from the "Other Linker Flags" build setting and then add both of those frameworks as frameworks.

  3. The use of @rpath is documented in the dyld(1) man page. A concise summary is that an executable can have a set of "runpath search paths" compiled into it, and libraries built with an install name relative to @rpath will be looked for in those paths. If they're not found, then DYLD_FALLBACK_FRAMEWORK_PATH and DYLD_FALLBACK_LIBRARY_PATH will be searched.

  4. You should always invoke [super setUp]; first thing in your -setUp override and invoke [super tearDown]; last thing in your -tearDown override. I didn't do this for years and it works fine if you just subclass SenTestCase (whose implementations are stubs). But it's useful to get into the habit because then you can extract test case superclasses that do extra work in these methods, and ensure that they'll get called at the appropriate time.

Your article is great and I hope this feedback is useful, feel free to get in touch if you have any further questions.

Posted by Chris Hanson on December 29, 2007 at 08:17 PM MST #

Thanks for the comment and clarifications Chris!

I updated the code to call super setUp or tearDown and that is a great tip that SenTestingKit is already part of the framework list.

Posted by Bill Dudney on December 29, 2007 at 10:53 PM MST #

Great walkthrough. Thanks a lot. After doing so much work in java and ruby lately, I don't think that I could have survived without this.

Posted by Omar Qazi on December 30, 2007 at 01:13 PM MST #

Thanks for consolidating all this info into such an easy walkthrough. It works great with one small correction: I believe BUNDLE_LOADER should include $(BUILT_PRODUCTS_DIR) instead of $(BUILD_PRODUCTS_DIR).

Posted by Scott Guelich on March 24, 2008 at 08:00 PM MDT #

Have you tried running unit tests with the new iphone SDK?

Posted by elaine on March 24, 2008 at 08:00 PM MDT #

Thanks Scott! Strange that it works for me even though not documented... Ah well the joy of Xcode build settings... :)

Posted by Bill Dudney on March 25, 2008 at 02:24 PM MDT #

I set BUNDLE_LOADER and TEST_HOST and my test work great. I have been playing around with lcov to show my coverage. Since I setup ocunit this way lcov doesn't seem to know the source code I am using. Has anyone have experience with lcov?

Posted by Chris Pesarchick on May 09, 2008 at 05:48 AM MDT #

1. I am new to everything MAC .. coming from JAVA server side dev

I am trying to debug your test app(I downloaded the project) and I am getting nothing. I have set a breakpoint in the Testtable.m add4 method. This method is never hit. I also set a break point in your main method and it was hit... So it seems like the Test are never getting started but I have no idea why...I set all the env variables and arguments you listed above to no avail.

I am using Xcode 3.1.. Would that make any difference?

Any Ideas... I have burned over a day on this..

TIA

Posted by Scott Shealy on June 30, 2008 at 06:48 PM MDT #

I am having the same issue, in both your sample project and my own, my breakpoint on my test method is never being called when i select '10.5|Debug|Unittests|i386' to build & go... this is on xcode 3.1

Posted by Jonathan Robson on July 26, 2008 at 08:40 AM MDT #

If you are looking for an iPhone unit testing stack (which is based on OCUnit) you may want to check out:

http://code.google.com/p/google-toolbox-for-mac

Posted by Dave MacLachlan on September 04, 2008 at 06:45 PM MDT #

Hey Bill,
I've tried this on Xcode 3.1. I can make everything work except debugging the unit tests. When I set a breakpoint inside the unit test and try running it, I haven't been able to find any way to make it hit the breakpoint. (Details on xcode-users or Dev Forum.) I've also seen feedback from several other people, none of whom have been able to make it work either.

Would you do me the favor of verifying that you can still do this on 3.1 and let me know what steps you took in this new version?

Thanks,

Pat

P.S. What ever happened to that Tenleytown NeXTStep contract we tried to bid on? P.M.

Posted by Pat McGee on September 10, 2008 at 07:12 PM MDT #

There's a fourth environment variable, XCInjectBundle, missing from the section "Configure Application for Test Bundle Injection". It should be set to the full path to the executable into whose process space the unit test bundle (with the default extension of octest) should be injected at runtime.

Posted by Alfonso Guerra on January 25, 2011 at 10:57 PM MST #

Post a Comment:
  • HTML Syntax: Allowed