Xcode and Asynchronous Unit Testing

Unit Test support is greatly improved in Xcode 5, with new features such as the Test Navigator and individual test runs. XCTest is the new framework, replacing OCUnit. I’d hoped there would be better support for testing asynchronous calls with all of this new stuff, but sadly this is not the case. So, what’s the issue here?

Some methods in the iOS API work asynchronously, on a background queue, and are completed on a completion block, at some point after invoking them. Here’s an example:

- (void)testSaveAndCreateDocument {
    NSURL *url = ...; // URL to file
    UIManagedDocument *document = [[UIManagedDocument alloc] initWithFileURL:url];

    // Call the asynchronous method with completion block
    [document saveToURL:document.fileURL
        forSaveOperation:UIDocumentSaveForCreating completionHandler:^(BOOL success) {
            STAssertTrue(success, @"Should have been success!");
        }];
}

In this Unit Test, we’re testing the creation and saving of a blank UIManagedDocument. The completion handler block is invoked when the operation completes and we test for a True value of the “success” block parameter. If we run the test from Xcode it will pass. It will always pass, and never fail. That’s because the test exits before the completion handler is invoked. We can check this is so by setting a breakpoint on the following line:

STAssertTrue(success, @"Should have been success!");

Run the test. The breakpoint is never reached and so we can never test the value.

So how can we ensure that the test method waits until the asynchronous method invokes the completion block? There have been a few solutions proposed on places like Stack Exchange, and after playing with a few, I use the following one.

In this solution, we need to declare a Boolean flag that indicates that we’re waiting for the operation to complete, then call the asynchronous method, and then block the run loop until the test completes when the flag is set to NO.

Here’s how we do it. We need to declare our Boolean flag before calling the asynchronous method:

__block BOOL waitingForBlock = YES;

Then we must set it to NO inside the method’s completion block:

waitingForBlock = NO;

And loop while the condition is true:

while(waitingForBlock) {
    [[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode
                             beforeDate:[NSDate dateWithTimeIntervalSinceNow:0.1]];
}

Here’s the full code:

- (void)testSaveAndCreateDocument {
    NSURL *url = ...; // URL to file
    UIManagedDocument *document = [[UIManagedDocument alloc] initWithFileURL:url];

    // Set the flag to YES
    __block BOOL waitingForBlock = YES;

    // Call the asynchronous method with completion block
    [document saveToURL:document.fileURL
        forSaveOperation:UIDocumentSaveForCreating completionHandler:^(BOOL success) {
            // Set the flag to NO to break the loop
            waitingForBlock = NO;
            // Assert the truth
            STAssertTrue(success, @"Should have been success!");
        }];

    // Run the loop
    while(waitingForBlock) {
        [[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode
                                 beforeDate:[NSDate dateWithTimeIntervalSinceNow:0.1]];
    }
}

Run the test again, and this time the breakpoint is reached.

This seems somewhat cumbersome, especially if we need to do this in many Unit Tests. So we can convert this into a set of Macros and declare them in a supporting header file:

// Set the flag for a block completion handler
#define StartBlock() __block BOOL waitingForBlock = YES

// Set the flag to stop the loop
#define EndBlock() waitingForBlock = NO

// Wait and loop until flag is set
#define WaitUntilBlockCompletes() WaitWhile(waitingForBlock)

// Macro - Wait for condition to be NO/false in blocks and asynchronous calls
#define WaitWhile(condition) 
do { 
    while(condition) { 
        [[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode beforeDate:[NSDate dateWithTimeIntervalSinceNow:0.1]]; 
    } 
} while(0)

Now we can simplify our test:

- (void)testSaveAndCreateDocument {
    NSURL *url = ...; // URL to file
    UIManagedDocument *document = [[UIManagedDocument alloc] initWithFileURL:url];

    // Set the flag
    StartBlock();

    // Call the asynchronous method with completion block
    [document saveToURL:document.fileURL
        forSaveOperation:UIDocumentSaveForCreating completionHandler:^(BOOL success) {
            // Set the flag to NO to break the loop
            EndBlock();
            // Assert the truth
            STAssertTrue(success, @"Should have been success!");
        }];

    // Run the Wait loop
    WaitUntilBlockCompletes();
}

Using the Macros makes things much clearer. We’re simply initialising a flag to YES with the StartBlock() pseudo-function, setting it to NO with EndBlock() and waiting for the flag to be set to NO in the WaitUntilBlockCompletes() call.

It’s not the most elegant solution, but it suffices for my needs.

I’ve uploaded the Macros to GitHub as a Gist:

https://gist.github.com/Phillipus/6537635

A caveat – sometimes this solution does not work using Xcode 5 with iOS 7, especially when dealing with a UIManagedDocument. In some cases there seems to be a race condition or threading issue going on. (See the workaround in the comments for UIManagedDocument.) Also, with the iOS 7 simulator, I’m seeing more and more cases where this wait loop is not working, and some unit tests are not being run. It’s as if some tests are not waiting for the loop to break and failing to run. I’ve tried increasing the value of dateWithTimeIntervalSinceNow in this line of the macro:

[[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode beforeDate:[NSDate distantFuture]];

But it seems as if the main UI thread seems to get stuck in some tests, and can only be unstuck by clicking on the simulator, or waiting a few seconds. Something has changed in Xcode 5 and/or iOS 7, but I don’t know what it is. If you have problems with this, experiment with the date value in this line. It may just be an issue with the simulator. Try testing on the device as well.

I still have my reservations about this technique, and I’m still looking for the perfect solution for asynchronous unit testing in Xcode. You would think that Apple might have provided a solution in XCTest, perhaps similar to the implementation in GHUnit.

Update 14 July 2014 – Good news! Apple has introduced a great new framework to support asynchronous unit testing in Xcode 6 (still in beta at the moment). I’ve written a new post on how to implement this here.