Testing in Swift

Software tests are great for verifying software behavior and improving the quality of your code. In this talk, we learn from Jeff Hui about tooling, techniques, and writing tested code with the Quick testing framework. He also talked about generative testing, a prevalent functional programming approach (and some of its non-functional applications). Sample code from this talk can be found on GitHub.

---

Why Test? (0:00)

Why should we test software? There are many, many reasons, but I’m going to keep it very simple. The primary reason we write tests is to improve the quality of the software that we have. It increases our confidence in the code changes we’re making because we’re getting feedback of where and how those changes are affecting the software that we’re building.

Testing obviously not the only thing that affects software quality. There are many other things that can affect this, but testing is just one of many of those. There are plenty of different ways of how you test - there’s test driven development, and all different kinds of techniques and strategies for testing. You can do manual testing, but for the sake of this talk I won’t be covering any of that.

When I talk about testing, I’m talking specifically about automated programs that describe intent and verify behaviour repeatably. When I’m talking about automated, I’m saying that there should be one command or hotkey that you can press to run your entire test suite that can verify that you have some degree of competence and that you didn’t completely break the application. There shouldn’t be any complex or manual steps besides this to bootstrap your test suite. Describing intent is actually a very important one because for a test suite, besides knowing that you’ve broken something, you want to know why you’ve broken something. Software always changes, so requirements will change, and when you’re writing tests it’s good to know why are they breaking. The primary benefit for writing these automated tests is that they verify behavior. This is the goal of telling us whether or not our code is behaving as we expect. Finally, but not least, is repeatability. Tools can help you make this repeatable, but in the end it is all of our jobs, when we’re writing tests, to make sure that they’re repeatable and that they don’t flake out and fail occasionally. Otherwise, we lose a lot of the benefits that automated testing gives us when a test suite is flaky and then you’re less likely going to trust the messages that it’s telling you if there’s some behavior here that might not behave as expected. And so these tests should be repeatable regardless of what environment you’re running.

Some Tools

XCTest (4:24)

Apple provides great tooling with XCTest. It comes out of the box and is very easy to setup. In fact, I think now it comes built in so you don’t even have a checkbox that you need to check. In the following basic code, you import XCTest and have a test case with shared setup and teardown.

import XCTest

class TestSort : XCTestCase {
    var values: [Int] = []

    // Shared setup goes here
    override func setUp() {
        values = [2, 5, 3]
    }

    func testReorderingOfSmallerIntegersFirst() {
        // Performing an action
        sort(&values)

        // Assertion
        XCTAssertEqual(values, [2, 3, 5],
            "Expected \(values) to equal [2, 3, 5]")
    }
}

If we take my definition of automated testing and apply it to XCTest, we can evaluate how well it does in this. It’s automated - XCTest provides very nice ways to run this, both from the command line and in XCode. It verifies behavior because we have assertions, we can assert what we want here, and it’s repeatable as long as we’re not doing anything very fishy. But it doesn’t really describe intent. That’s something I’ve chosen to do through the name of this method, which gives the purpose of the test.

Quick (6:31)

So while XCTest is really good, the fact that it doesn’t describe intent tends to lead many developers to other tools like Quick. Quick is a BDD testing frakework that is almost the same as XCTest, but with the additional focus on describing your intent and why you are testing parts of your code.

This code is the same example but in Quick. It looks about the same, but is a little bit longer. Each “describe” is a test case class in XCTest, each “it” corresponds to a test case, and “beforeEach” is a setup. The difference, which is mostly cosmetic, is that you can arbitrarily nest “describe”s, so there’s not some language limitation for that. The API is designed to force you to describe your intent. ```swift import Quick import Nimble

class SortSpec: QuickSpec { override func spec() { describe(“sorting integers”) { var values: [Int] = []

beforeEach {
        values = [2, 5, 3]
        }

        it("reorders smaller integers first in the array") {
            sort(&values)
            expect(values).to(equal([2, 3, 5]))
        }
    }
} } ```

Testing

The App (8:40)

The example app that I’ll be returning to in this talk is a sample app of mine called RandomApp, and it just generates random numbers from the internet. The app is not super complicated and design is not part of the focus, since design is mostly about visual effects and not about behaviours, which is what we want to test. The architecture of the app is composed of a ListViewController, which links to the RandomClient and the DetailViewController. Now where do I put the source files? Your production code should be housing your app, your test code should be in your test bundle, and you should never have files that mix both. Test bundles act as private namespaces for your code, so say if you have a shared class that is referenced in both targets, the test bundle will refer to its own internal representation of the class. This will result in situations that are difficult to debug, so definitely don’t do that.

Networking (14:08)

There are many ways to write networking, a lot of them are taken from the way Objective-C has done testing. There are some really good libraries I would encourage you to try, like Nocilla and OHHTTPStubs. These provide conveniences to test HTTP code, very similar to webmock for Ruby. They have nice APIs and do all the infrastructure to stub out network requests so that under tests, your test suite will never talk to the Internet. This is really useful and valuable, especially if you’re adding tests to an existing application. Both Nocilla and OHHTTPStubs also do method swizzling. That’s much more of an Objective-C feature, and there are plenty of great articles online so I won’t cover that in this talk.

Adapters (15:11)

The adapters pattern is very versatile, and it lets us test other things besides networking. “Adapter” is really a fancy word for protocol. This protocol has just one method, which sends a request and can call a callback. Then, in my production code I can do the usual thing and call out to NSURL Connection. ```swift public protocol HTTPClient { func sendRequest( request: NSURLRequest, complete: (NSURLResponse?, NSData?, NSError?) -> Void) }

public class URLConnectionHTTPClient: HTTPClient { let queue: NSOperationQueue

public init() {
    queue = NSOperationQueue.mainQueue()
}

public func sendRequest(request: NSURLRequest,
    complete: (NSURLResponse?, NSData?, NSError?) -> Void) {
        NSURLConnection.sendAsynchronousRequest(request,
        queue: queue,
        completionHandler: complete)
    } } ```

The benefit that the protocol gives through slight indirection is that now under tests, say if you use a random client, you can create a fake one. We can control to make sure that you’re sending the right request, or you can see what happens when it gets JSON data that it didn’t expect. What happens if you get a network error? You can do this based on this simple adapter pattern.

This adapter pattern applies to many, many more things, and it’s generally a good rule because it keeps your implementation separate from your application’s core logic. This works for networking, logging, analytics, and persistence. So if you need to, say, swap from CoreData to Realm, you could totally do that with an adapter because your API-specific infrastructure has all been centralized. This is also very common for analytics because people always like to switch analytics.

UIKit (18:14)

I’m going to test RandomApp through UIKit directly. If you test using UIKit, there’s definitely a tradeoff that you can make if you have a lot of regressions on user interfaces. But UIKit does provide very many conveniences to actually test drive through it. One of the more common ones is ViewControllers and their management of the view lifecycle. The following two methods invoke that lifecycle, and are public APIs for navigation controlling. The first one we’ll implicitly call viewWillAppear and the second one viewDidAppear, and if you change the first argument of true to false, you can do it for disappear. The ViewController lifecycle methods are pretty much mostly done for you by Apple.

viewController.beginAppearanceTransition(
    true, animated: false)
viewController.endAppearanceTransition()

Similarly, with other controls, there are very similar APIs that you can use. In the following example, if we take a bar button, the button is part of this navigation item, and there’s a there’s a target and action, or in Objective-C terms, an object and a selector that you’re targeting to. Swift currently doesn’t have full support for calling the Objective-C runtime for message sends, so I made a class in Objective-C that’s pretty boring. It just calls performSelector withObject.

// Swift
func tap(barButtonItem: UIBarButtonItem) {
    SelectorProxy(target: barButtonItem.target).
        performAction(barButtonItem.action,
        withObject: barButtonItem)
}

// Objective-C
void tap(UIBarButtonItem *barButtonItem) {
    id target = barButtonItem.target;
    SEL action = barButtonItem.action;
    [target performSelector:action withObject:barButtonItem];
}

Probably the more difficult parts to test is actually some of the more common elements you use because they’re heavily optimized, which is TableViews and CollectionViews. They behave slightly more strangely because they are super-duper optimized. They’re in C++, for the most part, and they are aware in the context of how they’re being drawn. The most unfortunate part for TableViews and CollectionViews is the delegate methods. For delegates, you have to resort to callem them through, and so I prefer calling them through the actual views that are they’re normally assigned to so that I can make sure the wiring is done properly.

tableView.selectRowAtIndexPath(
    indexPath,
    animated: false,
    scrollPosition: .Middle)

tableView.delegate?.tableView?(
    tableView,
    didSelectRowAtIndexPath: newIndexPath)

QuickCheck & Fox (23:20)

There’s always more interesting ways we can test and verify our code. For example, say I have this test case of 2, 5 and 3. Why did I pick those 3 numbers? I just picked random numbers, but what’s the difference between that and if I pick all fives? In this case, if I’m testing a sort, this can be a no op, and so that’s why I specifically chose inputs of particular values that are different.

There’s a tool in the functional programming community that lets us abstract that process of choosing inputs to test, and instead of manually specifying test cases, generates them for us. QuickCheck allows you to generally test your code if you don’t know the problem space well enough or if we’re not sure how our program behaves under certain inputs. You define properties, or universal quantifiers that apply to your code, and describe how the input should behave. It is then QuickCheck’s job to generate test cases to try to falsify your test.

Fox is the version of QuickCheck that I’ve ported for Swift and Objective-C. It generates a bunch of input to test the assertions, and just keeps trying until it finds a failuare. It tries a bunch of different permutations until it finds the smallest possible one, and that is what it uses for its error. Obviously it’s sitll your job to understand why it did fail, because it can only infer the behaviour just by generating inputs, but it provides a good blend between random data generation and making it easier to debug. Fox does that extra step of reducing how much you need to debug, about what is the signal and the noise in a particular test input data.

State (29:01)

If you have state, can you describe it as values? This is like Andy Matuschak’s talk, where values give you power and leverage that is hard to achieve in many other ways. What Fox does is it says that state machines are now values, and it treats one state specifically as a model state.

Model state is some conceptual representation of the user-state, what data your application stores, and is in memory. Fox doesn’t really case about any kind of persistent requirements, and it runs and generates all of its tests in memory, so the implementation for this can be very simplified. If we think about the demo RandomApp, this would be the active requests that are going on and the received numbers that need to be displayed.

Transitions are where you describe to Fox how your program behaves in an abstract level, and how you take this and apply it to your actual application. You have a pre-condiction, which is based off the model state of when it is acceptable to perform this action or behaviour. There’s some code of how you interact with the actual code you’re testing, in this case my app, and how does it change your state of your program. If you want, you can also have assertions - in my app, I have the act of rolling for numbers, and when a network request comes back.

Resources + Q&A (37:01)

• Code from today’s talk
• Quick
• Fox
• Nocilla
• OHHTTPStubs

Q: What are you experiences with integration tests or any things that run the simulator? Do you recommend them, or are they too slow?
Jeff: From my perspective, integration testing is actually really interesting because there’s a lot of potential and value in it. However, I feel that currently they are not reliable enough indicators when your program misbehaves. There’s certainly value in doing it from a full integration of testing through the iOS simulator or many of these other integration tools, but the reliability kind of hinders a lot of its usefulness. By reliability, i Mean that it’s brittle or breaks between versions.

Q: Is speed ever an issue when you’re using generative testing?
Jeff: Not covered specifically in this is that QuickCheck is a semi-random test generator. Part of it includes how to shrink, but part of it is how it generates data, so there are a bunch of tweakable parameters like “how many test cases do I want?”.

Q: If QuickCheck is randomly generating test cases each time, does that mean it’s non-deterministic?
Jeff: Yes. And the way they preserve the same repeatability is that they have a random number generated seed, so if you want to reproduce the same test that you’ve generated before, you use the same seed.

Q: One of the things I really like about XCTest is that I can run just one test case or one class, so I was wondering, how is the Quick XCode integration?
Jeff: Unfortunately XCode has tightly coupled integration with XCTest that doesn’t preclude its impossibility. I also run an Objective-C testing framework called cedar, which does provide Xcode integration.

Q: Does Fox include random generation for building types and containers?
Jeff: Yes. It also includes ways to derive data generation, say if you have your own types or values that you want to generate. You can build it on top of the set that Fox provides and you get shrinking for free.

Q: With Swift there’s the access modifiers, and this is something I found tricky about testing. The tests live outside of your modules so your test code doesn’t look like your actual application code. Do you have any suggestions how to reconcile that?
Jeff: Unfortunately there’s gonna be a lot of public typing, which is just the nature of the current state of Swift. I don’t have any ways to mitigate that other than you have to type more publics.

Q: Swift also has the nonnull and nullable. Can Fox test null, nonnull, and things like that?
Jeff: It provides capabilities to do that, if you choose that API accepts nils or doesn’t accept nils, you can tell Fox to optionally generate nils for its data set.

Q: Given that Swift is, for the most part, not dynamically dispatched, what’s the state of mocking existing objects in Swift?
Jeff: The state of mocking is that if you are using an Objective-C object, something that derives from Objective-C, either through NSObject or one of its other classes, you can do mocking because it uses the Objective-C runtime. For Swift there is none now, although maybe in the future there will be. That’s why I recommend using the adapter pattern, because there are other languages besides Swift that aren’t dynamic and this is a common pattern that they use.

Q: Even knowing that you’re working on an app in a code base that’s mostly Objective-C, what are some of the pros and cons writing unit tests or other test code in Swift? Do you have any other thoughts on doing that sort of interop testing approach?
Jeff: Quick and XCTest do work in Objective-C, so that doesn’t preclude you from writing Objective-C test code for Obj-C code you want to test. I know a lot of people like to write tests in the same language as the implementation. Having a proper type checked language removes some of the worry about having types you didn’t expect in some places. Other than that, I feel like using what you’re most comfortable with will produce software that is of high quality.

Q: I have almost no experience with generative testing, so could you talk a little more about properties? Are they declarative, and how do you provide constraints on the kind of data you want to be generated?
Jeff: If you want to choose other types, you can do it. I know in other languages of QuickCheck, like in Haskell, types are inferred because Haskell has a crazy type system. In other languages, you just specify the type of data you want. Besides data generation, Fox provides ways to compose on top of the existing data generators. Person 2: There’s also a book called Functional Programming in Swift that actually does a QuickCheck implementation in Swift.

Q: How do I generate a very constrained set of multiple sets of input data, so that I’m not writing more test code than implementation code?
Jeff: I’ve been working on dependency resolution algorithm, so you have your set of specifications which have dependencies. I’ve thought about building up more random testing than the few edge cases that I test now, but I’ve been scared away by just trying to make solvable constraints that can hit the edge cases better than just letting users go wild and hit the edge cases for me.

Q: On a practical note, when do you decide that it’s time to stop the generated tests? Say you never hit an edge case for 10 minutes. Is there tooling provided for that?
Jeff: Currently there is not tooling, especially in Fox, about when to stop. Right now it’s based off a number, like how many tests should I generate per property. The default is 200, but that’s just mostly an arbitrary number. On the roadmap is the option to say, generate as many tests within 10 minutes, but it’s not implemented.

Q: Do you have a workaround for when exceptions don’t stop the test when running XCTest? Do you know any other framework that can help?
Jeff: Quick also has the same issue because it’s based on XCTest but Apple has conveniently provided a way to not have to worry about that. There is a property on each XCTest case that’s called continues after failure. You just set that to false and then you’re good!

Q: About your general workflow at Pivotal with testing, are there certain products or certain stages of the product that you are implementing using unit tests? Do you prototype first and then do testing?
Jeff: Pivotal Labs, which is the company I work for, is a consulting company. We are very enthusiastic about test driven development, and so we have unit tests mostly. We mostly drive out code with unit testing, and until recently, we expected from clients, that they come into Labs with execution as a focus. So they’ve done research of what they want to build and how they want to build it. If you’re just doing more exploratory stuff, don’t test it. If you’re just going to write some program and toss it afterwards, maybe there’s not as much value you need to derive from writing tests.

---