Supercharging Your Android Testing — JUnit & Espresso

Testing almost always helps catching bugs and write better code, but it can be a huge pain without a proper framework and setup. At the Bay Area Android Dev Group meeting, Ellen Shapiro explains how to super-charge your Android testing, including UI testing, with JUnit 4 and Espresso 2. She demonstrates both technologies, providing tips and best practices to use testing painlessly in your projects.

---

Introduction (00:00)

Over the last few months, a couple of useful tools have taken good shape: the JUnit4 Android runner and version two of the Espresso UI testing library. These are very much intertwined, and I want to start with the JUnit4 runner.

JUnit4: What is? (00:19)

JUnit4 itself is actually a huge open-source testing library for Java. It has been around for ages. The version four was actually initially released in 2006, but it is just now coming to Android for reasons I am going to explain in a minute.

The biggest feature of JUnit4 is that it has subclassable test runners, which make it easier to write runners for the various different things that you can do with Java. For example, there is one feature that specifically runs Scala tests, and many others that run very specific things (i.e. Spring).

Testing: Ye Olde Way (01:03)

Google introduced the JUnit4 runner as part of Espresso 2.0 in December 2014. Before that, the Android test landscape looked like this:

ActivityInstrumentationTestCase2<T extends Activity>

You always had to subclass this nonsense and pass into it the Activity class, because it used JUnit3’s test runner. It subclassed a whole bunch of stuff all the way back up to JUnit’s actual Assert class, mostly so you did not have to import all of the Assert methods.

In the old version, the name of the test had to be started with the word test. That was something that seemed really goofy to me, as someone who was learning to code: “How do I start to distinguish between what each of these tests is doing, when every single test starts with the word test? Why is the word test there in the first place?” That was annoying and redundant after a while.

If you ever had a test where you wanted it to NOT run because you know you need its functionality at some point, but you did not want it to be constantly monitoring your work (i.e. your server is temporarily down, or you have a third party library where somebody broke something in it), you could comment that case out or temporarily delete it. Then you would have to restore it from version control. That was annoying. If you ever wanted to test the same functionality with a whole bunch of different datasets, you would have to create a private helper method and then pass the datasets in with each test.

Testing: Ye JUnit4 Way (02:48)

In the JUnit4 way, instead of using naming conventions to figure out what method does what, you can now use these annotations to:

1. Declare which method is supposed to fire off before every test.
2. See which method is a test.
3. See which method is supposed to fire off after all of your tests.

It also allows tests to be marked as ignored. You can have these tests specifically marked as ignored, rather than commenting them out. It helps you to see that the test is ignored because you now know that it is simply not being checked, rather than being absent. For example, tests that have run look like a little “yellow ball” (instead of a green ball), or a little “red ball” for failing.

The runner itself has subclasses which allow all sorts of fancy things. For instance, not having to use a specific subclass of a test class means that you can use different runners in each one if you need to. I am going to try to show an example of parameterized testing (one of the more useful things that I have seen). If you want, you can use your own test runner subclasses — I have not gone that far down the rabbit hole on that one yet (it seems like a pretty deep rabbit hole), but it is something that you can theoretically do.

JUnit4 allows you to have tests skipped based on an assumption. The idea of an assumption is that it can automatically skip your test based on a certain precondition. For example, when running a bunch of tests that should generally be pointed at a staging server, I want to make sure that my base URL actually is the staging server URL. I can then set up an assumption where if my base URL is the production server URL, I can now ignore all of these tests. That is really helpful because the data are different in staging production. You do not want to change data on production (that is a horrible idea), and you want to just make sure that it really works.

It is a great idea in theory. The problem is that this is not presently working in Espresso 2.1. I assume it relies on the subclass of JUnit4’s runner to make sure that things are properly marked as skipped when an assumption fails. Instead, right now it just throws an assumption violated exception and fails your test. The reason for that is that every runner that subclasses the JUnit4 runner has to implement this itself. Initially the Android team had not done that, so my colleague Bill Best (our director of Android engineering) said, “Great idea! Android JUnit4 should skip tests with failing assumptions! This has been fixed in Espresso 2.1”, and then a couple weeks later “…this does not seem to be fixed in Espresso 2.1 — sorry about that, I reopened the issue and will check it again.” So, they are still looking into exactly what happened. Hopefully that will be working soon.

JUnit4 Benefits (05:59)

On a happier note, let’s talk about some of the benefits. The biggest benefit to me is that tests do not reference an activity unless they actually need to. One of the big advantages over the old method is that you would always need to spin up an activity, which would often take a lot of time. Now there is an activity test rule that allows you to set that up really easily: if you don’t need an activity, you don’t have to spin one up; if you do need an activity, it takes two seconds to set it up. It’s really the best of both worlds.

Another benefit is the possibility of naming things in a way that makes sense to you. Removing the tie to these naming conventions allows you to set up how you want to name your tests in a way that makes it clear to YOU. Instead of just: public static void setUp and public static void tearDown, you can say public static void before all tests. If someone is not familiar with that naming convention coming into your code base, you can have something that is annotated with @BeforeClass, and then you can just name it before all tests. Hence, you don’t have to follow those existing naming conventions.

Another benefit is, again, the parameterized testing, which allows you to run the same test many times with different parameters.

JUnit4 Drawbacks (07:29)

There is an obnoxious workaround. Hopefully, even by the time this video gets posted, this will no longer be an issue: FlakyTest does not work on JUnit4 tests, which is a really useful annotation that can be added to test. It allows you to say, “I want this to fail X many times before it officially fails, for example when you’re dealing with UI tests, or when working with a server that perhaps you do not control.”

I had an issue with a particular Samsung device. I would tell it to tap a button, and about 50% of the time, it will tap the button, but only the accessibility label will show up. Tthe button will not actually think that it has been tapped. To write a bunch of try-catch code to deal with this is really infuriating. In an ideal world, a FlakyTest would be completely unnecessary. This not being an ideal world, it is super helpful, and unfortunately it’s tied directly to the JUnit3 runner that ActivityInstrumentionTestCase2 has, so it does not work with JUnit4 tests yet.

Another annoying thing is that you have to static import more stuff. Because everything no longer inherits up the tree from JUnit Assert, you have to import. Sorry. It’s not even static import, it’s import static. If you write static import, it will yell at you and make you very sad. Just remember that you will have to bring in a lot of stuff statically that used to be basically handed to you by the inheritance chain of the testing cases.

JUnit 4 - Demo (09:57)

[Ellen demonstrates how to write and run tests using JUnit in her sample project.]

Espresso: What is? (23:50)

Espresso is the lovely open-source UI testing framework for Android created by Google. It is really a great way to integrate tests in your application. Traditional unit testing is all about answering, “Does this thing work in isolation?”, and UI testing is about making sure that everything is integrated. Your stuff can work perfectly well in isolation, but have weird finicky bugs when you try to integrate all of it without having some kind of UI testing.

I think there is still a lot of value to testing your UI in a somewhat isolated fashion, but by necessity, there are at least some pieces of the app that you are going to test to see how they are working together. These can be, for example, fragments, activities, and views.

Espresso: How work? (24:51)

If you have ever used Hamcrest matchers with Mockito, this will look familiar. They dig through the view hierarchy, and allow you to specify that you want something that matches a particular view rather than having to directly reference that view. This is similar to FindViewWithID(), except from a standpoint of “dig through this view hierarchy and figure out which view matches this ID, and then return me a matcher that has that”.

It only considers the portions of view immediately visible to the user. It makes sense when you think about the experience of trying to use the app: “If the Espresso framework can’t see it, your user probably can’t either.”

Espresso can enter text and tap things and scroll and swipe out of the box. The framework from Google directly has a bunch of the most common actions that you’ll want to move through your UI already set up for you (something where they’ve got a fair amount of stuff), but if there is something that is not covered by Google, you can actually create your own actions.

You can have virtually limitless complexity with this, which is both a good and a bad thing. You can compose actions from other custom actions. For instance, instead of having this big, long chain of Hamcrest matchers and view actions, you replace text and view with placeholder. That is readable, makes sense, and gives a story about what you are testing.

I want to give a few more general tips on writing UI tests, but I think they are a lot more useful after you’ve seen some of these in action.

Espresso: UI Testing Demo (26:35)

[Using the Wino sample app, Ellen demonstrates writing and running UI tests with Espresso.]

UI Testing Best Practices (35:27)

UI testing best practices are important to avoid billion of hours testing your application. Don’t navigate through your entire UI every single time. When I first started writing tests about a year ago with Espresso, my tests took forever. Every time they ran, they would have the test framework login, then navigate to the part that you actually want to test, and then actually test the thing that you were trying to test. If you have a complex flow, this wastes so much time on every single test.

A better way to do this is to use class and instance Before and After methods. You can use one of the static methods annotated with before class to run a piece of code before all the tests in a given file (i.e. login under the hood for the user). On an After class, you can nuke all the state that you set up. You can also do the same with the instance methods, so it runs before and after every single test. Make sure that you are unwinding whatever state that you have setup in your UI test so that you are always starting at the same point.

Create a test class for each piece of your UI. Isolating all the tests for a given piece of your UI in a given class gives you, first of all, the ability to just run that particular class of tests. That is, you’re isolating what you’re testing. Isolating all the tests also gives you the ability to create a test activity to only test stuff in a particular fragment.

If there are things that you can test without the UI, test them without the UI. Spinning up the UI takes more time, and if you have pieces of common code that are used across different bits of UI (especially validation stuff, i.e. e-mail/phone number validation), you have to run these validations on them in order for you as a business to be able to use them. Use the UI test to make sure that when somebody puts in an invalid phone number, the proper error is showing up. However, you don’t want to test 400 different invalid phone numbers using UI testing. Instead, you want to test it with a parameterized test that is running off the UI.

More UI Testing Tips (38:56)

Making a mock flavor and testing on it makes it really easy to mock objects (like a networking stack), and makes sure that they are not in your final app. It makes sure that your UI is behaving in a consistent fashion, and that you are testing against a known set of data. “What should that actually look like when it is displayed to the user?” That’s what you’re actually testing, rather than, “Is everything unchanged on the server?”

I have to give a shameless plug alert for this. This is a really awesome thing that our team at vokal developed in-house and open-sourced. Mocktrofit is an open-source library started by Bill Best. We have a similar mock testing library for iOS: VOKMockUrlProtocol. What it does is, you get a mock data file, and it uses the name of the file to determine which API call it’s trying to make. If you are interested in mocking out your networking stack, particularly if you’re already using retrofit, I highly recommend taking a look at it.

Compare string resources to what is on screen. Some people like to write UI tests by passing raw strings into the functions that they are checking. For instance, composing a string (e.g. putting the PM in the 4:30 (PM) in parentheses) is harder to do without passing in a raw string. But if you want to make sure that a given error message is showing up in the correct place or a given thing is displaying to the user, use those strings to your advantage. They save you so much time. It also allows you to continue to have all of your stuff in the same place so that if you or your copywriter comes through, you don’t have to rewrite your test.

Again, remember when you need to scroll. Espresso searches through what is currently visible on screen. If you are using a smaller simulator, or you are using a smaller device to do this, you have to be able to scroll to whatever is showing or your tests are going to fail.

The thing that I like about UI testing in the context of the application is that you can get to a piece of UI that is buried without having to tap into it yourself. For instance, when working on registration pages, you don’t want to type all the stuff out every single time. Basically, you can say: “I want the test to not fail for the next million seconds so that I can sort of poke around in this portion of the application where I’m working”; and, “given all of this state that I have built up in the test, is this doing what I want it to?” Not only have you saved yourself time in having to enter all of this information, you have already written the test that gets you to it. It’s really pretty cool to be able to do that.

Q&A (44:01)

Q: In your experience with Espresso, I have personally heard a lot of pain over using different device types with your Espresso scripts, for everything from OEM to screen-size differentiation. Can you talk a little bit about the fragmentation problem and how you account for it?

Ellen: I think the biggest thing is when I was talking about the need for FlakyTests. That is a huge piece of what that is necessary for. That, and just being able to scroll. I think it is also helpful to be able to know if there are features where it’s supposed to gracefully degrade. It would be really helpful to know whether that was being accomplished in the way that you want it to.

The app that I did most of this work on for a client was simpler and was not something where it was, “you can’t use this particular feature and there’s no support library fallback for it”. Almost everything that I was using had a support library fallback, so I haven’t had as much experience with dealing with things that may or may not necessarily be there. But, I can definitely say that that is going to be a pain in the butt.

Q: If you are unit testing an activity with Espresso, and that activity expects some type of input, like an intent coming in, how are you currently handling that?

Ellen: It depends on what I want to test. In Espresso 2.1, in addition to the activity rule, there is an intents test rule that basically allows you to mock intent data. For example, someone is opening up your application from another application, where they are sending you a shared text intent. You are able to mock out that intent so that you can start your application using that, rather than setting everything up to use as much of the UI as possible. That’s something where I’ve only played with that a little bit, but from what I’ve seen, it is really cool.

Q: Is it possible to take screenshots with Espresso 2, or do you have to get an add-on in order to make it work, in case something fails?

Ellen: That’s not something that I have done. My recollection is that that is possible, but don’t quote me on that. It is possible to have UI tests — I know that Apple just introduced a bunch of stuff within their new testing stuff in Xcode 7, where it will actually automatically take all these UI screenshots whenever something fails. You’re actually able to use comparison tools to be able to do that. That is definitely something that I can look into and I can tweet something about it, but otherwise I don’t know.

Q: Is it possible to check the view’s positions in a sense of, let’s say, this is ADP, or aligning correctly on the views?

Ellen: The view matchers don’t actually return the views. I believe that there’s way to go and grab the views from the view matcher, but it’s a little bit complicated. It’s something where you have positioning, and exact pixel-perfect design is not really what this is designed for. It’s more designed for making sure that your user can work through your application and actually get stuff done. I think that by using FindViewWith and by using the stuff that actually checks to make sure a view is visible, if you’re on a smaller phone, all the various things that you actually need to be visible to get everything done there, or can at least be scrolled to. But in terms of checking the exact pixel-perfect positioning of it, I don’t know that that’s something that’s possible without a lot of voodoo.

Q: About remembering when to scroll, what is the best practice for that? Do you expect that I might need to scroll to find a element? How would you handle that part?

Ellen: Generally what I have been doing is, if I have something that is of any length where I think the keyboard is going to wind up in front of it, I will just scroll, particularly because I know that on smaller stuff, it’s just going to need to. If it is something where maybe there is just a single edit text that I need to change, then I usually don’t bother with scrolling. I think a good rule of thumb is anything that you feel a need to shove in a scroll view, you might as well just say scroll to. If you think that there is a good enough reason to put it in a scroll view because you think a user may need to scroll it, the robot that is testing your application will probably also need to scroll it, if it is running on the same kind of device.

The scroll through basically says to keep going until this view is at least 90% visible, and it’ll scroll for a while before it’s like, ”I don’t know where it went!”. Some can be affected by how different manufacturers mess around with Android UI, but there are places where you can, if you really want to, spend the time and effort to say, “on this particular thing, either skip this test or basically don’t deal with this in this particular fashion”. Once they get assume stuff working, that’ll be a lot easier.

Q: How do you like this compared to the Xcode 7 UI automation testing that is going to be released by Apple?

Ellen: Yeah, I’ve played with it a little tiny bit. I work with a library from Square called KIF most of the time. One of the things that is a little bit frustrating with the UI automation is that you don’t get any kind of insight into what is going on under the hood of the application, because the application is basically a completely separate testable. You can’t actually access anything within it; whereas KIF is going through a bunch of private API and when I update, “can I confirm that it did what it needed to under the hood?” I think that is a huge advantage of the Espresso approach because you can make sure that not only are things doing what they need to in terms of displaying stuff, but every single piece of data that you need to get saved is actually getting saved properly. Or, you can also have your tests yell at you, like I just did.