Continuous Delivery for iOS & Mac Apps

Build your code on every commit and distribute the app directly to your users. Ship a bugfix for your app without waiting for the next Apple review. The tools for continuous integration and delivery are improving as fast as the mobile market itself. Combined with platforms to monitor the quality and usage of your app, a powerful Mobile DevOps workflow is evolving. This talk gives an overview of the available free and commercial tools and shows how to put them together in 3 live demos.

---

Introduction (0:00)

I’m talking about continuous delivery and I wanted to start with the Xcode icon.

Have you ever noticed how the Xcode icon has the hammer on it? The hammer is one of the oldest tools used to impact objects. It’s a symbol of mankind, but if you want to start a mining operation, you wouldn’t really use a hammer. You would use something much bigger.

Maybe something like a LEGO toy. This is a bucket wheel excavator, and it has that wheel of buckets in the front to continuously mine material and then transport it. It enters through those belts and goes onto the dump truck.

That process is very similar to what I’m talking about today. I’m transporting code over a pipeline to my users’ devices to enable them to continuously get value from my development team. This is continuous delivery.

Continuous delivery is about continuously building, testing and releasing your app, with the goal of reducing costs, time, and the risk of the changes that come with your code base. In a world with billions of devices and an endless number of good device configurations, like device size, locale, or device type, you cannot afford to not have a continuous delivery process anymore.

Apple just released iOS 10, but even Google and Microsoft now release their operating systems in yearly or even more shorter cycles. There is a stress on developers to test all their apps.

Continuous delivery process (2:09)

This is how I envisioned the continuous delivery process, but really, the components are not fixed.

Here I show six bubbles around my machine in the middle where I create code. Then I use source control to manage my code. I have a build pipeline. I have a test pipeline. I distribute my apps. Then I want to monitor the results and get feedback from users. We all get feedback in the App Store every day.

Fundamentally, everybody should use source control. Without source control you can’t have a continuous delivery system because every code commit should create a build and every build should go to your users.

The other two fundamentals are monitoring and feedback. If you don’t have monitoring, you cannot figure out whether your build is good or not good. Without feedback, you don’t know what the users think about your app.

Automated build process (2:48)

The second thing I want to talk about is the automated build process. As I’ve mentioned, every code commit should create a build. Whenever you push something to your repository, a build should be triggered on your build server that is delivered to your customers.

Now, there are various options available. You can pick between a self-hosted option. Jenkins is one of the most popular free alternatives. Team Foundation Server is a Microsoft product that you can buy. And Xcode Server comes with your Mac and Xcode installation. The key for all of those is that you need a Mac to build your app. Regardless of whether you own a Windows environment or a Linux environment, to build an iOS or Mac app, you need a Mac in your environment.

On the other side, there are cloud options available. For example, Bitrise, Buddybuild, CircleCI or Travis CI, provide the Mac environment for you, so you don’t have to take care of installing a Mac, keeping it up to date or installing an agent.

Now, there are hybrid solutions available, for example for both Jenkins and Team Foundation Server, or the equivalent cloud product called Visual Studio Team Services. You can use a remote Mac on MacinCloud or MacStadium to run the build itself and then have a cloud product that runs your build’s pipeline.

At the end of the day, those cloud solutions give you everything in one step. Specifically, Buddybuild does even more, as they manage your iTunes connect account for you and your Apple developer IDs. They actually auto-provision your builds whenever you want to add a new device to the provision profile.

Those are all very sophisticated solutions that you run as part of your continuous delivery process, but the first step of automation really happens on your Mac.

Automation (4:44)

For that, there is a tool available called fastlane created by Felix Krause, open source and GitHub. It was acquired by Twitter last year. It’s still open source, luckily. It provides a set of tools that have all nice names describing what they do.

Fastlane is a command line interface running on your Mac that connects and automates all interactions with the iTunes connect and the Apple developer portal. You can download your provisioning profiles and their certificates. You can upload screenshots. You can actually automate the whole process of creating screenshots.

You can also ship builds to TestFlight, to HockeyApp or to any other pipeline you want to. That’s really super cool because after you’re done with coding in Xcode, you can fully automate your build process on your Mac itself, and then you go into the next stage of automating your build process on Jenkins or Bitrise.

I want to give you a quick demo of fastlane.

Demo 1 (5:47)

I created a little demo project that is a React Native app. For those of you that don’t know React Native, it’s a product created by Facebook that allows you to use a JavaScript layer on top of native frameworks. By overriding your app in JavaScript the end product is still a Native app and not a mixed hybrid product, like Cordova.

I can actually run this app directly here from my editor in the simulator. It just shows a little Hello World example. The editor here is Visual Studio Code. That’s a Microsoft product that comes with React Native extensions. You can open your React Native product in there, and then use it like a compiler. Like an IDE.

Here’s my little Hello World app. It really does nothing else and just shows Hello World. It runs in a simulator and what you can do in React Native is refresh your code. You can type in Command + R and then it refreshes the JavaScript in the simulator.

Now I want to ship that app to my iPhone and I want to use fastlane. I’ve already prepared the demo project. This is calling fastlane in it, which creates my Fastfile. Then I also want to use a tool called Match. I then call match, which creates my Matchfile. Match connects to my Apple developer ID and automatically creates certificates and provisioning profiles for me. I don’t have to go to the provisioning portal and create a profile, download it to my Mac,…then Lucas wants to use the same provisioning profile so I have to send him the certificate which means I have to export it from the keychain, all that stuff. Match does all of that for me.

Now, one of the cool features of Match which you don’t need often in real life is that you can also nuke everything. You can delete all your certificates and provisioning profiles from your developer ID, and then recreate them. That’s cool for my demo because I can show you how that works without faking it.

What I’m doing now is I’m running match nuke distribution, and that deletes all my certificates, my distribution certificate and all my provisioning profiles. It asks me whether I want to delete everything, I say yes, and now all my certificates in the developer portal and all my provisioning profiles are gone. The next thing I want to do is create a new Ad Hoc profile to distribute my app to my own phone. I do that.

The way Match works is that they have a private git repository where my certificate and my profiles are stored. I can use a passkey to secure them so even if somebody gets access to my git repository, they cannot steal my certificate. The other cool thing is that if I set up a new Mac, I can just run Match again and I can run it in read-only mode, so even if I don’t have access to the developer account or some of my team members don’t have access to the developer account, they can get access to the same certificate and provisioning profile provided they have access to the git repository and the passkey.

Now it has created the certificate for me, so the next thing I want to do is create an IPA file that I can upload to HockeyApp. To do that I have created that Fastfile. Fastlane has done that for me.

The way fastlane works is that it creates “lanes” that I use to distribute my app. I have two lanes here, a beta lane for beta distribution and an appstore lane that I’m currently not using. In the beta lane, I do four things. I load my private key, which in my case is the HockeyApp API token, but it can be your Crashlytics token, or whatever token you have. That’s not part of the Fastfile, but it’s coming from an environment variable that’s local to my Mac. I tell it that it should use the Match adhoc profile and then with Gem I build my app and that runs Xcode on the command line to build my full React Native app. I give it the pass to the Xcode project here because in a React Native project that is in a sub folder.

Then last but not least, I upload to Hockey. You can upload to any kind of service. I’ve done all that and the next thing I need to do is call fastlane beta to compile my React Native app with the Xcode project. It downloads my profile through Match and then it uploads my API file to Hockey.

Here’s my iPhone, so I can show you the installation process via the fastlane and the background is still running. This is HockeyApp and I can simply tap install, confirm the installation and if I go back to the home screen, you can see the app is now installing. The installation works and once the installation is done, I can launch the app.

The last thing I want to show is another Microsoft product called Visual Studio Team Services. I want to show an example of how Microsoft operates these days. This is our own build solution and it uses fastlane in the background to allow me to upload apps to the App Store. I have two build environments. Beta which takes my build and uploads it to HockeyApp and Store which has an upload task to the App Store, which in the background uses fastlane to upload my app to the store.

Now the cool thing here is that if I don’t want to allow every team member to upload an app to the App Store - I want to restrict it to some approvers in the company - I can click on this approval icon and it should show a dialog that lets me enter the name of a team member. Then this team member gets an approval email, so whenever I ship an app to the beta channel, the approver gets an email and he can decide whether he wants to approve that same app for the App Store, or whether he wants to decline and wait for the next situation.

The point I want to make is that the combination of different tools, from open source tools to commercial tools, to Microsoft, and other very powerful release pipelines, lets me map my company processes to the continuous delivery process.

Automated device testing (12:43)

Then the next step in my continuous delivery pipeline is automated device testing. We used to see that on Androids and laugh about it, saying, “Oh my god, those people. Those developers they have a crazy life where they have to test on so many device configurations and get so much feedback and crash reports from multiple configurations.”

Now, it’s not as horrible on iOS today, but it is getting a lot worst than it was in the early days. We now have three different sizes of iPhones, we still have the iPod around, we have iPads in different sizes and then in addition to the form factors, we also have different features for different devices classes. For instance, 3D touch on the iPhone 6S, the pen on the iPad Pro, and then of course, all the multitasking features that enable me to show my app in more configurations than ever before.

A solution for all of these is cloud testing. Cloud testing allows me to test my app on thousands of devices at the same time in the cloud in an automated way. I also have access to all the test results, so I can see which test has failed. Most importantly though, the tests all run in parallel, so I don’t have to have a QA team that taps through hundreds of devices and finds bugs.

I have a service that does it for me. An additional feature that some cloud testing services offer is exploratory testing. That allows me to run my app in the browser on a real device that is somewhere connected to a server. There are multiple cloud testing services available. Amazon has what’s called Device Farm, there’s a service called Perfecto Mobile, and Microsoft recently acquired Xamarin, which is Xamarin Test Cloud. It is a prefix so Xamarin and works for all Native apps.

Demo 2 (14:43)

For my second demo, I want to show you is how cloud testing looks. As an example, we’ve taken the conference app from Xamarin Evolve.

I can click on my iOS app and see all the test results in a nice overview chart. Then I notice something failed on April 15. All of a sudden 21 tests failed. I can click on the test results to see what has happened. On the left-side I see all my tests and I can click on a test to expand the test steps.

On the left-side you can see all the test steps and on the right-side you can see all the devices that the test has been running on. You can see this test has been running on 20 or so devices and there is one failed device. Now that is actually not an app failure, it’s a test failure, because storage apparently was full on that device.

Regardless, you can see how automated testing helps you to find problems on a single device out of multiple devices easily. Usually all those services come with additional features. For example, on this iPad you can see how memory usage and CPU usage where distributed during the test. You can also get a device log and a full size screenshot. If you’re just interested in a quick screenshot for some apps to deploy or some pull material, you can easily grab that from Test Cloud.

The other service that I mentioned is Perfecto Mobile. This is Perfecto Mobile. I have launched an iPhone 6 that is running iOS 8.4 to figure out if my app is running on that device. I can launch multiple iOS devices to Test Cloud. They offer me multiple devices that I can launch. And now what do you do for presentations like that? You can lock or reserve a device for certain times in the week.

The downside of that, and also of Test Cloud, is if you’re using Ad Hoc provisioning on iOS, which means you have to add UDIDs to provisioning profile, you have to add all the UDIDs of those test devices to your provisioning profile. Now, I have my device and I want to install my app. I can do that manually. I’ve already uploaded the IPA file. I click through to my media library and there is my awesomeproject.ipa. It’s installing the app automatically on that device.

That’s a real device somewhere in some data center connected to a server. It’s shows me screenshots from that device. Now that the app is installed, I can see my icon and I can use my mouse cursor to interact with that device as if it were a touch device. If I want to swipe to a different home screen, I can make a swipe gesture and it swipes over to the other screens.

You can also see that this device actually is connected to Verizon on LTE, so if I want to do some carrier-specific testing, I can do so using those kinds of devices. Then I can launch my app through a click or a tap, a simulated tap, and you can see my app. In addition to that, fastlane in the mean time has uploaded my new build to HockeyApp.

Also, I can completely wipe the device. If I want to have a clean start I can simply click Cleanup and it will completely remove all the apps that I’ve installed on the device. I get a fresh start on the same device. That’s really powerful if you want to try out something quickly and then move over to the automated testing to run your test automatically. You can see the automation tab allows you to define test cases based on accessibility and other test frameworks.

Continuous delivery pipeline (18:51)

The next step in my continuous delivery pipeline is the distribution to the actual device. There are multiple ways to do that on iOS.

Today, most of you probably know how Ad Hoc distribution works. The only change that we have seen in years is that the 100 UDID limit is now per device type. So you get a hundred iPhones, a hundred iPads, a hundred iPods, a hundred Watches and a hundred TVs, but you’re still limited to a subset of your users.

But the advantage is that you can use all the App Store features, like push notification and app purchases. You can distribute to internal and external testers and the app is directly installable and runnable on your device. You don’t have to trust the developer.

The opposite is in-house distribution, or some call it enterprise distribution where you have no UDID limits. In theory, you can distribute such an app to everybody, but you cannot use any store features like in-app purchases because it’s fully intended for in-house distribution to the employers of your app.

Since iOS 9, Apple shows an untrusted dialog that doesn’t give you any kind of indication regarding how to trust the developers. You have to find that setting yourself in the settings app of your phone. Apple intentionally did that because they don’t want people to use that for piracy. They make the process intentionally hard to not make it too easy for people to use that kind of setting to steal your apps.

Then on the other side, using iTunes Connect, we have TestFlight with 2,000 testers. This uses external testers, so you have to go through a review, which is the biggest downside of TestFlight. Although the review times have been going down quite a lot. Also, you can use all store features, it’s not limited to your team internally.

You can also use the App Store for some kinds of testing. I think Facebook is a good example. They roll out apps every two weeks. They put out a new version every two weeks, but also use internal mechanisms in the app to enable features and have features roll out while the app is available in the store.

We have no TestFlight on the Mac. There is Ad Hoc distribution if you want to try out sent box features. Since yesterday, Apple announced that the developer ID now also allows MapKit, CloudKit, and all the iCloud features. With the developer ID, you now have the best way to test your Mac apps.

There is another way to distribute updates to your apps, if you are doing a React Native or Cordova App, which is called CodePush. It’s an open source project from Microsoft. It allows you to update your apps without resubmitting them to the App Store. When the app gets into the foreground, it checks the update server for whether a new JavaScript package is available for your app. Then it downloads that JavaScript package and in the next app launch, you can see your changes live in the app without ever going through any Apple review.

Now to stay within the limits of Apple’s developer guidelines, you’re not allowed to really change the scope or focus of your app, but you can easily ship bug fixes through that channel. There is a similar tool called Rollout.io, which is a startup from Israel, which does that for Objective C apps using the JavaScriptCore API.

Demo 3 (22:23)

I want to give you a quick demo of that too. I’m back in my React Native project and I want to do a little change to the app. Instead of saying Hello AltConf, I want to say Hello WWDC.

To do that, I have already integrated CodePush into my app, I change the string. So I say Hello WWDC. Then I want to push my package to CodePush. For that I have to add an NPM package to my React Native app, and HockeyApp helps me with that.

If I go back to HockeyApp, we detect that I have uploaded a React Native app and I can click on the CodePush link to get direct instructions on how to use CodePush as my HockeyApp app. I can use my HockeyApp app ID to instrument CodePush. The same app ID that you used for HockeyApp updates, you can use for CodePush updates.

Now it shows me how to install the CodePush CI and the CodePush NPM package. I’ve already done that, so all I really need is the push step at the end that pushes my JavaScript package over to the CodePush server. Then on my device I restart the app to get the new package. I do that in the command line.

It’s now building my React Native JavaScript bundle, which is a single JavaScript bundle with all the assets and JavaScript file. It uploads them to the CodePush server and now on my device, in the background, the device network connection still works, has checked against the CodePush server, and downloaded the new JavaScript bundle. Now we don’t want to interfere with the user using the app while it’s running, so we wait until the next code start and then install the new JavaScript bundle. It shows “Hello WWDC.”

Without going through HockeyApp and without going through my build server, I simply have uploaded a new CodePush package, a new JavaScript package to CodePush, and allowed my users to get a new version of the app. You don’t want to overdo that with your users, but it’s really helpful if you find parts that you want to fix quickly, of if you have little changes or want to enable a configuration option on your app that rolls out a new feature to users.

Conclusion (25:11)

The last thing I want to talk about is DevOps, which is the combination of developers and operations.

We have talked about continuous delivery as an automated process, but DevOps is not only about continuous delivery. It’s about the union of people, process and products to enable the continuous delivery of value to our end users. If the value only reaches QA testers or my internal team, we have still failed. We have to deliver the value to our end customers.

The tools that I have shown you enable you to start that process, but it’s really about you enabling that with your team and enabling your team to continuously commit code to the repository and ship those changes to the users with the confidence that it won’t break the user’s apps.

Q & A (26:01)

Q: Are the users of CodePush covered in the App Store guidelines or are we just playing the game of don’t annoy Apple too much with the changes that we make? Thomas: It is covered as long as you don’t change the focus of or the description of your app. If you’re just fixing it back, it’s fully covered by Apple. As long as you’re not going out of bounds and shipping something that is clearly not what you have described in the app description, you’re fine. Apple has a review process and a committee that decides whether you are good or not good. But so far, we haven’t heard of any complaints from Apple about this and there are over a thousand apps using it.

Q: I’m curious about the cloud, with all the different devices you have. Are they running builds on each device or are they actually running automation scripts as well? Because if they are running automation scripts, how do you tell the cloud which scripts to run and how does it know which scripts to run? Thomas: With Test Lab, for example, you upload your own scripts. They’re using the UI automation slash accessibility features to record those scripts. There’s actually a Mac app that you can install in your Mac to record the steps. You install the Mac app, you run a debug build on your device and they record what you’re tapping on in a .NET-like language. Then you upload those steps to Test Cloud and they run those same tests on the cloud on each device, or a subset of the device, like the top 10 devices or top 20 devices.

Q: When you test, what’s a good size for a test group? How many users do you need to give your specific feedback? Thomas: If you’re working with iOS you’re limited to 2,000 testers in TestFlight or over a hundred plus testers to Ad Hoc distribution. At Microsoft, all the Microsoft apps for example, they’re tested with all the employees. Your test population can never be big enough to find crash reports and usage data. At the same time though, you want smaller test audiences, like in a ring or whatever you call that, to get more detailed feedback. Usually if you have 50,000 testers, the feedback is no longer as qualified as you would like it, and it’s more about usage data and crash reports.