Building Postmates' Realtime Tracking on Android

Have you ever wondered just how your favorite delivery or ride sharing app manages to keep you updated of your progress throughout the duration of your task? In this talk, we’ll cover Postmates’ approach to obtaining user location data and sending back real-time updates at every step of the way including the challenges of supporting various scenarios where a device is not connected to a power source throughout the duration of a job.

---

Intro

My name is Mike and I’m here with Torrey and we are here to speak to you a little bit about real time tracking with PostMates, and how we handle location across our set of apps.

What is PostMates?

We’re an on-demand delivery company, we operate in about 45 markets, that’s over 200 cities across the United States. Our goal is to get you anything delivered from anywhere, as long as it’s legal, usually in under 30 minutes, depending on the place.

Why is real time tracking important to PostMates?

We want everything to be as fast as possible, in the span of an hour or 30 minutes, there are about four to five different people that have to coordinate a delivery. That’s the customer, the courier, a support person, and a merchant, trying to get you your burrito within 30 minutes to an hour. No one likes cold food, so in order to that, we need to find someone that’s as close to the pickup location as possible and pick that up. As well, whenever you want that burrito, you want to know where it is every step of the way, it’s important to our customers to know where that burrito is, Today we launched alcohol, so people want to know where their beer is and you want to know where that is every step of the way.

Like I mentioned, we want to make sure that we match your deliveries to someone that’s in the vicinity of where you’re placing your order from. That we can get it to you as quickly as possible.

Finding the Buyer

One of the first things that we need in order to make sure this whole service works correctly is a buyer. We need a delivery, we need someone that signals the intent that you’re going to purchase something to be delivered on PostMates. In our app on the left-hand side there are a bunch of different places that a buyer can order goods from; on the right-hand side is the flow that you have when you’re getting ready to check out.

From an Android-y technical perspective, what do you need in order to render results like this?

The first thing you really need to know is where your buyer is. You don’t want to show a buyer results that are too far away from them, because they want it delivered to their house. In order to satisfy this use-case, we need to get the buyer’s last location. In the earlier days of Android, the user location provider is one of the mechanisms that you can use to get the last location. In order to use that API you used to have to go through the process of connecting to the Google API clan, connecting to the clan, and when it’s connected, then you can have permission to ask GooglePlay services for the location.

On our buyer app within PostMates, we actually use a nice library called Android React Provider. We used RX for some of this, I won’t go into too much detail about it, but RX is actually quite nice on Android development, it makes things quite elegant.

public Observable<Location> getRxLatestLocation() {
		LocationRequest request = LocationRequest.create()
			.setPriority(LocationRequest.PRIORITY_HIGH_ACCURACY)
			.setNumUpdates(NUM_UPDATES)
			.setInterval(UPDATE_INTERVAL_MS);

		ReactiveLocationProvider locationProvider = getReactiveLocationProvider();
	
		return locationProvider.getLastKnownLocation()
			.switchIfEmpty(locationProvider.getUpdatedLocation(request));
	}

In our code, we have a get last location function that creates a location request, which is the standard request that goes out to the GooglePlay services library. We also have their provider that says “get last known location.”

The return statement of this method is quite interesting, because in RX you have observables, which emit data back that you can subscribe to. This is used in the buyer app to subscribe to and get the last location, we have nice little functions which you have empty, which is really elegant. What it does is, flat location provider, get the last known location, sometimes on Android that comes back as empty, like null. This happens in certain situations you have let’s say, the device turned on just a few seconds ago, it doesn’t really have a fix on the last known location.

In these cases the switch of empty will say, “If last known location emits something that’s not there, because the device doesn’t have a fix yet, we’ll switch it out for the observable location provider, get updated location requests”, and that will actually signal to GooglePlay services that we want to get updates with these sort of parameters to give us a last-minute location. It’s really nice and really elegant. This is how we get the last known location, and then we can use that in order to show everything really nicely.

The Courier

Let’s say the buyer is getting a Chipotle burrito, and they created their delivery. Now that we have the order created, we need to be able to assign it to a courier; we need to have it assigned to someone who’s going to deliver the product. The next step in order to do that is we need a location from the courier. We need to be able to know where they are in order to determine on the platform if they’re the most appropriate courier to do the job.

PostMates supports a wide range of types of couriers. For instance, traditionally you have couriers in cars and trucks that can go out and do deliveries, you also have couriers on bicycles that can do deliveries, or on mopeds, you can have couriers that can go and walk. If you’re down in the financial district, you’ll have someone to actually go walk to the store, walk to you, and have your goods delivered.

Because of this, you’re not guaranteed to necessarily be connected to the battery at every step of the way. If your app is not connected to battery and it’s not in foreground, there’s different options you can do to ensure that you’re getting locations in the background. One of the most straightforward is just use a background service.

It’s retty straightforward. Here’s some of the code that you can do.

@Override
	public int onStartCommand(Intent intent, int flags, int startId) {
		super.onStartCommand(intent, flags, startId);
		if (mGoogleApiClient == null) {
			mGoogleApiClient = setUpGoogleClient();
		}
		if (!mGoogleApiClient.isConnected() && !mGoogleApiClient.isConnecting()) {
			mGoogleApiClient.connect();
		}

		// We want this service to continue running until it is explicitly
		// stopped, so return sticky.
		return START_STICKY;
	}

	@Override
	public void onDestroy() {
		if (mGoogleApiClient != null && mGoogleApiClient.isConnected()) {
			mFusedLocationWrapper.removeLocationUpdates(mGoogleApiClient, this);
		}
		super.onDestroy();
	}

Once you have the background services on Start command, you initialize the API client, you connect to it, and then when you’re done, disconnect from the location client. Now in order to make sure that this service stays running in the background, you need to supply this little flag called Start Sticky.

Start Sticky tells the platform that if a service is destroyed because the OS decides that it’s running low on memory or certain parameters, Start Sticky should actually tell it to restart on its own.

Now the keyword here is “should”. In most cases, this will actually work out just fine, Android will behave correctly. Except for on Kit Kat. What we found out in our courier application, in production, we had a bunch of users reporting that they would go online to be ready to accept deliveries, but all of a sudden we weren’t assigning deliveries to them because we didn’t know where they were.

As we drilled into this, we actually found out that Start Sticky, this flag on the service level, is actually broken on Kit Kat. When your service is destroyed in 441 and 442 it actually never starts again. It just dies and it ends there.

Obviously for PostMates this is actually not very good because that means we can’t assign you a job if we don’t know where you are.

How do we get around this?

One way you can do it is, on Kit Kat, in addition to starting your service as a sticky service, just schedule a repeating alarm. Make sure that you do a start service on this, whatever it is requested in location updates, to just get it started again. That’s one approach that we have for this particular version of Kit Kat.

Location Updates From the Courier

With all these locations, how do you get them up to the server? Do you do a network request with every location that comes in?

Maybe that’s a little excessive. At PostMates we decided to use an alarm manager. Alarm manager has been there for quite a while, and it’s a nice simple way to schedule a broadcast receiver to run at regular updates.

We have the alarm manager set where you can set an interval, for us we have somewhere around a minute to actually trigger a broadcast receiver that says get all the updates that have been received by the device and send them up to the server to let them know where the courier has moved their position to.

This actually works great except for on Kit Kat, again. As Android has sort of progressed, there’s been a big issue with battery life. One of the ways that Android and Google are trying to help this, is by modifying the implementation of some of these APIs.

For the alarmmanager.set call, here’s what it looks like.

private void startHeartbeatAlarm() {
		PendingIntent alarmIntent = getAlarmIntent();
		if (VERSION.SDK_INT >= VERSION_CODES.KITKAT) {
			mAlarmManager.setExact(AlarmManager.ELAPSED>REALTIME_WAKEUP,
				SystemClock,elapsedRealTime() + LOCATION_UPDATE_INTERVAL_MS,
				alarmIntent);
		} else {
			mAlarmManager.set(AlarmManager.ELAPSED_REALTIME_WAKEUP,
				SystemClock.elapsedRealtime() + LOCAION_UPDATE_INTERVAL_MS,
				alarmIntent);
		}
	}

In the original implementation of this class, you set the alarm, you can specify the elapsed time and then the intent to execute when it fires. In versions previous to Kit Kat, this actually works well and good. You might have a few hundreds of milliseconds, or even maybe a second before this fires, but it’s a pretty regular event.

When Kit Kat was introduced, Android changed the behavior of this API, without anything really happening. If you didn’t change your application, and you ran it on a Kit Kat device, it would actually change right from under you.

This was a big problem that we found out at PostMates; our couriers that were running the latest and greatest versions of Android also weren’t getting appropriate jobs, because it turns out this set is actually delayed by several minutes. This is a baby step towards a job scheduler that was introduced in five, is for them to start making these alarms a little more efficient, in terms of batching them together, which makes sense.

For PostMates, though, this is kind of a bad thing, because you can imagine, if you try and fire an alarm to send out updates for where the couriers were, and that alarm fires on five minute intervals. If you’re in a car driving down the street, you might be six blocks, seven blocks away, and then all of a sudden, that’s just not a good job for you. It thrashes the algorithm a little bit to try and get you assigned to the correct job.

The code above is on the courier app to help us use more precise timing on Kit Kat and above for the alarm manager.

Doze and App Standby

We had alarm manager and Kit Kat to sort of baby step improving battery life. In the more recent versions of Android, 6.0, we have doze and app standby. Not a lot of people know about doze and app standby, but I would say one of the biggest reasons that Android battery life has improved in the latest and greatest versions was because of these two features.

Where alarm manager changed the implementation of the set method behind the scenes, this actually changes the behavior of your application, regardless of you targeting anything. If you take an application today, you have background services running, you have alarms running, you even have jobs running through the job scheduler, and you run it on a pre-6.0 device, it’ll behave as expected. If you take that same APK, and you run it on Android 6.0, it’s not going to work correctly.

They introduced a bunch of these changes without targeting, which was kind of difficult, because usually, as developers, we like to test things before they go into the wild. This was a very big problem that we ran into, and it took us a while to identify what the actual problem was, because we’d run the same binary on one device and have a different behavior than on another one.

What doze and app standby do is, it introduces a concept of maintenance windows throughout the app. As long as the device is not connected to power, Android will reduce the frequency of your ability to connect to networking. It’s really good for battery, but for PostMates’s use case, it’s not as good. It’s yet another thing that we have to work around to try and make sure that we’re getting the courier updates out to the server correctly.

It was introduced in Android 6.0, in Android 7, the very latest, Nougat, they have even more aggressive scheduling of doze mode, so they’re really trying to shut down the communication in the background. These two take effect while the application is not connected to a power source. For PostMates’s use case, we have a lot of couriers that we know are on their bicycle or on their moped doing deliveries, and a good chunk of the time they’re not connected to power.

What are ways that we, as Android developers, can kind of improve the behavior of ensuring that your services continue to talk to their server in the background, given that doze and app standby have.

Some of the recommendations are:

1. Ask your user to plug the phone in. That’s actually on the GoogleDocs, one of the workarounds is just to ask them to plug it in.
2. Optimize it to function with the maintenance windows. The maintenance windows aren’t clearly defined on how often they are scheduled. It could be on the order of 30 seconds to a minute or even several minutes; it’s not quite sure when these maintenance windows are.
3. If you’re familiar with Google Cloud messaging, or Fire-based cloud messaging, you can use that to send push messages out to the clients, and the push messages are the signal to inform the client and then you sync the data. Now this approach is interesting, but for PostMates’s use case, you can understand if you have x amount of couriers on the fleet, sending them push messages every 30 seconds to ask them to update is not as efficient, it just pushes the requirement on to the backend.

That didn’t work out for us here at PostMates. What did we do?

You use a foreground service. I don’t know how many of you use foreground services, all it really takes in is to have a persistent notification on the status bar telling the user that you’re doing something. In PostMates’ case, we didn’t have this before doze mood came out, now we actually do, and we keep the user informed that we’re updating the location as they’re online and actively looking for deliveries.

This is a nice way that if you believe that your application needs to have data sent up in the background in a critical way, and can’t be delayed by several minutes that doze mode is providing you, foreground services are one way that you can do this.

You have music play in apps that operate in the foreground, that’s how they’re getting around, making sure they can still play the audio and download your favorite audio stream. From the user perspective, it’s a nice way to say “Okay, I’m online, very cool.”

What Can Go Wrong?(16:31)

All that has been leading up to just getting locations up to the server, and now we’ve transitioned to the courier has a job, they have a delivery, they’re going to go drop off the burrito to you at your destination.

Within the on-demand delivery world, there’s no shortage of things that could go wrong. It’s crazy the amount of stuff that goes wrong. As a courier, it can be a little bit stressful. We know this because at PostMates, we have something called Deliv-o-rama, which is where the employees will go out and do deliveries, to do real-time feedback on what’s working and what’s not.

What we’re trying to do on the courier side is making sure that when a courier does request for help, it’s relevant. If I am on my bicycle trying to deliver a burrito down to you, and I have a flat tire, I want to quickly be able to raise the issue, notify the customer that there is an issue, and be on my way before the burrito gets too cold.

Within PostMates we have more intelligent options that are presented to our couriers when they attempt to raise issues.

Resolving Issues Intelligently(18:59)

Being able to contextually provide more relevance, options to the courier, based on giving where we know where they are, how do you do this?

It’s actually not that difficult. All you need to really do is adjust how you’re showing options to the user based on the current location. Here’s some of the functions that we have in our courier app where we show the help location options.

private void showHelpOptions() {
		if (isNear(currentLocation, pickupAddress, NEAR_DISTANCE_METERS)) {
			// Show available options at pickup
		} else if (isNear(currentLocation, dropoffAddress, NEAR_DISTANCE_METERS)) {
			// Show available options in transit
		}
	}
	
	private boolean isNear(Location location, Address address, double distance) {
	
		return distanceBetween(address.lat, address.lng, location.getLatitude(),
			location.getLongitude()) <= distance;
	}

	private float distanceBetween(double startLat, double startLng,
								double endLat, double endLng) {
		float[] results = new float[1];
		Location.distanceBetween(startLat, starLng, endLat, endLng, results);
		return results[0];
	}

We have this function is near, which takes the current location, and the pickup address, and these distance meters. Then, we go and calculate it. Android has this really nice function called Location.distanceBetween, where you give it a lat long, the source lat long, and the destination lat long, and then it’ll give you back an array of results, depending on distances away from them. You can compare that distance with some threshold.

This is actually really the code that we use in order to check if the courier is near the pickup, and what are the options that are appropriate there versus if they’re at the drop-off, and so on and so forth.

Delivery

Now, getting closer to the delivery phase, that’s sort of handled all the cases that the courier has, but let’s say that they’re getting close, they pick up the food, and now, as anyone who buys products off of Amazon knows, you want to be able to, as a buyer, track your delivery as it’s getting closer to you.

Within PostMates, we have the source pin in green, the destination red, and we have the little bike icon which signals the courier’s current location. Very similar to Lyft and Uber, where you’re tracking your driver, we provide the ability for buyers to track their courier on their way so they can be more informed if he got lost or something along the way.

What we want to do is update the courier’s pin as they get closer, and now, on the implementation side, it’s pretty straightforward in terms of what do you need to do this. We’ll use Google Maps as our rendering for the maps, and then adding markers to it. It’s a pretty straightforward solution.

How do you do it?

The user goes into their buyer application, and they need to be able to zoom the map into points such that all the points are visible. In the RKs we have three points, we have the pickup address, we have the drop-off address, and we have the courier’s current location.

The next thing you need to do is you need to be able to pull updates on where the courier is, and then schedule some sort of service in order to inform the application that the courier has moved x meters so that you can update it on your mark.

Then you just need an update mechanism to go through and actually call the map functions to update it. While the app is in the foreground, you just keep updating, it’s pretty straightforward.

Here are some of the code snippets that we have for setting up the initial map.

private void zoomMap() {
		//get bounds of pickup, dropoff and courier locations
		LatLngBounds latLngBounds = getLatLngBounds(delivery);
		
		// move the map to reveal all the map markers
		View mapView = mapFragment.getView();
		int width = mapView.getMeasuredWidth();
		int height = mapView.getMeasuredHeight();
		
		// get cameraUpdate to zoom
		CameraUpdate cameraUpdate = CameraUpdateFactory.newLatLngBounds(
			latLngBounds, width, height, 0);

		map.moveCamera(cameraUpdate); // zoom
	}

	private LatLngBounds getLatLngBounds(Delivery delivery) {
		LatLngBounds.Builder builder = LatLngBounds.builder();
	
		LatLng pickupLatLng = new LatLng(delivery.pickupAddress.lat, delivery.pickupAddress.lng);
		LatLng dropoffLatLng = new LatLng(delivery.dropoffAddress.lat, delivery.dropoffAddress.lng);
		builder.include(pickupLatLng);
		builder.include(dropoffLatLng);

		LatLng courierLatLng = new LatLng(delivery.courier.lastLat, delivery.courier.lastLng);
		builder.include(courierLatLng);
	
		return builder.build();
	}

With Google Maps API, it’s pretty convenient how they have all the helper functions for you.

The first function is we get the bounds of the delivery. The bounds of the delivery consist of the lat long for the pickup, the lat long for the drop off, and the lat long for the courier location. Once you have all of these lat longs, you pass it into the camera update function, and you ask the map to move it to that location, and it has a nice effect of zooming in, making sure that all the pins are visible.

The next thing which is pretty straightforward for your activity side, we’ll call this map activity, on the on-create, initialize the map markers, and then just stopping and starting the sync while the app is in the foreground.

@Override
	public void onSync(Delivery delivery) {
		// Handle other delivery updates

		updateMapMarkers();
	}

	private void updateMapMarkers() {
		map.clear();
		
		LatLng pickupLatLng = new LatLng(pickupAddress.lat, pickupAddress.lng);
		LatLng dropoffLatLng = new LatLng(dropoffAddress.lat, dropoffAddress.lng);

		MarkerOptions pickupMarker = new MarkerOptions().position(pickupLatLng)
			.title(pickupAddress.streetAddress1)
			.icon(BitmapDescriptorFactory.fromResource(R.drawable.ic_green_pin));

		MarkerOptions dropoffMarker = new MarkerOptions().position(dropoffLatLng)
			.title(dropoffAddress.streetAddress1)
			.icon(BitmapDescriptorFactor.fromResource(R.drawable.ic_red_pin));

		map.addMarker(pickupMarker);
		map.addMarker(dropoffMarker);
		map.addMarker(getCourierMarker(courier));
	}

The last part, when you have the syncing of the delivery, you have the update map markers function. The first thing you do is you clear the map. You go back to creating these lat long for the pickup, and the lat long for the drop off, you create the marker for the courier and add them all to the map. This gives you the effect of actually seeing the courier update at every step of the way.

For the syncing part, for something like updating the map, you probably want something on a few seconds of precision, you know, obviously you don’t want your buyer or user waiting there wondering why my courier isn’t moving. A nice value of several seconds for this sync to update the position is quite a nice user experience.

Other Uses for Location Data

So far we covered a lot about how we actually collect location and what we do with it on the client side. There are a lot of other use cases for this location data and what we can do with it afterwards. I just wanted to go over a few examples of what we use it for.

A lot of this leads up to improving upon our pick up leg times. Getting a courier to go to the pick up location to pick up your burrito. One of our goals is to reduce this over and over and over again, so that once we’ve improved it enough, they’re only taking three minutes to get to the pick up location, for example. A lot of this location data helps us achieve that, if we’re able to optimize based on that and a lot of other factors.

One other example, giving us an overview of where couriers are in a city with respect to everything else in the city; and determining capacity, depending on different regions in a city or a larger region of a city and then determining what deliveries we can fulfill.

This is just an example of one of the tools that we use which gives an overview of where couriers are.

Questions

You mentioned trying to use a scheduler, have you tried to use a Fire-based job dispatcher?

No, we haven’t investigated that side, traditionally what we still use GooglePlay, Google Cloud messaging for saving the push messages to, let’s say, job updates and things of that nature to the couriers, but we haven’t used that API yet.

If I am a courier and using all these techniques, how long would I expect my battery to last without connecting to a power source?

It really depends. It’s highly dependent on what kind of location lock you have on your phone. As you can imagine, if you have a satellite lock, that’ll use up more battery. If you’re in New York City, as an example, the population there, and the buildings are very dense, and so your phone will actually use up a lot more battery to get a proper location lock than it would in any other city. It’s highly dependent on a lot of different factors. If you’re in a city that’s very sparse, you can expect to have a lot better battery usage there than in a city like New York City. There’s a ton of different factors around that. Depends on how long you’re online for, as a courier, doing deliveries, and other factors too.

What do you do to handle when users don’t give permission?

Because the PostMates customer experience is so much dependent on having the location of a courier, as well as our matcher is dependent on knowing where a courier is, with respect to pickup locations, drop off locations, we block them from continuing if they don’t give us permission. When you come on to the platform as a courier, you’re agreeing to sharing your location with us so we can better, or more efficiently, dispatch you or offer you deliveries.

Your radio has a particular way to make the waves move with respect to the cell tower. It’s also your broadcast, so you’re sending messages just often enough to wake it up every single time it goes to sleep?

We send up locations in order to ensure that the platform has the most up-to-date information in order for, on the back end, to say that you’re the most appropriate courier for the job. We haven’t gone into too much detail and sort of analysis on the impact on the battery from that side, but I think it’s a little bit of a trade-off of making sure that PostMates as a platform, has enough information that it needs to make intelligent decisions while not trying to destroy the user’s battery.

One thing that kind of relates to the last time, in terms of the location update, for the previous question, a lot of the time when a courier is doing deliveries they usually have something like a map application that’s a navigation running in the background, and so location updates are not as bad as you think because with an Android, you know the latest fuse location API, are acting in a passive mode, and if another app is actively using it for location updates already, it can piggy-back on top of that. We’re doing a lot of work in the near future to try and optimize the cases where we know that it’s a high likelihood that the courier’s not actually using another navigation app to try and improve the battery life more sort of on that side.

A lot of this will change as we continue to make improvements to the application, so you might even see us a few months from now, once again. Giving a completely different presentation.

**What are you guys are doing for failed uploads of any of these things and potentially making sure that it won’t update to get sent before a newer one? **

We queue them and batch them up.

**Do you guys actually use some kind of continuous creation, like HTTP 2.0 or Speedy, because it sounds like that would be kind of ideal for this.

We use a sort of persistent web socket, like a socket that’s continuously open and providing updates. That’s one of the ideas we have among others to try and help improve the performance. Right now with doze mode becoming more and more optimized on Androids, it’s not quite clear what is going to happen with the sockets. You do open a socket to the server, and your application goes to the background; Android might start killing the sockets, and then in that case, I don’t know how much you gain by having an open socket that’s going to be destroyed in Android 6 and Nougat because doze mode is shutting them down. It’s one area of research that we’re going to start looking into. We don’t have results yet, though, unfortunately.

Are there are any third-party libraries you’ve come to enjoy using that we may not have heard about or that have been really helpful for building this out and that you’re willing to share?

When I did the getLastLocation, that is known as the Android React to Location Library, that was actually a pretty nice one. Reactive RX is something that’s starting to be used more and more by the applications, so there’s a lot of interesting use cases around that, making the code a little more elegant, that’s one example of one.

In terms of other interesting libraries, we use a number of the fairly standard ones for image downloading, volume, and we have a bunch of stuff around dagger to help dependency injection.

A lot of them are pretty standard, it helps us as developers, it moves a lot more quickly. But for the location, that was the first time I saw that, I’m not on the buyer team, I’m on a different team, so being able to go through and understand how the buyer application actually did this, that was quite surprising to me, that’s why I added a slide for it.

Thank you.