Building Functional Apps

As things become more encapsulated & abstract, it becomes easier to reuse components. Sam Soffes, the man behind projects such as Hipstamatic, Cheddar & SSToolkit, demonstrates how to take this to the extreme, by reusing Swift code between Mac and iOS, for his popular app, Redacted.

---

Motivation for Redacted (1:42)

I recently traveled to Iceland with some friends, and I tweeted a screenshot of our itinerary. It had our confirmation numbers and prices, but I thought it was silly to use Photoshop just to cross things out. So, I decided to write an app called Redacted in the week of downtime I had before the trip. I wanted to solve this problem, and I paired it with a code challenge for myself.

Redacted is currently for the Mac, and will be coming out for iOS. This is the main point I want to talk about - iOS and Mac. My goal in writing Redacted was to see how much code I could share and reuse, and in fact, almost all of the code is shared between the two apps.

Sharing Code (3:33)

Share what makes sense! The “build once, run everywhere” idea is a terrible thing to do. For example, if you have a normal iPhone app that has tabs, the Android app would show the tabs on the bottom. This is ridiculous because on Android, tabs are at the top. Redacted’s menu bar has a lot of complexity on the Mac. Menu bars are the worst. You can drag things onto the dock icon, and you can also drag and drop from Safari. All of this is specific to the Mac, but if I made all of this for iOS, it would be terrible. There could be a navigation bar with segmented control, a label, and some buttons, but that wouldn’t fit on an iPhone 4.

The solution I’m proposing when I say ”share everything” is not that kind of a solution. Share all the relevant code that isn’t ridiculous. The code that I share between iOS and Mac is the part when you can click and drag, and see the image. The rest of it is platform-specific.

Within reason, how can we share as much as possible? The simple answer is to not touch the things that you can’t share. So, I wouldn’t touch UIKit or AppKit at all. That was my strategy, and I learned a lot of interesting things trying this. There are some cool functional things, as well as more Swift-y things in general. There will also be some clever things that made me feel smart when I wrote them.

Functional (6:00)

Functional concepts are interesting. I used to have a hate relationship with functional programming. We have had classes for years, so why did we need this new thing? But the more I dove into functional programming, the more I thought it was spectacular. I never wanted to write anything not functional again. Because we aren’t constantly touching AppKit and UIKit, we can use all the Swift concepts that we want! I feel like the worst part of Swift is its interoperability. We still have to use initializers, as well as deal with implicitly unwrapping things, but we can use all the great things that come with Swift.

Models are Structs (6:55)

Models are structs. More importantly, they are value types - and I am totally in love with value types. In RedactedKit, there is a property UUID of type String. There is also type of type RedactionType, and rect of type CGRect. This CGRect is a normal CoreGraphics rect that stores everything in percentages of the original image, so that you don’t have to constantly scale up and down. We can just multiply by the width or height to obtain the coordinates.

struct Redaction: Hashable, Equatable {
	let UUID: String
	let type: RedactionType
	var rect: CGRect
	// ...
}

RedactionType is an enum that has the cases Pixelate, Blur, and BlackBar. It’s pretty simple, and is also printable. I really enjoy making new types and seeing how many Swift protocols I can adhere to for fun. There is also a String function that we’ll look at as well. At the bottom, there is a static var all that allows you to call RedactionType.all to get an array of all the types.

enum RedactionType: Int, Printable {
	case Pixelate, Blur, BlackBar
	var description: String {
		switch self {
		case .Pixelate:
			return string("PIXELATE")
		case .Blur:
			return string("BLUR")
		case .BlackBar:
			return string("BLACK_BAR")
		}
	}

	static var all: [RedactionType] {
		return [.Pixelate, .Blur, .BlackBar]
	}
}

We get an NSBundle from a class using the funciton bundle. From that bundle, we want to get a localized string so that we don’t have to use the identifier. Having the string function is really great because it’s public for the whole framework. This app is also localized in French, with the help of one of the guys I went to Iceland with.

private func bundle() -> NSBundle? {
	return NSBundle(forClass: RedactedView.self)
}
func string(key: String) -> String {
	if let bundle = bundle() {
		return bundle.localizedStringForKey(key, value: nil, table: nil)
	}
	return key
}

Core Image (9:36)

Now let’s talk about something more exciting that a struct or enum - image rendering. I use Core Image, which is awesome. If you’re not using it, you should. You can use it on the GPU as well as the CPU, and it will be lightning fast if you are using it in GPU mode only. I used it in an app I wrote last year called Lightly, which won editor’s choice.

Redacted works by blurring the entire image, cropping the region that you selected, and adding that on top of the original image. In Redacted, we have the method filter that takes in an image and returns a filter. We do some frame math at the top to figure out where the blur will be. We use a preprocessor to preprocess the image, and then composite the two images. The preprocessor is a function that will blur, pixelate, or use a black bar on the image. It is also a typealias for a function, which is pretty cool. That function Preprocessor takes in an image and returns an image.

func filter(image: CIImage, preprocessor: Preprocessor =
	Redaction.preprocess) -> CIFilter {
		let extent = image.extent()
		let scaledRect = rectForBounds(extent).flippedInRect(extent)
		let processed = preprocessor(image: image, type: type)
		
		return CIFilter(name: "CISourceOverCompositing",
			withInputParameters: [
				"inputImage": processed.imageByCroppingToRect(scaledRect)
		])
}

typealias Preprocessor = (image: CIImage, type: RedactionType) -> CIImage

Function Types, Neat (11:33)

Function types are something you can do in Objective-C, although structs you cannot. I feel like I would have never used a block in this way, but because I’m writing in Swift, I am more creative. The following code was the default implementation that I had written. It’s a static func that basically switches on the type to do some code. It uses Core Image to return back the preprocessed image.

static func preprocess(image: CIImage, type: RedactionType) -> CIImage {
	switch type {
	case .Pixelate:
		return // ...
	case .Blur:
		return // ...
	case .BlackBar:
		return // ...
	}
}

Is Blur Secure? (12:14)

A lot of people have asked me if the blur is secure. This is something I hadn’t thought of beforehand. No blur is really, truly secure. A blur is basically a hash, like MD5. You could try every string that exists until you found the unhashed string. Blurring an image does something similar to obscure information, but is harder to figure out because the kerning is different. There are more variables in play. According to Stack Overflow, there are 281,219,526 possible six letter English words. If you multiply that by the number of typefaces, font sizes, and kerning, there are so many things that a blur could be.

In order to make a blur more secure, obfuscate first. Put some noise on it, or distort the image first. This way, someone who tries to use brute force to figure out what is behind the blur will never get the input because it is totally crazy and random.

Blur is Slow (14:54)

I realised that I had a terrible problem, where the blurring was really slow. As you dragged the selection area, the entire image was being re-rendered in every frame. The solution I came up with was this RedactionsController. This is a class that contains an array of Redactions. It basically takes an image and converts it to a private ciImage.

class RedactionsController {
	var redactions = [Redaction]()

	var image: UIImage? {
		didSet {
			if let image = image {
				ciImage = CIImage(CGImage: image.CGImage)
			} else {
				ciImage = nil
			}
		}
	}

	private var ciImage: CIImage? {
		didSet {
			updateImages()
		}
	}
	// ...
}

I just wanted to work internally with CIImages, and to not touch UIKit. In the didSet of ciImage, we call updateImages. There are private properties for pixelated for blurred images. I use the same static function preprocess from before to cache the images.

private var pixelatedImage: CIImage?
private var blurredImage: CIImage? 

private func updateImages() {
	if let ciImage = ciImage {
		pixelatedImage = Redaction.preprocess(ciImage, type: .Pixelate)
		blurredImage = Redaction.preprocess(ciImage, type: .Blur)
	} else {
		pixelatedImage = nil
		blurredImage = nil
	}
}

This is the process method in RedactionsController. If you want to process an image, you would set up a RedactionsController, add your redactions, add your image, and call process to get back a processed image that has everything redacted in the correct spot. The outputImage starts out the same as the inputImage, so it’s easy to pass through if there are no redactions. We can do some great Swift-y things, where we map our redactions to a filter. After we pass in our image and preprocessor, which is self.preprocess, we have a chain filter. That is a simple CI Filter that I made, which just chains together inputs and outputs for composite filters. We then return and crop the original, in case someone drew outside the bounds.

func process() -> CIImage? {
	if let ciImage = ciImage {
		var outputImage = ciImage
		if redactions.count > 0 {
			let chain = ChainFilter()
			chain.inputImage = ciImage
			chain.inputFilters = redactions.map({ $0.filter(ciImage,
				preprocessor: self.preprocess) })
			outputImage = chain.outputImage!
		}
		return outputImage.imageByCroppingToRect(ciImage.extent())
	}
	return nil
}

In the preprocess function, I switch on the type, whether that type is Pixelate or Blur. If it’s anything else, I fall back to the default implementation. I was in love with this pattern when I wrote it. In my mind, it is better than inheritance. We have a model that has this filter function, and when it’s in this specific controller I want to change what that does. It gets complicated really quickly when you convert between this model and the fancy cache model. You could use delegates, but that isn’t as nice as just calling filter when you have worry about cleaning up all this other state.

private func preprocess(image: CIImage, type: RedactionType) -> CIImage {
	switch type {
	case .Pixelate:
		return pixelatedImage!
	case .Blur:
		return blurredImage!
	default:
		return Redaction.preprocess(image, type: type)
	}
}

Value Types (19:06)

Let’s talk about value types a little more. In RedactedView, we have the following code. There is the RedactionsController, as well as the Redactions array redactions. This basically proxies through to the controller, and internally we just talk to redactions. In set, we call updateRedactions. The redactions array is like an array of structs, and inside of that is the rect property. If I change the x coordinate of any of those, this function is run.

private let redactionsController = RedactionsController()
public var redactions: [Redaction] {
	get {
		return redactionsController.redactions
	}
	set {
		redactionsController.redactions = newValue
		updateRedactions()
	}
}

In Objective-C, you have to KVO all the properties and all those things in your array. You have to use so much KVO and try not to leak or crash. There would be a lot of bookkeeping just to see if anything anywhere changed. You could use notifications, but that is kind of gross if you have multiple instances.

It’s so nice to have value types. When you change something, you get a new value with new things. It’s not a reference type, where you have the same object. That’s how this works in Swift, with structs and Swift arrays. You get a copy of the whole thing back, but with the change made.

Automatic Rendering (21:02)

So, in Redacted, we can easily get updated redactions when anything changes. This is great for automatic rendering. I can change some properties in my model and everything changes for me. I don’t ever have to think about it, since it’s all values and niceness. The function updateRedactions is in the view, and sets the image to redactionsController.process. This is the only other time we talk to the controller. It renders the image and calls updateSelections.

func updateRedactions() {
	image = redactionsController.process()
	updateSelections()
}

RedactedView is a subclass of CoreImageView, which is itself a subclass of OpenGLView. There is a lot of work to make this code cross-platform, especially on the OpenGL side. We have OpenGL ES for iOS, but OpenGL for Mac. These are not even related. You have GLKView on iOS, which is awesome, but GLKit on Mac, which doesn’t have the view. You have to make your own, which is a huge pain.

class CoreImageView: OpenGLView {
	var image: CIImage? {
		didSet {
			draw()
		}
	}
	// ...
}

The Mac version is actually written in Objective-C, and there is Core Image code on top that basically renders Core Image into the OpenGL magic. That keeps it all in the GPU at super fast speeds. I definitely plan on open sourcing this part as soon as it’s finished. It works great on iOS, and the Mac version still needs a little bit of work.

Selection (22:50)

Selection and creation were my biggest struggles when it came to thinking about value types and the functional mindset. I’m very happy with my solution. In RedactedView, we have a Swift set of Strings Set<String> of the selectedUUIDs. Whenever we create a redaction, we have an associated UUID. A UUID is a random string that is so large it is guaranteed to be random. This way, you don’t have to worry about collisions. On selection, we updateSelections. We can see very easily if something is selected by checking if the UUID is in the Set.

var selectedUUIDs = Set<String>() {
	didSet {
		updateSelections()
	}
}

func isSelected(redaction: Redaction) -> Bool {
	return selectedUUIDs.contains(redaction.UUID)
}

The select method is rather long, but it uses some interesting concepts. It already selected, we do anything. Insert into the set, and then we use Core Animation. We’ll talk about that in a minute. We then make a bounding box, which is just a CALayer, put that in our dictionary of boundingBoxes, and add that as a subview to our layer. The reason we have this layer local variable that’s optional to self.layer is for cross-platforms. On Mac, layer is optional. This is way annoying because you didn’t have layers on Mac until several years ago. On iOS, it’s always there, so it isn’t optional. This code is a nice way to handle this without rewriting code for the rest of the file. You can’t make an extension to change the optionality of a property, unfortunately, unless you maybe make a new property. You would have to change the new, which I did not want to do.

func select(redaction: Redaction) {
	if isSelected(redaction) {
		return
	}

	selectedUUIDs.insert(redaction.UUID)
	CATransaction.begin()
	CATransaction.setDisableActions(true)

	let box = BoundingBoxLayer()
	boundingBoxes[redaction.UUID] = box
	
	let layer: CALayer? = self.layer
	layer?.addSublayer(box)
	
	updateSelections()
	CATransaction.commit()
}

Core Animation (25:41)

I didn’t want to touch UIKit or AppKit at all, and the only thing that does is a subclass of UIViewer and NSView. Nothing else does, except for some colours here and there. Those are actually CGColors. Doing this, I’m using layers everywhere for my subviews, which works out for cross-platform. CALayer has a very similar API to NSView or UIView because on top of being able to add sublayers, have a superlayer, set frame and bounds, and draw rect, you can animate all of that. It does most of the work for you too.

The function updateSelections takes the bounding box that we’ve already made previously and then lays them out. As you’re dragging, the bounding boxes will track the rendering. Everything stays in sync really nicely, and we don’t have to really worry about it. The reason I inset it is so it draws on the outside and not on the inside, which makes sense. We post a notification, mainly because the NSMenu is really hard to work with. Because you can’t KVO in Swift, it was much easier to do this.

func updateSelections() {
	CATransaction.begin()
	CATransaction.setDisableActions(true)

	for redaction in selectedRedactions {
		if let layer = boundingBoxes[redaction.UUID] {
			var rect = redaction.rectForBounds(imageRect)
			rect.inset(dx: -2, dy: -2)
			layer.frame = rect
		}
	}
	
	CATransaction.commit()
	NSNotificationCenter.defaultCenter().postNotificationName(...
}

Dragging (28:26)

Thinking about selections was interesting, because I needed to think about them as not values. I need a reference as they change, so I used UUIDs. That pattern of thinking was helpful in figuring out how I thought about them as they changed. With draggingMode, there are associated values. If it’s Creating, we have an associated string; if it’s Moving, we have a string, a rect, and a point. The view just has an optional DraggingMode?.

enum DraggingMode {
	case Creating(String)
	case Moving(String, CGRect, CGPoint)
}

var draggingMode: DraggingMode?

The drag method is incredibly long. Figuring this out was another huge step forward for me and my Swift-y valueness. I had to figure out how to edit something in place. Part of it is using UUIDs to find things, like I’ve talked about already. This method takes a point and state from a GestureRecognizer. Mac has that now, which is great for cross-platforms.

public func drag(#point: CGPoint, state: GestureRecognizerState) {
	if image == nil {
	return
	}

	...
}

One of the two things a view controller does is take the gesture recognizer and forward it to the view. I convert the point to percentages, which is the only time I convert it that way. When the drag begins, we need to deselect everything and see whether we already have a redaction. If we’re dragging one we already have, we’ll make the mode and associate the UUID, the original rect, and the starting point of the drag. If we’re making a new one, we’ll make a new one with a new model, set its origin to the start point, set its size to zero, and set its associated value to the UUID. Then we’ll add it to the array.

let point = convertPointToUnits(point)

// Begin dragging
if state == .Began {
	deselectAll()

	// Start moving
	if let redaction = hitTestRedaction(point) {
		draggingMode = .Moving(redaction.UUID, redaction.rect, point)
		select(redaction)
	}

	// Start creating
	else {
		let redaction = Redaction(type: mode, rect:
			CGRect(origin: point, size: .zeroSize))
		draggingMode = .Creating(redaction.UUID)
		redactions.append(redaction)
		select(redaction)
	}
}

Once the setup is finished, we switch to see if we’re creating. We update the size based on the origin and the current point. The origin is the start point, and the current point is where we’re currently dragging. The math is easy enough. The moving involves similar frame math. If we’re ending the drag, we stop dragging. Enum associated values were the perfect solution for this editin gmode. I had four or five properties to store all of this state, and without this solution, I could get out of sync due to all the interdependencies. With UUIDs, I can find any redaction, edit it, and replace it in the array to cause everything to be redrawn. Everything happens automatically just by editing the array.

// Continue dragging
if let draggingMode = draggingMode {
	switch draggingMode {
	case let .Creating(UUID):
		// Update size based on origin and current point
	case let .Moving(UUID, rect, startPoint):
		// Update rect based on original rect and start point
	}
}

// End dragging
if state == .Ended {
	draggingMode = nil
}

Clever Things (33:42)

There some things that made cross-platform coding easier. My goal setting out for this part was to not have ifdef anywhere in my code. All my code would then be shareable, and I wouldn’t have to worry about what platform I was on. For easier cross-platform development, I then wrote X, which is open source on GitHub. One fairly common pattern is to alias FontType to UIFont and NSFont, conditionally. Then I alias Font to FontType. like UIFont and NSFont, conditionally. I can do the same for colours and images.

#if os(iOS)
	import UIKit.UIFont
	public typealias FontType = UIFont
#else
	import AppKit.NSFont
	public typealias FontType = NSFont
#endif

public typealias Font = FontType

ContentMode doesn’t exist on the Mac at all. There are also nice functions on CGRect to apply a content mode to a rect or a size, so you can aspect fit. EdgeInsets on the Mac doesn’t have InsetRect, so I have my own implementation. There is a lot more, including Screen and Window, and I’m doing this as I go. I’m not sitting down to alias everything in AppKit, because that would be a huge amount of work for something I would probably never use. I’ll definitely be working ond documentation next, and it would be spectacular if you want to send me a pull request.

Resources (37:58)

Lastly, there are three resources that were very helpful to me. At the Functional Swift Conference, the talks by Andy Matuschak and Justin Spahr-Summers are spectacular. Definitely watch those if you are interested in value types. I also like some things from the book Functional Programming in Swift.

Q&A (38:22)

Q: I was really fascinated by the way you used UUIDs everywhere, in the way you would use pointers if you were using class types. Were there anything things that surprised you using this approach?

Sam: Since everything is immutable value types, it isn’t as big as a problem as it seems like it would be. I didn’t really have any problems, mainly because I’m only using them in one spot. Everything was very contained, so the only things referencing the UUIDs were something that created it in that same class. I’m not passing them around between different objects. If I was, I would probably try a different approach. Archiving also made it easier, and state restoration works really well. I definitely think of them similar to pointers, but in more of a database way. Honestly, I had no problems with this.

Q: I like what you said about Swift offering the opportunity to be more creative, but that can be dangerous on a team. If you want to do something like add a custom operator, how do you avoid issues like code readability?

Sam: Venmo is a very old app, so most of it is all Objective-C. All of the new code we’re writing is in Swift. Across the whole team, we don’t have really formal rules. Thoughts are that making a custom operator is a bad idea, because no one is going to know what it means or how to use it. Most of our Swift code is encapsulated because it has to work with all of the Objective-C, so that is usually just a view or view controller. Even if you have some clever code, it still works. One thing we do is when we realise we’re going to have interoperability problems - we just go back to Objective-C.

The way I think about things in general is that I want to make them libraries first. For example, with Redacted, I made all of it a library before I even wrote the UI. That’s just how I like to work. With that kind of mindset, it works pretty well to have all the Swift encapsulated away, and you can kind of do crazy things.

Q: What are some challenges of writing some things functionally that you used to do with classes and objects? Did you have shifts in your thinking?

Sam: The biggest thing was the selections and the editing. Before, I would have had an array of objects that were all reference types. I could have edited them in place, and then called some reload method. Now, I had to think about replacing them with IDs. That was a really hard shift in my thinking, and I struggled with that for a while. It also took me several days to think of the solution where I used the function to patch in the preprocessor. To think of functions as first class objects that can be passed around, that was really great. I think just the biggest things were value types. That was a huge goal for me. Those two talks I mentioned at the end were definitely super helpful in thinking through those issues.

Q: Could you elaborate more on CIImage versus GPUImage?

Sam: The API for GPUImage is really confusing, and backwards the way I would think about things. That is my biggest frustration. If you look at the code, it’s pretty scary. I haven’t had great experiences taking over a client project that uses GPUImage. CIImage is supported by Apple, which has its pros and cons, but it’s really fast if you use it on the GPU. In iOS 8, you can make your own kernels. That was a huge drawback for GPUImage. CIImage also handles memory a lot better and is a lot more performant. There’s really no reason not to use CIImage, and my default stance is to use Apple tools if they’re available. There’s no reason to use GPUImage because the Apple tools do everything I need. I’ve never had an app crash from CI, but I have had quite a few crashes from GPU.

Q: I didn’t notice any background dispatch, so I was wondering, do you have any performance issues with Core Image?

Sam: Currently, everything is in the main thread in this app, and I haven’t had any problems. On iOS it’s still sixty frames, tracks my finger. Because it’s on the GPU only, it’s not like CGImage. All of that you can definitely use from the background, and I’ve done that before. If you’re computing your filters, it takes more time. These filters are really simple, but if that logic was complicated you could definitely do it from the background. The GLKView doesn’t care.

Q: What are your thoughts on WatchKit so far?

Sam: I like WatchKit a lot, actually. I built a WatchKit app called Shares, which is no longer in the app store. I used an old Yahoo API that changed, causing the app to break. I didn’t have time to fix it, so I just pulled it. My favorite part of WatchKit is just the constraint, because it’s so simple. I wish I could do more, but I do really enjoy the constraint. Using IB wasn’t nearly as bad as I thought it’d be. It has definitely improved. I don’t really have any big complaints. It’s annoying that you can’t use the Darwin notification system from Swift, but there is a great library called Portal that does that for you.

Q: I noticed that you were using enums to manage state transitions when handling the gesture recognisers. Could you comment on how you used enums in that context?

Sam: The gesture recognizer state type in the parameter, which is part of X, just bridges the NS and UI. That has its own state and can track the gesture. My problem was that dragging on something that was already redacted is different than dragging out in the open. My first attempt was to have properties that would be true or false, and an optional. I needed my starting point, my starting rect, the UUID. There was all this state that I had to track, which I didn’t like because it was all separate.

I think the associated value enum works really well because I can say I’m doing this or that. If I’m doing this, this is the data I need to do that; if I’m doing that, this is the data I need for that. I think this worked out super well. It wasn’t hashing around the original redaction. It was passing around the pieces of the original redaction that I might need, as well as a reference to the current one so I can replace it. It’s unfortunate that I had another layer of state behind the gesture recognizer. For example, the tap code doesn’t have any of this, it just handles the gesture recognizers.

---