Swift structs can provide mutating and nonmutating versions of their functions. However, the Swift standard library is inconsistent about when it provides one form of a function or the other. Some APIs come in only mutating forms, some come only in nonmutating, and some come in both. Because both types of functions are useful in different situations, I argue that almost all such functions should have both versions provided. In this post, we’ll examine when these different forms are useful, examples APIs that provide one but not the other, and solutions for this problem.

(Quick sidebar here: I’ll be referring to functions that return a new struct with some field changed as nonmutating functions. Unfortunately, Swift also includes a keyword called nonmutating, which is designed for property setters that don’t require the struct to be copied when they’re set. Because I think that nonmutating is the best word to describe functions that return an altered copy of the original, I’ll keep using that language here. Apologies for the confusion.)

Mutating and Nonmutating Forms

These two forms, mutating and nonmutating, are useful in different cases. First, let’s look at when the nonmutating version is more useful.

This code, taken from Beacon’s OAuthSignatureGenerator, uses a long chain of immutable functions to make its code cleaner and more uniform:

public var authorizationHeader: String {
	let headerComponents = authorizationParameters
		.dictionaryByAdding("oauth_signature", value: self.oauthSignature)
		.urlEncodedQueryPairs(using: self.dataEncoding)
		.map({ pair in "\(pair.0)=\"\(pair.1)\"" })
		.sorted()
		.joined(separator: ", ")
	return "OAuth " + headerComponents
}

However, it wouldn’t be possible without an extension adding dictionaryByAdding(_:,value:), the nonmutating version of the subscript setter.

Here’s another, slightly more academic example. Inverting a binary tree is more commonly a punchline for bad interviews than practically useful, but the algorithm illustrates this point nicely. To start, let’s define a binary tree an enum like so:

indirect enum TreeNode<Element> {
	case children(left: TreeNode<Element>, right: TreeNode<Element>, value: Element)
	case leaf(value: Element)
}

To invert this binary tree, the tree on the left goes on the right, and the tree on right goes on the left. The recursive form of nonmutating version is simple and elegant:

extension TreeNode {
	func inverted() -> TreeNode<Element> {
		switch self {
		case let .children(left, right, value):
			return .children(left: right.inverted(), right: left.inverted(), value: value)
		case let .leaf(value: value):
			return .leaf(value: value)
		}
	}
}

Whereas the recursive mutating version contains lots of extra noise and nonsense:

extension TreeNode {
	mutating func invert() {
		switch self {
		case let .children(left, right, value):
			var rightCopy = right
			rightCopy.invert()
			var leftCopy = left
			leftCopy.invert()
			self = .children(left: rightCopy, right: leftCopy, value: value)
		case .leaf(value: _):
			break
		}
	}
}

Mutating functions are also useful, albeit in different contexts. We primarily work in apps that have a lot of state, and sometimes that state is represented by data in structs. When mutating that state, we often want to do it in place. Swift gives us lots of first class affordances for this: the mutating keyword was added especially for this behavior, and changing any property on a struct acts as a mutating function as well, reassigning the reference to a new copy of the struct with the value changed.

There are concrete examples as well. If you’re animating a CGAffineTransform on a view, that code currently has to look something like this.

UIView.animate(duration: 0.25, animations: {
	view.transform = view.transform.rotated(by: .pi)
})

Because the transformation APIs are all nonmutating, you have to manually assign the struct back to the original reference, causing duplicated, ugly code. If there were a mutating rotate(by:) function, then this code would be much cleaner:

UIView.animate(duration: 0.25, animations: {
	view.transform.rotate(by: .pi)
})

The APIs aren’t consistent

The primary problem here is that while both mutating functions and nonmutating functions are useful, not all APIs provide versions for both.

Some APIs include only mutating APIs. This is common with collections. Both the APIs for adding items to collections, like Array.append, Dictionary.subscript, Set.insert, and APIs for removing items from collections, like Array.remove(at:), Dictionary.removeValue(forKey:), and Set.remove, have this issue.

Some APIs include only nonmutating APIs. filter and map are defined on Sequence, and they return arrays, so they can’t be mutating (because Sequence objects can’t necessarily be mutated). However, we could have a mutating filterInPlace function on Array. The aforementioned CGAffineTransform functions also fall in this category. They only include nonmutating versions, which is great for representing a CGAffineTransform as a chain of transformations, but not so great for mutating an existing transform, say, on a view.

Some APIs provide both. I think sorting is a great example of getting this right. Sequence includes sorted(by:), which is a nonmutating function that returns a sorted array, whereas MutableCollection (the first protocol in the Sequence and Collection heirarchy that allows mutation) includes sort(by:). This way, users of the API can choose whether they want a mutating sort or a nonmutating sort, and they’re available in the API where they’re first possible. Array, of course, conforms to MutableCollection and Sequence, so it gets both of them.

Another example of an API that gets this right: Set includes union (nonmutating) and formUnion (mutating). (I could quibble with these names, but I’m happy that both version of the API exist.) Swift 4’s new Dictionary merging APIs also include both merge(_:uniquingKeysWith:) and merging(_:uniquingKeysWith:).

Bridging Between the Two

The interesting thing with this problem is that, because of the way Swift is designed, it’s really easy to bridge from one form to the other. If you have the mutating version of any function:

mutating func mutatingVersion() { ... }

You can synthesize a nonmutating version:

func nonMutatingVersion() -> Self {
	var copy = self
	copy.mutatingVersion()
	return copy
}

And vice versa, if you already have a nonmutating version:

func nonMutatingVersion() -> Self { ... }

mutating func mutatingVersion() {
	self = self.nonMutatingVersion()
}

As long as the type returned by the nonmutating version is the same as Self, this trick works for any API, which is awesome. The only thing you really need is the new name for the alternate version.

Leaning on the Compiler

With this, I think it should be possible to have the compiler trivially synthesize one version from the other for us. Imagine something like this:

extension Array {

	@synthesizeNonmutating(appending(_:))
	mutating func append(_ newElement) {
		// ...
	}
	
}

The compiler would use the same trick above — create a copy, mutate the copy, and return it — to synthesize a nonmutating version of this function. You could also have a @synthesizeMutating keyword. Types could of course choose not to use this shorthand, which they might do in instances where there are there are optimizations for one form or another. However, getting tooling like this means that API designers no longer have to consider whether they’re APIs are likely to be used in mutating or nonmutating ways, and they can easily add both forms. Because each form is useful in different contexts, providing both allows the consumer of an API to choose which form they want and when.

Very early this year, I posted about request behaviors. These simple objects can help you factor out small bits of reused UI, persistence, and validation code related to your network. If you haven’t read the post, it lays the groundwork for this post.

Request behaviors are objects that can exceute arbitrary code at various points during some request’s execution: before sending, after success, and after failure. They also contain hooks to add any necessary headers and arbitrarily modify the URL request. If you come from the server world, you can think of request behaviors as sort of reverse middleware. It’s a simple pattern, but there are lots of very powerful behaviors that can be built on top of them.

In the original post, I proposed 3 behaviors, that because of their access to some piece of global state, were particularly hard to test: BackgroundTaskBehavior, NetworkActivityIndicatorBehavior, AuthTokenHeaderBehavior. Those are useful behaviors, but in this post, I want to show a few more maybe less obvious behaviors that I’ve used across a few apps.

API Verification

One of the apps where I’ve employed this pattern needs a very special behavior. It relies on receiving 2XX status codes from sync API requests. When a request returns a 200, it assumes that sync request was successfully executed, and it can remove it from the queue.

The problem is that sometimes, captive portals, like those used at hotels or at coffee shops, will often redirect any request to their special login page, which returns a 200. There are a few ways to handle this, but the solution we opted for was to send a special header that the server would turn around and return completely unmodified. No returned header? Probably a captive portal or some other tomfoolery. To implement this, the server used a very straightforward middleware, and the client needed some code to handle it as well. Perfect for a request behavior.

class APIVerificationBehavior: RequestBehavior {

    let nonce = UUID().uuidString

    var additionalHeaders: [String: String] {
        return ["X-API-Nonce": nonce]
    }

    func afterSuccess(response: AnyResponse) throws {
        guard let returnedNonce = response.httpResponse.httpHeaderFields["X-API-Nonce"],
            returnedNonce == nonce {
                throw APIError(message: "Sync request intercepted.")
        }
    }
}

It requires a small change: making the afterSuccess method a throwing method. This lets the request behavior check conditions and fail the request if they’re not met, and it’s a straightforward compiler-driven change. Also, because the request behavior architecture is so testable, making changes like this to the network code can be reliably tested, making changes much easier.

OAuth Behavior

Beacon relies on Twitter, which uses OAuth for authentication and user identification. At the time I wrote the code, none of the Swift OAuth libraries worked correctly on Linux, so there was a process of extracting, refactoring, and, for some components, rewriting the libraries to make them work right. While I was working on this, I was hoping to test out one of the reasons I wanted to created request behaviors in the first place: to decouple the authentication protocol (OAuth, in this case) from the data that any given request requires. You should be able to transparently add OAuth to a request without having to modify the request struct or the network client at all.

Extracting the code to generate the OAuth signature was a decent amount of work. Debugging in particular is hard for OAuth, and I recommend this page on Twitter’s API docs which walks you through the whole process and shows you what your data should look like at each step. (I hope this link doesn’t just break when Twitter inevitably changes its documentation format.) I also added tests for each step, so that if anything failed, it would be obvious which steps succeeded and which steps failed.

Once you have something to generate the OAuth signature (called OAuthSignatureGenerator here), the request behavior for adding OAuth to a request turns out to not be so bad.

struct Credentials {
    let key: String
    let secret: String
}

class OAuthRequestBehavior: RequestBehavior  {

    var consumerCredentials: Credentials

    var credentials: Credentials
    
    init(consumerCredentials: Credentials, credentials: Credentials) {
        self.consumerCredentials = consumerCredentials
        self.credentials = credentials
    }
    
    func modify(request: URLRequest) -> URLRequest {
        let generator = OAuthSignatureGenerator(consumerCredentials: consumerCredentials, credentials: credentials, request: request)
        var mutableRequest = request
        mutableRequest.setValue(generator.authorizationHeader, forHTTPHeaderField: "Authorization")
        return mutableRequest
    }
    
}

Using modify(request:) to perform some mutation of the request, we can add the header for the OAuth signature from OAuthSignatureGenerator. Digging into the nitty-gritty of OAuth is a out of the scope of this post, but you can find the code for the signature generator here. The only thing of note is that this code relies on Vapor’s SHA1 and base 64 encoding, which you’ll have to swap out for implementations more friendly to your particular environment.

When it’s time to use the use this behavior to create a client, you can create a client specific to the Twitter API, and then you’re good to go:

let twitterClient = NetworkClient(
	configuration: RequestConfiguration(
		baseURLString: "https://api.twitter.com/1.1/",
		defaultRequestBehavior: oAuthBehavior
	)
)

Persistence

I also explored saving things to Core Data via request behaviors, without having to trouble the code that sends the request with that responsibility. This was another promise of the request behavior pattern: if you could write reusable and parameterizable behaviors for saving things to Core Data, you could cut down on a lot of boilerplate.

However, when implementing this, we ran into a wrinkle. Each request needs to finish saving to Core Data before the request’s promise is fulfilled. However, the current afterSuccess(result:) and afterFailure(error:) methods are synchronous and called on the main thread. Saving lots of data to Core Data can take seconds, during which time the UI can’t be locked up. We need to change these methods to allow asynchronous work. If we define a function that takes a Promise and returns a Promise, we can completely subsume three methods: beforeSend, afterSuccess, afterFailure.

func handleRequest(promise: Promise<AnyResponse>) -> Promise<AnyResponse> {
	// before request
	return promise
		.then({ response in
			// after success
		})
		.catch({ error in
			// after failure
		})
}

Now, we can do asynchronous work when the request succeeds or fails, and we can also cause a succeeding to request to fail if some condition isn’t met (like in the first example in this post) by throwing from the then block.

Core Data architecture varies greatly from app to app, and I’m not here to prescribe any particular pattern. In this case, we have a foreground context (for reading) and background context (for writing). We wanted simplify the creation of a Core Data request behavior; all you should have to provide is a context and a method that will be performed by that context. Building a protocol around that, we ended up with something like this:

protocol CoreDataRequestBehavior: RequestBehavior {

	var context: NSManagedObjectContext { get }
	
	func performBeforeSave(in context: NSManagedObjectContext, withResponse response: AnyResponse) throws
	
}

And that protocol is extended to provide a handle all of the boilerplate mapping to and from the Promise:

extension CoreDataRequestBehavior {

	func handleRequest(promise: Promise<AnyResponse>) -> Promise<AnyResponse> {
		return promise.then({ response in
			return Promise<AnyResponse>(work: { fulfill, reject in
				self.context.perform({
					do {
						try self.performBeforeSave(in: context, withResponse: response)
						try context.save()
						fulfill(response)
					} catch let error {
						reject(error)
					}
				})
			})
		})
	}

}

Creating a type that conforms to CoreDataRequestBehavior means that you provide a context and a function to modify that context before saving, and that function will be called on the right thread, will delay the completion of the request until the work in the Core Data context is completed. As an added bonus, performBeforeSave is a throwing function, so it’ll handle errors for you by failing the request.

On top of CoreDataRequestBehavior you can build more complex behaviors, such as a behavior that is parameterized on a managed object type and can save an array of objects of that type to Core Data.

Request behaviors provide the hooks to attach complex behavior to a request. Any side effect that needs to happen during a network request is a great candidate for a request behavior. (If these side effects occur for more than one request, all the better.) These three examples highlight more advanced usage of request behaviors.

The model layer of a client is a tough nut to crack. Because it’s not the canonical representation of the data (that representation lives on the server), the data must live a fundamentally transient form. The version of the data that the app has is essentially a cache of what lives on the network (whether that cache is in memory or on-disk), and if there’s anything that’s true about caches, it’s that they will always end up with stale data. The truth is multiplied by the number of caches you have in your app. Reducing the number of cached versions of any given object decreases the likelihood that it will be out of date.

Core Data has an internal feature that ensures that there is never more than one instance of an object for a given identifier (within the same managed object context). They call this feature “uniquing”, but it is known more broadly as the identity map pattern. I want to steal it without adopting the rest of baggage of Core Data.

I think of this concept as a “flat cache”. A flat cache is basically just a big dictionary. The keys are a composite of an object’s type name and the object’s ID, and the value is the object. A flat cache normalizes the data in it, like a relational database, and all object-object relationships go through the flat cache. A flat cache confers several interesting benefits.

  • Normalizing the data means that you’ll use less bandwidth while the data is in-flight, and less memory while the data is at rest.
  • Because data is normalized, modifying or updating a resource in one place modifies it everywhere.
  • Because relationships to other entities go through the flat cache, back references with structs are now possible. Back references with classes don’t have to be weak.
  • With a flat cache of structs, any mutation deep in a nested struct only requires the object in question to change, instead of the object and all of its parents.

In this post, we’ll discuss how to make this pattern work in Swift. First, you’ll need the composite key.

struct FlatCacheKey: Equatable, Hashable {

    let typeName: String
    let id: String
    
    static func == (lhs: FlatCacheKey, rhs: FlatCacheKey) -> Bool {
	    return lhs.typeName == rhs.typeName && lhs.id == rhs.id
    }
		    
	var hashValue: Int {
		return typeName.hashValue ^ id.hashValue
	}
}

We can use a protocol to make the generation of flat cache keys easier:

protocol Identifiable {
    var id: String { get }	
}

protocol Cachable: Identifiable { }

extension Cachable {
    static var typeName: String {
        return String(describing: self)
    }
    
    var flatCacheKey: FlatCacheKey {
        return FlatCacheKey(typeName: Self.typeName, id: id)
    }
}

All of our model objects already have IDs so they can trivially conform to Cachable and Identifiable.

Next, let’s get to the basic flat cache.

class FlatCache {
    
    static let shared = FlatCache()
    
    private var storage: [FlatCacheKey: Any] = [:]
    
    func set<T: Cachable>(value: T) {
        storage[value.flatCacheKey] = value
    }
    
    func get<T: Cachable>(id: String) -> T? {
        let key = FlatCacheKey(typeName: T.typeName, id: id)
        return storage[key] as? T
    }
    	    
    func clearCache() {
        storage = [:]
    }
    
}

Not too much to say here, just a private dictionary and get and set methods. Notably, the set method only takes one parameter, since the key for the flat cache can be derived from the value.

Here’s where the interesting stuff happens. Let’s say you have a Post with an Author. Typically, the Author would be a child of the Post:

struct Author {
	let name: String
}

struct Post {
	let author: Author
	let content: String
}

However, if you wanted back references (so that you could get all the posts by an author, let’s say), this isn’t possible with value types. This kind of relationship would cause a reference cycle, which can never happen with Swift structs. If you gave the Author a list of Post objects, each Post would be a full copy, including the Author which would have to include the author’s posts, et cetera. You could switch to classes, but the back reference would cause a retain cycle, so the reference would need to be weak. You’d have to manage these weak relationships back to the parent manually.

Neither of these solutions is ideal. A flat cache treats relationships a little differently. With a flat cache, each relationship is fetched from the centralized identity map. In this case, the Post has an authorID and the Author would have a list of postIDs:

struct Author: Identifiable, Cachable {
	let id: String
	let name: String
	let postIDs: [String]
}

struct Post: Identifiable, Cachable {
	let id: String
	let authorID: String
	let content: String
}

Now, you still have to do some work to fetch the object itself. To get the author for a post, you would write something like:

FlatCache.shared.get(id: post.authorID) as Author

You could put this into an extension on the Post to make it a little cleaner:

extension Post {
	var author: Author? {
		return FlatCache.shared.get(id: authorID)
	}
}

But this is pretty painful to do for every single relationship in your model layer. Fortunately, it’s something that can be generated! By adding an annotation to the ID property, you can tell a tool like Sourcery to generate the computed accessors for you. I won’t belabor the explanation of the template, but you can find it here. If you have trouble reading it or understanding how it works, you can read the Sourcery in Practice post.

It will let you write Swift code like this:

struct Author {
	let name: String
	// sourcery: relationshipType = Post
	let postIDs: [String]
}

struct Post {
	// sourcery: relationshipType = Author
	let authorID: String
	let content: String
}

Which will generate a file that looks like this:

// Generated using Sourcery 0.8.0 — https://github.com/krzysztofzablocki/Sourcery
// DO NOT EDIT
	
extension Author {
	var posts: [Post] {
		return postIDs.flatMap({ id -> Post? in
			return FlatCache.shared.get(id: id)
		})
	}
}

extension Post {	
	var author: Author? {
		return FlatCache.shared.get(id: authorID)
	}	
}

This is the bulk of the pattern. However, there a few considerations to examine.

JSON

Building this structure from a tree of nested JSON is messy and tough. The system works a lot better if you use a structure of the JSON looks like the structure of the flat cache. All the objects exist in a big dictionary at the top level (one key for each type of object), and the relationships are defined by IDs. When a new JSON payload comes in, you can iterate over this top level, create all your local objects, and store them in the flat cache. Inform the requester of the JSON that the new objects have been downloaded, and then it can fetch relevant objects directly from the flat cache. The ideal structure of the JSON looks a lot like JSON API, although I’m not surpassingly familiar with JSON API.

Missing Values

One big difference between managing relationships directly and managing them through the flat cache is that with the flat cache, there is a (small) chance that the relationship won’t be there. This might happen because of a bug on the server side, or it might happen because of a consistency error when mutating the data in the flat cache (we’ll discuss mutation more in a moment). There are a few ways to handle this:

  • Return an Optional. What we chose to do for this app is return an optional. There are a lot of ways of handling missing values with optionals, including optional chaining, force-unrwapping, if let, and flatmapping, so it isn’t too painful to have to deal with an optional, and there aren’t any seriously deleterious effects to your app if a value is missing.
  • Force unwrap. You could choose to force-unwrap the relationship. That’s putting a lot of trust in the source of the data (JSON in our case). If a relationship is missing becuase of a bug on your server, your app will crash. This is really bad, but on the bright side, you’ll get a crash report for missing relationships, and you can fix it on the server-side quickly.
  • Return a Promise. While a promise is the most complex of these three solutions to deal with at the call site, the benefit is that if the relationship doesn’t exist, you can fetch it fresh from the server and fulfill the promise a few seconds later.

Each choice has its downsides. However, one benefit to code generation is that you can support more than one option. You can synthesize both a promise and an optional getter for each relationship, and use whichever one you want at the call site.

Mutability

So far I’ve only really discussed immutable relationships and read-only data. The app where we’re using the pattern has entirely immutable data in its flat cache. All the data comes down in one giant blob of JSON, and then the flat cache is fully hydrated. We never write back up to the server, and all user-generated data is stored locally on the device.

If you want the same pattern to work with mutable data, a few things change, depending on if your model objects are classes or structs.

If they’re classes, they’ll need to be thread safe, since each instance will be shared across the whole app. If they’re classes, mutating any one reference to an entity will mutate them all, since they’re all the same instance.

If they’re structs, your flat cache will need to be thread safe. The Sourcery template will have to synthesize a setter as well as a getter. In addition, anything long-lived (like a VC) that relies on the data being updated regularly should draw its value directly from the flat cache.

final class PostVC {
    var postID: String
    
    init(postID: String) {
        self.postID = postID
    }
    
    var post: Post? {
		get {
			return cache.get(id: postID)
		}
		set {
			guard let newValue = newValue else { return }
			cache.set(value: newValue)
		}
	}
}

To make this even better, we can pull a page from Chris Eidhof’s most recent blog post about struct references, and roll this into a type.

class Cached<T: Cachable> {
	
	var cache = FlatCache.shared
	
	let id: String
	
	init(id: String) {
		self.id = id
	}
	
	var value: T? {
		get {
			return cache.get(id: id)
		}
		set {
			guard let newValue = newValue else { return}
			cache.set(value: newValue)
		}
	}
}

And in your VC, the post property would be replaced with something like:

lazy var cachedPost: Cached<Post>(id: postID)

Lastly, if your system has mutations, you need a way to inform objects if a mutation occurs. NSNotificationCenter could be a good system for this, or some kind of reactive implementation where you filter out irrelevant notifications and subscribe only to the ones you care about (a specific post with a given post ID, for example).

Singletons and Testing

This pattern relies on a singleton to fetch the relationships. This has a chance of hampering testability. There are a few different options to handle this, including injecting the flat cache into the various objects that use it, and having the flat cache be a weak var property on each model object instead of a static let. At that point, any flat cache would be responsible for ensuring the integrity of its child objects references to the flat cache. This is definitely added complexity, and it’s a tradeoff that comes along with this pattern.

References and other systems

  • Redux recommends a pattern similar to this. They have a post describing how to structure your data called Normalizing State Shape.
  • They discuss it a bit in this post as well:

    The solution to caching GraphQL is to normalize the hierarchical response into a flat collection of records. Relay implements this cache as a map from IDs to records. Each record is a map from field names to field values. Records may also link to other records (allowing it to describe a cyclic graph), and these links are stored as a special value type that references back into the top-level map. With this approach each server record is stored once regardless of how it is fetched.

Vapor’s JSON handling leaves something to be desired. Pulling something out of the request’s JSON body looks like this:

var numberOfSpots: Int?

init(request: Request) {
	self.numberOfSpots = request.json?["numberOfSpots"]?.int
}

There’s a lot of things I don’t like about this code.

  1. The int hanging off the end is extraneous: The type system already knows self.numberOfSpots is an int; ideally, I wouldn’t have to tell it twice.
  2. The optional situation is out of control. The JSON property might not exist on the request, the title property might not exist in the JSON, and the property might not be a int. Each of those branches is represented by an Optional, which is flattened using the optional chaining operator. At the end of the expression, if the value is nil, there’s no way to know which one of the components failed.
  3. At the end of the ridiculous optional chain, the resulting numberOfSpots value must be optional. If I need the numberOfSpots property to be required, I need to add an extra variable and a guard.

     guard let numberOfSpots = request.json?["numberOfSpots"]?.int else {
         throw Abort(status: .badRequest, reason: "Missing 'numberOfSpots'.")
     }
     self.numberOfSpots = numberOfSpots
    

    Needless to say, this is bad code made worse. The body of the error doesn’t contain any information besides the key “numberOfSpots”, so there’s a little more duplication there, and that error isn’t even accurate in many cases. If the json property of the request is nil, that means that either the Content-Type header was wrong or that the JSON failed to parse, neither of which are communicated by the message “Missing ‘numberOfSpots’.” If the “numberOfSpots” key was present, but stored a string (instead of an int), the .int conversion would fail, resulting in an optional, and the error message would be equally useless.

Probably more than half of the requests in the Beacon API have JSON bodies to parse and dig values out of, so this is an important thing to get right.

The broad approach here is to follow the general model for how we parse JSON on the client. We can use type inference to deal with the extraneous conversions, and errors instead of optionals.

Let’s look at the errors first. We’ve discussed three possible errors: missing JSON, missing keys, and mismatched types. Perfect for an error enum:

enum JSONError: AbortError {

	var status: Status {
		return .badRequest
	}
	
	case missingJSON
	case missingKey(keyName: String)
	case mismatchedType(keyName: String, expectedType: String, actualType: String)
	
	var reason: String {
		switch self {
		case .missingJSON:
			return "The endpoint requires a JSON body and a \"Content-Type\" of \"application/json\"."
		case let .missingKey(keyName):
			return "This endpoint expects the JSON key \(missingKey), but it wasn't present."
		case let .mismatchedType(keyName, expectedType, actualType):
			return "This endpoint expects the JSON key '\(key)'. It was present, but did not have the expected type \(expectedType). It had type '\(actualType).'"
		}
	}
}

Once the errors have been laid out, we can begin work on the rest of this implementation. Using a similar technique to the one laid out Decoding JSON in Swift, we can begin to build things up. (I call it NiceJSON because Vapor provides a json property on the Request, and I’d like to not collide with that.)

class NiceJSON {
	let json: JSON?

	public init(json: JSON?) {
		self.json = json
	}

	public func fetch<T>(_ key: String) throws -> T {
		// ...
	}

}

However, here, we run into the next roadblock. I typically store JSON on the client as a [String: Any] dictionary. In Vapor, it’s stored as a StructuredData, which is an enum that can be stored in one of many cases: .number, .string, .object, .bool, .date, and so on.

While this is strictly more type-safe (a JSON object can’t store any values that aren’t representable in one of those basic forms — even though .date and .data are a cases of StructuredData, ignore them for now), it stands in the way of this technique. You need a way to bridge between compile-time types (like T and Int), and run-time types (like knowing to call the computed property .string). One way to handle this is to check the type of T precisely.

public func fetch<T>(_ key: String) throws -> T {

	guard let json = self.json else { throw JSONError.missingJSON }
	
	guard let untypedValue = json[key] else { throw JSONError.missingKey(key) }
	
	if T.self == Int.self {
		guard let value = untypedValue.int else {
			throw JSONError.mismatchedType(key, String.self)
		}
		return value
	}
	// handle bools, strings, arrays, and objects
}

While this works, it has one quality that I’m not crazy about. When you access the .int computed property, if the case’s associated value isn’t a int but can be coerced into a int, it will be. For example, if the consumer of the API passes the string “5”, it’ll be silently converted into a number. (Strings have it even worse: numbers are converted into strings, boolean values become the strings "true" and "false", and so on. You can see the code that I’m referring to here.)

I want the typing to be a little stricter. If the consumer of the API passes me a number and I want a string, that should be a .mismatchedType error. To accomplish this, we need to destructure Vapor’s JSON into a [String: Any] dictionary. Digging around the vapor/json repo a little, we can find code that lets us do this. It’s unfortunately marked as internal, so you have to copy it into your project.

extension Node {
	var backDoorToRealValues: Any {
		return self.wrapped.backDoorToRealValues
	}
}

extension StructuredData {
	internal var backDoorToRealValues: Any {
		switch self {
		case .array(let values):
			return values.map { $0.backDoorToRealValues }
		case .bool(let value):
			return value
		case .bytes(let bytes):
			return bytes
		case .null:
			return NSNull()
		case .number(let number):
			switch number {
			case .double(let value):
				return value
			case .int(let value):
				return value
			case .uint(let value):
				return value
			}
		case .object(let values):
			var dictionary: [String: Any] = [:]
			for (key, value) in values {
				dictionary[key] = value.backDoorToRealValues
			}
			return dictionary
		case .string(let value):
			return value
		case .date(let value):
			return value
		}
	}
}

The code is pretty boring, but it essentially converts from Vapor’s JSON type to a less well-typed (but easier to work with) object. Now that we have this, we can write our fetch method:

var dictionary: [String: Any]? {
	return json?.wrapped.backDoorToRealValues as? [String: Any]
}

public func fetch<T>(_ key: String) throws -> T {
	guard let dictionary = dictionary else {
		throw JSONError.jsonMissing()
	}
	
	guard let fetched = dictionary[key] else {
		throw JSONError.missingKey(key)
	}
	
	guard let typed = fetched as? T else {
		throw JSONError.typeMismatch(key: key, expectedType: String(describing: T.self), actualType: String(describing: type(of: fetched)))
	}
	
	return typed
}

It’s pretty straightforward to write implementations of fetchOptional(_:), fetch(_, transformation:), and the other necessary functions. I’ve gone into detail on them in the Decoding JSON in Swift post and in the Parser repo for that post, so I won’t dwell on those implementations here.

For the final piece, we need a way to access our new NiceJSON on a request. For that, I added a computed property to the request:

extension Request {
    public var niceJSON: NiceJSON {
        return NiceJSON(json: json)
    }
}

This version of the code creates a new NiceJSON each time you access the property, which can be optimized a little bit by constructing it once and sticking it in the storage property of the Request.

Finally, we can write the nice code that we want at the call site.

var numberOfSpots: Int

init(request: Request) throws {
	self.numberOfSpots = try request.niceJSON.fetch("numberOfSpots")
}

This code has provides a non-optional value, no duplication, and generates descriptive errors.

There’s on last gotcha that I want to go over: numbers in JSON. As of Vapor 2, all JSON numbers are stored as Double values. This means fetching numbers will only work if you fetch them as a Double. This doesn’t appear to be documentated anywhere, so I’m not sure how much we should rely on it, but it appears to currently work this way. I think the reason for it is NSNumber subtyping weirdness. On Mac Foundation, numbers in JSON are stored in the NSNumber type, which can be as casted to a Double, Bool, Int, or UInt. Because of the different runtime, that stuff doesn’t work the same way in Linux Foundation, so everything is stored in the least common denominator format, Double, which can (more or less) represent Int and UInt types.

I have a small workaround for this, which at the top of the fetch method, add a special case for Int:

public func fetch<T>(_ key: String) throws -> T {
    if T.self == Int.self {
        return try Int(self.fetch(key) as Double) as! T
    }

Bool works correctly without having to be caught in a special way, and you shouldn’t use UInt types regularly in your code. The note is copied below for posterity and link rot:

Use UInt only when you specifically need an unsigned integer type with the same size as the platform’s native word size. If this isn’t the case, Int is preferred, even when the values to be stored are known to be nonnegative. A consistent use of Int for integer values aids code interoperability, avoids the need to convert between different number types, and matches integer type inference, as described in Type Safety and Type Inference.

NiceJSON is a small abstraction that lets you work with JSON in clean way, without having to litter your code with guards, expectations, and hand-written errors about the presence of various keys in the JSON body.

Sourcery is a code generation tool for Swift. While we’ve talked about code generation on the podcast, I haven’t really talked much about it on this blog. Today, I’d like to go in-depth on a concrete example of how we’re using Sourcery in practice for an app.

The app in question uses structs for its model layer. The app is mostly read-only, and data comes down from JSON, so structs work well. However, we do need to persist the objects so they load faster the next time, and so we need NSCoding conformance.

Swift 4 will bring Codable, a new protocol that supports JSON encoding and decoding, as well as NSCoding. Using Codable with NSKeyedArchiver is a little different than you’re used to, but it basically works. I’ve written up a small code sample here that you can drop into a playground to test. While that will obviate this specific use case for codegen eventually, the technique is still useful in the abstract. The new Codable protocol works by synthesizing encoding and decoding implementations for your types, and until we get access to this machinery directly, Sourcery is the best way to steal this power for ourselves.

To persist structs, I’m using the technique that I lay in this blog post. Essentially, each struct that needs to be encodable will get a corresponding NSObject wrapper that conforms it to NSCoding. If you haven’t read that post, now is a good time. The background in that post is necessary for the approach detailed here.

The technique in that blog post somewhat pedantically subscribes to the single responsiblity principle. One type (the struct) stores the data in memory, and another type (the NSObject wrapper) adds the conformance to NSCoding. The downside to this seperation is that you have to maintain a second type: if you add a new property to one of your structs, you need to add it to the init(coder:) and encode(with:) methods manually. The upside is that the separate type can be really easily generated.

This is where Sourcery comes in. With Sourcery, you use a templating language called Stencil to define templates. When the app is built, one of the build phases “renders” these templates into actual Swift code that is then compiled into your app. Other blog posts go into detail about how to set Sourcery up, so I won’t go into detail here, except to say that we check the Sourcery binary into git (so that everyone is on the same version) and the generated source files as well. Like CocoaPods, it’s just easier if those files are checked in.

Let’s discuss the actual technique. First, you need a protocol called AutoCodable.

protocol AutoCodable { }

This protocol doesn’t have anything in its definition — it purely serves as a signal to Sourcery that this file should have an encoder generated for it.

In a new file called AutoCodable.stencil, you can enumerate the objects that conform to this protocol.

{% for type in types.implementing.AutoCodable %}
	
// ...

{% endfor %}

Inside this for loop, we have access to a variable called type that has various properties that describe the type we’re working on this on this iteration of the for loop.

Inside the for loop, we can begin generating our code:

class Encodable{{ type.name }}: NSObject, NSCoding {
    
    var {{ type.name | lowerFirstWord }}: {{ type.name }}?
    
    init({{ type.name | lowerFirstWord }}: {{ type.name }}?) {
        self.{{ type.name | lowerFirstWord }} = {{ type.name | lowerFirstWord }}
    }
	 
    required init?(coder decoder: NSCoder) {
        // ...
    }
    	    
    func encode(with encoder: NSCoder) {
        // ...
    }
}

Sourcery’s templates mix Swift code (regular code) with stencil code (meta code, code that writes code). Anything inside the double braces ({{ }}) will be printed out, so

class Encodable{{ type.name }}: NSObject, NSCoding

will output something like

class EncodableCoordinate: NSObject, NSCoding

Stencil and Sourcery also provide useful “filters”, like lowerFirstWord. That filter turns an upper-camel-case identifier into a lower-camel-case identifier. For example, it will convert DogHouse to dogHouse.

Thus, the line

var {{ type.name | lowerFirstWord }}: {{ type.name }}?

converts to

var coordinate: Coordinate?

which is exactly what we are after.

At this point, we can run our app, have the build phase generate our code, and take a look at the file AutoCodable.generated.swift file and ensure everything is generating correctly.

Next, let’s take a look at the init(coder:) function that we will have to generate. This is tougher. Let’s lay out the groundwork:

required init?(coder decoder: NSCoder) {
    {% for variable in type.storedVariables %}
    
    // ...
    
    {% endfor %})
    
    {{ type.name | lowerFirstWord }} = {{ type.name }}(
        {% for variable in type.storedVariables %}
        {{ variable.name }}: {{ variable.name }}{% if not forloop.last %},{% endif %}
        {% endfor %}
     )

}

We will loop through all of the variables in order to pull something useful out of the decoder. The last 4 lines here use an initializer to actually initialize the type from all the variables we will create. It will generate code something like:

coordinate = Coordinate(
    latitude: latitude,
    longitude: longitude
)

This corresponds to the memberwise initializer that Swift provides for structs. While working on this feature, I became worried that the user of this template could become confused if the memberwise initailizer disappeared or if they re-implemented it with the parameters in a different order. At the end of the post, we’ll take a look at a second Sourcery template for generating these initializers.

Because our model objects (“encodables”) can contain properties which are also encodables, we have to make sure to convert those to and from their encodable representations. For something like a coordinate, where the only values are two doubles (for latitude and longitude), we don’t need to do much. For more interesting objects, there are three cases of encodables (array, optional, and regular) to handle in a special way, so in total, we have 5 situations to handle: an encodable array, an encodable optional, a regular encodable, a regular optional, and a regular value.

required init?(coder decoder: NSCoder) {
    {% for variable in type.storedVariables %}
    {% if variable.typeName.name|hasPrefix:"[" %} // note, this doesn't support Array<T>, only `[T]`

    // handle arrays

    {% elif variable.isOptional and variable.type.implements.AutoCodable %}

    // handle encodable optionals

    {% elif variable.isOptional and not variable.type.implements.AutoCodable %}
    
    // handle regular optionals
    
    {% elif variable.type.implements.AutoCodable %}

    // handle regular encodables

    {% else %}
    
    // handle regular values

    {% endif %}
    
    {% endfor %}

}

This sets up our branching logic.

Next, let’s look at each of the 5 cases. First, arrays of encodables:

guard let encodable_{{ variable.name }} = decoder.decodeObject(forKey: "{{ variable.name }}") as? [Encodable{{ variable.typeName.name|replace:"[",""|replace:"]","" }}] else { return nil }
let {{ variable.name }} = encodable_{{ variable.name }}.flatMap({ $0.{{ variable.typeName.name|replace:"[",""|replace:"]",""| lowerFirstWord}} })

The first line turns decodes an array of encodables, and the second line converts the encodables (which represent the NSCoding wrappers) into the actual objects. The code that’s generated looks something like:

guard let encodable_images = decoder.decodeObject(forKey: "image") as? [EncodableImage] else { return nil }
let images = encodable_images.flatMap({ $0.image })

Frankly, the stencil code is hideous. Stencil doesn’t support things like assignment of processed data to new variables, so things like {{ variable.typeName.name|replace:"[",""|replace:"]","" }} (which extracts the type name from the array’s type name) can’t be factored out. Stencil is designed more for presentation and less for logic, so this omission is understandable, however, it does make the code uglier.

The astute reader will note that I used underscores in a variable name, which is not the typical Swift style. I did this purely out of laziness: I didn’t want to deal with correctly capitalizing the variable name. Ideally, no one will ever look at this code, and it will work transparently in the background.

Next up, optionals. Encodable optionals first:

let encodable_{{ variable.name }} = decoder.decodeObject(forKey: "{{ variable.name }}") as? Encodable{{ variable.unwrappedTypeName }}
let {{ variable.name }} = encodable_{{variable.name}}?.{{variable.name}}

which generates something like

let encodable_image = decoder.decodeObject(forKey: "image") as? EncodableImage
let image = encodable_image?.image

And regular optionals, for things like numbers:

let {{ variable.name }} = decoder.decodeObject(forKey: "{{ variable.name }}") as? {{ variable.unwrappedTypeName }}

And its generated code:

let imageCount = decode.decodeObject(forKey: "imageCount") as? Int

Next, regular encodables:

guard let encodable_{{ variable.name }} = decoder.decodeObject(forKey: "{{ variable.name }}") as? Encodable{{ variable.typeName }},
let {{ variable.name }} = encodable_{{ variable.name }}.{{ variable.name }} else { return nil }

which generates:

guard let encodable_images = decoder.decodeObject(forKey: "image") as? EncodableImage,
let images = encodable_image?.image else { return nil }

And finally regular values:

guard let {{ variable.name }} = decoder.decodeObject(forKey: "{{ variable.name }}") as? {{ variable.typeName }} else { return nil }

And its generated code:

guard let imageCount = decoder.decodeObject(forKey: "imageCount") as? Int else { return nil }

I won’t go into too much detail on these last few, since they work similarly to the ones above.

Next, let’s quickly look at the encode(coder:) method. Because NSCoding is designed for Objective-C and everything is “optional” in Objective-C, we don’t have to handle optionals any differently. This means the number of cases we have to deal with are minimized.

func encode(with encoder: NSCoder) {
    {% for variable in type.storedVariables %}
    {% if variable.typeName.name|hasPrefix:"[" %}
    
    // array

    {% elif variable.type.implements.AutoCodable %}
    
    // encodable
    
    {% else %}
    
    // normal value
    
    {% endif %}
    {% endfor %}
}

The array handling code looks like this:

let encoded_{{ variable.name }} = {{ type.name | lowerFirstWord }}?.{{ variable.name }}.map({ return Encodable{{ variable.typeName.name|replace:"[",""|replace:"]","" }}({{ variable.typeName.name|replace:"[",""|replace:"]",""| lowerFirstWord }}: $0) })
encoder.encode(encoded_{{ variable.name }}, forKey: "{{ variable.name }}")

The encodable handling code looks like this:

encoder.encode(Encodable{{ variable.unwrappedTypeName }}({{ variable.name | lowerFirstWord }}: {{ type.name | lowerFirstWord }}?.{{ variable.name }}), forKey: "{{ variable.name }}")

And the normal properties can be encoded like so:

encoder.encode({{ type.name | lowerFirstWord }}?.{{ variable.name }}, forKey: "{{ variable.name }}")

The stencil code is a bit complex, hard to read, and messy. However, the nice thing is that if you write this repetitive code once, it will generate a lot more repetitive code for you. For example, the whole template is 86 lines, and for all of our models, it generates about 500 lines of boilerplate.

One thing I was surprised to learn is that the code is surprisingly robust. We wrote this template in a day back in February. In the intervening 6 months, we haven’t edited this template at all. No modifications have been necessary to support any of the changes we made to our models since then.

The last thing we need to support a few protocol that show our system how to bridge between the encodables and the structs:

extension Encodable{{ type.name }}: Encoder {
    typealias Value = {{ type.name }}
    
    var value: {{ type.name }}? {
        return {{ type.name | lowerFirstWord }}
    }
}

extension {{ type.name }}: Archivable {
    typealias Encodable = Encodable{{ type.name }}
    
    var encoder: Encodable{{ type.name }} {
        return Encodable{{ type.name }}({{ type.name | lowerFirstWord }}: self)
    }
}

To learn more about the purpose of these extra conformances, you can read the original blog post.

Finally, let’s take a quick look at the Sourcery template for something we could call AutoInitializable. The approach is similiar to what we’ve looked at for AutoCodable, with one additional component. Because of Swift’s initialization rules, it only works with structs (classes in Swift can’t add new initializers in extensions).

{# this template currently only works with structs (no classes) #}
{% for type in types.implementing.AutoInitializable %}

extension {{ type.name }} {
{% if type.kind == "struct" and type.initializers.count == 0 %}
    // no initializer, since there is a free memberwise intializer that Swift gives us
{% else %}
    {{ type.accessLevel }} init({% for variable in type.storedVariables %}{{ variable.name }}: {{ variable.typeName }}{% if variable.annotations.initializerDefault %} = {{ variable.annotations.initializerDefault }}{% endif %}{% if not forloop.last %}, {% endif %}{% endfor %}) {
        {% for variable in type.storedVariables %}
        self.{{ variable.name }} = {{ variable.name }}
        {% endfor %}
    }
{% endif %}
}

{% endfor %}

I won’t belabor this with a line-by-line breakdown, but I will note that it takes advantage of a Sourcery feature called “annotations”. This lets you add additional information to a particular property. In this case, we had certain cases where some properties needed to have initializerDefault values (usually nil), so we were able to add support for that. A Sourcery annotation is declared like so:

// sourcery: initializerDefault = nil
var distance: Distance?

This post lays out one of the more involved uses for code generation. Sourcery gives you the building blocks you need to build complex templates like this one, and it removes the need to maintain onerous boilerplate manually. This particular template code was designed for our needs and may not suit every app. It will ultimately be rendered obsolete by Swift 4’s Codable. However, the example serves as case study for more complex Sourcery templates. They are flexible without any loss in robustness. I was initially worried that this meta-code would be brittle, breaking frequently, but in practice, these templates haven’t required a single change since they were first written, and they ensure that our encodable representations always stay perfectly up-to-date with any model changes.

One of the selling points of server-side Swift is that you get to use the same tools that you’re used to on the client. One of these tools is Grand Central Dispatch. Dispatch is one of the best asynchronous toolboxes in terms of its API and abstractions, and getting to use it for the Beacon server is an absolute pleasure.

While there’s a broader discussion to be had about actors in Swift in the future, spurred by Chris Lattner’s concurrency manifesto, and perhaps in the future, and some of the patterns for asynchronous workers will change, for now, Dispatch is the best tool that we have.

On the client, we rely on Dispatch for a few reasons. Key among them, and notably irrelevant on the server, we use Dispatch to get expensive work off the main thread to keep our UIs responsive. While the server does not have this need specifically, services with faster response times (under 250ms per request) are used more often than those that are slower. (Other uses of Dispatch, like synchronization of concurrent tasks and gating access to resources, is similarly valuable on both platforms.)

To make requests faster, a lot of nonessential work can be deferred until after the consumer’s request has been responded to. Examples of this are expensive calculations or external side effects, like sending email or push notifications. Further, some code should be executed on a regular basis: hourly or daily or weekly.

Dispatch is well-suited for these types of tasks, and in this post, we’ll discuss how Dispatch usage is similar relative to the client. My experience here is with the framework Vapor, though I suspect much of this advice holds true for other frameworks as well.

Your server app is long running. Some web frameworks tear down the whole process between requests, to clear out any old state. Vapor doesn’t work like this. While each request is responded to in a synchronous fashion, Vapor will respond to multiple requests at the same time. The same instance of the application handles these requests. This means that if you want something to happen, but don’t want to block returning a response for the current request, you can follow your intuition and use DispatchQueue.async to kick that block to another queue for execution, and return the response immediately.

A concrete example of this is firing off a push notification in reaction to some request the user makes: the user makes a new event and the user’s friends need to be notified. If you don’t use Dispatch for this, then the response to the user will be delayed by however long it takes to successfully send the push notification payload to an APNS server. In particular, if you have many push notifications to send, this can greatly delay the user’s request. By deferring this until after the user’s request is responded to, the request will return faster. Once the side effect is deferred, it can take as long it needs to without affect the user’s experience.

Lastly, sometimes you want to delay push notifications by a few seconds so that if the user deletes the resource in question, the user’s friends aren’t notified about an object that doesn’t exist. To accomplish this, you can swap async for asyncAfter, just as you would expect from your client-side experience.

You can’t use the main queue. The “main” queue is blocked, constantly spinning, in order to prevent the progam from ending. Unlike in iOS apps, there’s no real concept of a run loop, so the main thread has no way to execute blocks that are enqueued to it. Therefore, every time you want to async some code, you must dispatch it to a shared, concurrent .global() queue or to a queue of your own creation. Because there is no UI code, there’s no reason to prefer the main thread over any other thread.

Thread safety is still important. Vapor handles many requests at once, each on their own global queue. Any global, mutable data needs to be isolated behind some kind of synchronization pattern. While you can use Foundation’s locks, I find isolation queues to be an easier solution to use. They’re slightly more performant than locks, since they enable concurrent reads, and they work exactly the same way on the server as they do on iOS.

Semaphores are good for making async code synchronous. Other Swift server frameworks work differently, but Vapor expects the responses to requests to be synchronous. Therefore, there’s no sense in using code with completion blocks. APIs like URLSession’s dataTask(with:completionHandler:) can be made synchronous using semaphores:

extension URLSession {
    public func data(with request: URLRequest) throws -> (Data, HTTPURLResponse) {
        var error: Error?
        var result: (Data, HTTPURLResponse)?
        let semaphore = DispatchSemaphore(value: 0)

        self.dataTask(with: request, completionHandler: { data, response, innerError in
            if let data = data, let response = response as? HTTPURLResponse {
                result = (data, response)
            } else {
                error = innerError
            }
            semaphore.signal()
        }).resume()

        semaphore.wait()

        if let error = error {
            throw error
        } else if let result = result {
            return result
        } else {
            fatalError("Something went horribly wrong.")
        }
	}
}

This code kicks off a networking request and blocks the calling thread with semaphore.wait(). When the data task calls the completion block, the result or error is assigned, and we can call semaphore.signal(), which allows the code to continue, either returning a value or a throwing an error.

Dispatch timers can perform regularly scheduled work. For work that needs to occur on a regular basis, like database cleanup, maintenance, and events that need to happen at a particular time, you can create a dispatch timer.

let timer = DispatchSource.makeTimerSource()
timer.scheduleRepeating(deadline: .now(), interval: .seconds(60))

timer.setEventHandler(handler: {
	//fired every minute
})

timer.resume()

The only thing of note here is that, like on the client, this timer won’t retain itself, so you have to store it somewhere. Because it’s pretty easy to build your own behaviors on top of something like a Dispatch timer, I think we won’t see job libraries, like Rails’s ActiveJob, have quite the uptake in Swift that they have had in other environments. Nevertheless, I think it’s worth linking to the job/worker queue libraries I’ve found on GitHub:

Dispatch is a useful library with tons of awesome behaviors that can be built with its lower-level primitives. When setting out, I wasn’t sure how it would work in a Linux/server environment, and I’m pleased to report that working with it on the server is about as straightforward as you would want it to be. It’s a real delight to use, and it makes writing server applications that much easier.

This is a post I’ve been trying to write for a long time — literally years — and have struggled for want of the perfect example. I think I’ve finally found the one, courtesy of David James, Tim Vermeulen, Dave DeLong, and Erica Sadun.

Once upon a time, Erica came up with a way for constraints to install themselves. This code was eventually obviated by isActive in UIKit, but the code moved from Objective-C to Swift. It wasn’t perfect or particularly efficient, but it got the job done.

The following code comes from a rote Swift migration. It calculates the nearest common ancestor between two items in a tree of views. This was an early stab at this concept, abandoned after isActive was added.

Back when I worked at Rap Genius, we would often say the first cut is the deepest. Your first attempt at something, while it might not be the cleanest or most polished, involves the most work because it provides the superstructure for what you’re building. Erica’s version is that superstructure. It’s a solution and it works, but it’s ripe for cleaning up.

public extension UIView {

	// Return nearest common ancestor between two views
	public func nearestCommonAncestor(with otherView: UIView) -> UIView? {
		
		// Two equal views are each other's NCA
		guard self != otherView else { return self }
		
		// Compute superviews
		let mySuperviews = sequence(first: self.superview, next: { $0?.superview }).flatMap({ $0 })
		let theirSuperviews = sequence(first: otherView.superview, next: { $0?.superview }).flatMap({ $0 })	 
		
		// Check for direct ancestry
		guard !mySuperviews.contains(otherView)
			else { return otherView }
		guard !theirSuperviews.contains(self)
			else { return self }
		
		// Check for indirect ancestry
		for view in mySuperviews {
			guard !theirSuperviews.contains(view)
				else { return view }
		}
		
		// No shared ancestry
		return nil
	}
}

There’s a lot wrong with this code. It’s complex. There’s lots of cases to think about. It’s a simple piece of functionality, and yet there’s four guards and three different lookups. Simplifying this code will make it easier to read, understand, and maintain.

Perhaps you’re fine with this code in your codebase. The old saying goes, “if it ain’t broke, don’t fix it”. However, my experience has shown me that when there’s an inelegant algorithm like this, there’s a pearl in the center of it that wants to come out. Even a function this long is too hard to keep in your brain all at once. If you can’t understand it all, things slip through the cracks. I’m not confident that there aren’t any bugs in the above code; as a friend said, “every branch is a place for bugs to hide”. (This concept is known more academically as cyclomatic complexity.) And bugs or no bugs, with the power of retrospection, I can now see a few performance enhancements hiding in there, obscured by the current state of the code.

The refactoring process helps eliminate these potential bugs and expose these enhancements by iteratively driving the complex towards the simple. Reducing the algorithm down to its barest form also helps you see how it’s similar to other algorithms in your code base. These are all second-order effects, to be sure, but second-order effects pay off.

To kick off our refactoring, let’s look at the sequence(first:next:) function. Erica’s version started with self.superview, which is an optional value. This creates a sequence of optionals, which then forced Erica to flatMap them out. If we can remove this optionality from the sequence, we can remove the flatMap too. We changed the sequence to start from self instead (which isn’t optional), and added dropFirst() to remove that self:

let mySuperviews = sequence(first: self, next: { $0.superview }).flatMap({ $0 }).dropFirst()

Next, we killed the flatMap({ $0 }), because there are no nils to remove any more:

sequence(first: self, next: { $0.superview }).dropFirst()

This change led to this intermediate state, with plenty of code still left to trim:

public extension UIView {

	// Return nearest common ancestor between two views
	public func nearestCommonAncestor(with otherView: UIView) -> UIView? {
		
		// Two equal views are each other's NCA
		guard self != otherView else { return self }
		
		// Compute superviews
		let mySuperviews = sequence(first: self, next: { $0.superview }).dropFirst()
		let theirSuperviews = sequence(first: otherView, next: { $0.superview }).dropFirst()

		// Check for direct ancestry
		guard !mySuperviews.contains(otherView)
			else { return otherView }
		guard !theirSuperviews.contains(self)
			else { return self }

		// Check for indirect ancestry
		for view in mySuperviews {
			guard !theirSuperviews.contains(view)
				else { return view }
		}
		
		// No shared ancestry
		return nil
	}
}

At this point, we looked at the indirect ancestry component.

// Check for indirect ancestry
for view in mySuperviews {
	guard !theirSuperviews.contains(view)
		else { return view }
}

A for loop with an embedded test is a signal to use first(where:). The code simplified down to this, removing the loop and test:

if let view = mySuperviews.first(where: { theirSuperviews.contains($0) }) { return view }

Function references make this more elegant, readable, and clear:

if let view = mySuperviews.first(where: theirSuperviews.contains) { return view }

Our function now looks like this:

public extension UIView {

	// Return nearest common ancestor between two views
	public func nearestCommonAncestor(with otherView: UIView) -> UIView? {
		
		// Two equal views are each other's NCA
		guard self != otherView else { return self }

		// Compute superviews
		let mySuperviews = sequence(first: self, next: { $0.superview }).dropFirst()
		let theirSuperviews = sequence(first: otherView, next: { $0.superview }).dropFirst()
		
		// Check for direct ancestry
		guard !mySuperviews.contains(otherView)
			else { return otherView }
		guard !theirSuperviews.contains(self)
			else { return self }

		// Check for indirect ancestry
		if let view = mySuperviews.first(where: theirSuperviews.contains) { return view }
		if let view = theirSuperviews.first(where: mySuperviews.contains) { return view }
		
		// No shared ancestry
		return nil
	}
}

After this point, we stepped back and looked at the algorithm as a whole. We realized that if we include self and otherView in their respective superview sequences, the “direct ancestry” check and the “two equal views” check at the top would be completely subsumed by the first(where:) “indirect ancestry” checks. To perform this step, we first dropped the dropFirst():

sequence(first: self, next: { $0.superview })

And then we could kill the “direct ancestry” check:

// Check for direct ancestry
guard !mySuperviews.contains(otherView)
	else { return otherView }
guard !theirSuperviews.contains(self)
	else { return self }

And finally we could remove the first guard as well:

guard self != otherView else { return self }

After deleting them both, the function now looked like this:

public extension UIView {

	// Return nearest common ancestor between two views
	public func nearestCommonAncestor(with otherView: UIView) -> UIView? {
		
		// Compute superviews
		let mySuperviews = sequence(first: self, next: { $0.superview })
		let theirSuperviews = sequence(first: otherView, next: { $0.superview })
		
		if let view = mySuperviews.first(where: theirSuperviews.contains) { return view }
		if let view = theirSuperviews.first(where: mySuperviews.contains) { return view }
		
		// No shared ancestry
		return nil
	}
}

That was a major turning point in our understanding of the function. At this point, this code was starting to reveal its own internal structure. Each step clarifies the next potential refactoring to perform, to get closer to the heart of the function. For the next refactoring, Tim realized we could simplify the tail end of the function by applying a nil-coalescing operator:

return mySuperviews.first(where: theirSuperviews.contains) ?? theirSuperviews.first(where: mySuperviews.contains)

But here the first test before the nil-coalescing operator has covered all the views in both hierarchies. Because we’re looking for the first intersection between mySuperviews and theirSuperviews, there’s no reason to test both it and its opposite. We can drop everything after the ??:

public extension UIView {
	// Return nearest common ancestor between two views
	public func nearestCommonAncestor(with otherView: UIView) -> UIView? {
		let mySuperviews = sequence(first: self, next: { $0.superview })
		let theirSuperviews = sequence(first: otherView, next: { $0.superview })
		
		return mySuperviews.first(where: theirSuperviews.contains)
	}
}

The algorithm has revealed its beautiful internal symmetry now. Very clear intent, very clear algorithm, and each component is simple. It’s now more obvious how to tweak and modify this algorithm. For example,

  • If you don’t want the views self and otherView to be included in the calculation of ancestry, you can restore dropFirst() to the superview sequences.
  • If you want to know if the views have a common ancestor (rather than caring about which ancestor it is), you can replace the first(where:) with a contains(where:).
  • If you want to know all the common ancestors, you could replace the first(where:) with a filter(_:).

With the code in its original state, I couldn’t see before that these kinds of transformations were possible; now, they’re practically trivial.

From here, there are two potential routes.

First, there’s a UIView API for determining if one view is a descendant of another, which makes for a super readable solution:

extension UIView {
	// Return nearest common ancestor between two views
	public func nearestCommonAncestor(with other: View) -> View? {
		return sequence(first: self, next: { $0.superview })
			.first(where: { other.isDescendant(of: $0) })
	}
}

The second option is to explore performance. We noticed that theirSuperviews was only used for a contains check. If we wrap that sequence in a Set, existence lookup becomes O(1), and this whole algorithm gets blisteringly fast.

public extension UIView {
	// Return nearest common ancestor between two views
	public func nearestCommonAncestor(with otherView: UIView) -> UIView? {
		let mySuperviews = sequence(first: self, next: { $0.superview })
		let theirSuperviews = Set(sequence(first: otherView, next: { $0.superview }))
		return mySuperviews.first(where: theirSuperviews.contains)
	}
}

For view hierarchies that are pathologically deep (10,000 or so levels), this solution leaves the other one in the dust. Almost no view hierarchies contain that many layers, so this isn’t really a necessary optimization. However, if it were necessary, it would have been very hard to find without this refactoring process. Once we performed it, it become obvious what to tweak to speed things up.

Thomas Aquinas writes:

Properly speaking, truth resides in the intellect composing and dividing; and not in the senses; nor in the intellect knowing “what a thing is.”

This quote reflects the process of refactoring. If you’re doing it right, you don’t need to understand what the original code actually does. In the best of cases, you won’t even need to compile the code. You can operate on the code, composing and dividing, through a series of transformations that always leave the code in a correctly working state.

Perhaps you could have written the final version of this code from the very start. Perhaps it was obvious to you that this combination of APIs would yield the correct behavior in all cases. I don’t think I could have predicted that the original code would end up as an elegant one-line solution that handles all edge cases gracefully. I definitely couldn’t have predicted that there was a big performance optimization that changes this algorithm from O(n²) to O(n). Refactoring is an iterative process, and continual refinement reveals the code’s true essence.

Part of the promise of Swift is the ability to write simple, correct, and expressive code. Swift’s error system is no exception, and clever usage of it vastly improves the code on the server. Our app Beacon uses Vapor for its API. Vapor provides a lot of the fundamental components to building an API, but more importantly, it provides the extension points for adding things like good error handling yourself.

The crucial fact is that pretty much every function in your server app is marked as throws. At any point, you can throw an error, and that error will bubble all the way through any functions, through the response handler that you registered with the router, and through any registered middlewares.

Vapor typically handles errors by loading an HTML error page. Because the Beacon’s server component is a JSON API, we need some middleware that will translate an AbortError (Vapor’s error type, which includes a message and a status code) into usable JSON for the consumer. This middleware is pretty boilerplate-y, so I’ll drop it here without much comment.

public final class JSONErrorMiddleware: Middleware {
    	    
    public func respond(to request: Request, chainingTo next: Responder) throws -> Response {
        do {
            return try next.respond(to: request)
        } catch let error as AbortError {
            let response = Response(status: error.status)
            response.json = try JSON(node: [
                "error": true,
                "message": error.message,
                "code": error.code,
                "metadata": error.metadata,
            ])
            return response
        }
    }
}

In Vapor 1.5, you activate this middleware by adding it to the droplet, which is an object that represents your app.

droplet.middleware.append(JSONErrorMiddleware())

Now that we have a way to present errors, we can start exploring some useful errors. Most of the time when something on the server fails, that failure is represented by a nil where there shouldn’t be one. So, the very first thing I added was the unwrap() function:

struct NilError: Error { }

extension Optional {
    func unwrap() throws -> Wrapped {
        guard let result = self else { throw NilError() }
        return result
    }
}

This function enables you to completely fail the request whenever a value is nil and you don’t want it to be. For example, let’s say you want to find an Event by some id.

let event = Event.find(id)

Unsurprisingly, the type of event is Optional<Event>. Because an event with the given ID might not exist when you call that function, it has to return an optional. However, sometimes this doesn’t make for the best code. For example, in Beacon, if you try to attend an event, there’s no meaningful work we can do if that event doesn’t exist. So, to handle this case, I call unwrap() on the value returned from that function:

let event = Event.find(id).unwrap()

The type of event is now Event, and if the event doesn’t exist, function will end early, and bubble the error up until it hits the aforementioned JSONErrorMiddleware, ultimately resulting in error JSON for our user.

The problem with unwrap() is that it lacks any context. What failed to unwrap? If this were Ruby or Java, we’d at least have a stack trace and we could figure out what series of function calls led to our error. This is Swift, however, and we don’t have that. The most we can really do is capture the file and line of the faulty unwrap, which I’ve done in this version of NilError.

In addition, because there’s no context, Vapor doesn’t have a way to figure out what status code to use. You’ll notice that our JSONErrorMiddleware pattern matches on the AbortError protocol only. What happens to other errors? They’re wrapped in AbortError conformant-objects, but the status code is assumed to be 500. This isn’t ideal. While unwrap() works great for quickly getting stuff going, it quickly begins to fall apart once your clients start expecting correct status codes and useful error messages. To this end, we’ll be exploring a few useful custom errors that we built for this project.

Missing Resources

Let’s tackle our missing object first. This request should probably 404, especially if our ID comes from a URL parameter. Making errors in Swift is really easy:

struct ModelNotFoundError: AbortError {
    
    let status = Status.notFound
    	    
    var code: Int {
        return status.statusCode
    }
    	    
    let message: String
    
    public init<T>(type: T) {
        self.message = "\(typeName) could not be found."
    }
}

In future examples, I’ll leave out the computed code property, since that will always just forward the statusCode of the status.

Once we have our ModelNotFoundError, we can guard and throw with it.

guard let event = Event.find(id) else {
	throw ModelNotFoundError(type: Event)
}

But this is kind of annoying to do every time we want to ensure that a model is found. To solve that, we package this code up into an extension on every Entity:

extension Entity {
	static func findOr404(_ id: Node) throws -> Self {
		guard let result = self.find(id) else {
			throw ModelNotFoundError(type: Self.self)
		}
		return result
	}
}

And now, at the call site, our code is simple and nice:

let event = try Event.findOr404(id)

Leveraging native errors on the server yields both more correctness (in status codes and accurate messages) and more expressiveness.

Authentication

Our API and others use require authenticating the user so that some action can be performed on their behalf. To execute this cleanly, we use a middleware to fetch to the user from some auth token that the client passes us, and save that user data into the request object. (Vapor includes a handy dictionary on each Request called storage that you can use to store any additional data of your own.) (Also, Vapor includes some authentication and session handling components, but it was easier to write this than to try to figure out how to use Vapor’s built-in thing.)

final class CurrentSession {

	init(user: User? = nil) {
		self.user = user
	}
    
	var user: User?
    
	@discardableResult
	public func ensureUser() throws -> User {
		return user.unwrap()
	}
}

Every request will provide a Session object like the one above. If you want to ensure that a user has been authenticated (and want to work with that user), you can call:

let currentUser = try request.session.ensureUser()

However, this has the same problem as our previous code. If the user isn’t correctly authed, the consumer of this API will see a 500 with a meaningless error about nil objects, instead of a 401 Unauthorized code and a nice error message. We’re going to need another custom error.

struct AuthorizationError: AbortError {
	let status = Status.unauthorized

	var message = "Invalid credentials."
}

Vapor actually has a shorthand for this kind of simple error:

Abort.custom(status: .unauthorized, message: "Invalid credentials.")

Which I used until I needed the error to be its own object, for reasons that will become apparent later.

Our function ensureUser() now becomes:

@discardableResult
public func ensureUser() throws -> User {
	guard let user = user else {
		throw AuthorizationError()
	}
	return user
}

Bad JSON

Vapor’s JSON handling leaves much to be desired. Let’s say you want a string from the JSON body that’s keyed under the name “title”. Look at all these question marks:

let title = request.json?["title"]?.string

At the end of this chain, of course, title is an Optional<String>. Even throwing an unwrap() at the end of this chain doesn’t solve our problem: because of Swift’s optional chaining precedence rules, it will only unwrap the last component of the chain, .string. We can solve this in two ways. First, by wrapping the whole expression in parentheses:

let title = try (request.json?["title"]?.string).unwrap()

or unwrapping at each step:

let title = try request.json.unwrap()["title"].unwrap().string.unwrap()

Needless to say, this is horrible. Each unwrap represents a different error: the first represents a missing application/json Content-Type (or malformed data), the second, the presence of the key, and the third, the expectation of the key’s type. All that data is thrown away with unwrap(). Ideally, our API would have a different error message for each error.

enum JSONError: AbortError {

	var status: Status {
		return .badRequest
	}
	
	case jsonMissing
	case missingKey(keyName: String)
	case typeMismatch(keyName: String, expectedType: String, actualType: String)
}

These cases represent the three different errors from above. We need to add a function to generate a message depending on the case, but that’s really all this need. We have errors that are a lot more expressive, and ones that help the client debug common errors (like forgetting a Content-Type).

These errors, combined with NiceJSON (which will hopefully be the topic of its own post), result in code like this:

let title: String = try request.niceJSON.fetch("title")

Much easier on the eyes. title is also usually an instance variable (of a command) with a pre-set type, so the : String required for type inference can be omitted as well.

By making the “correct way” to write code the same as the “nice way” to write code, you never have to make a painful trade-off between helpful error messages or type safety, and short easy-to-read code.

Externally Visible Errors

By default, Vapor will wrap an error that fails into an AbortError. However, many (most!) errors reveal implementation details that users shouldn’t see. For example, the PostgreSQL adapter’s errors reveal details about your choice of database and the structure of your tables. Even NilError includes the file and line of the error, which reveals that the server is built on Swift and is therefore vulnerable to attacks targeted at Swift.

In order to hide some errors and allow others to make it through the user, I made a new protocol.

public protocol ExternallyVisibleError: Error {
    
    var status: Status { get }
    
    var externalMessage: String { get }
}

Notice that ExternallyVisibleError doesn’t inherit from AbortError. Once you conform your AbortError to this protocol, you have to provide one more property, externalMessage, which is the message that will be shown to users.

Once that’s done, we need a quick modification to our JSONErrorMiddleware to hide the details of the error if it’s not an ExternallyVisibleError:

public func respond(to request: Request, chainingTo next: Responder) throws -> Response {
    do {
        return try next.respond(to: request)
    } catch let error as ExternallyVisibleError {
        let response = Response(status: error.status)
        response.json = try JSON(node: [
            "error": true,
            "message": error.externalMessage,
            "code": error.status.statusCode,
        ])
        return response
    } catch let error as AbortError {
        let response = Response(status: error.status)
        response.json = try JSON(node: [
            "error": true,
            "message": "There was an error processing this request.",
            "code": error.code,
        ])
        return response
    }
}

I also added some code that would send down the AbortError’s message as long as the environment wasn’t .production.

Swift’s errors are a powerful tool that can store additional data, metadata, and types. A few simple extensions to Vapor’s built-in types will enable you to write better code along a number of axes. For me, the ability to write terse, expressive, and correct code is the promise that Swift offered from the beginning, and this compact is maintained on the server as much as it is on the client.

Beacon is built with Swift on the server. Since we have all of the niceties of Swift in this new environment, we can use our knowledge and experience from building iOS app to build efficient server applications. Today, we’ll look at two examples of working with sequences on the server to achieve efficiency and performance.

Over the network

For its social graph, Beacon needs to find your mutual Twitter followers — that is, the people you follow that follow you back. There’s no Twitter API for this, so we have to get the list of follower IDs and the list of following IDs, and intersect them. The Twitter API batches these IDs into groups of 5,000. While people rarely follow more than 5,000 people, some users on Beacon have a few hundred thousand Twitter followers, so these will have to be batched. Because of these contraints, this problem provides a pretty interesting case study for advanced sequence usage.

We do this on the server instead of the client, because there will be a lot of requests to the Twitter API, and it doesn’t make much sense to perform those on a user’s precarious cellular connection. For our backend, we use the Vapor framework, and Vapor’s request handling is completely synchronous. Because of this, there’s no sense in using completion blocks for network requests. You can just return the result of the network request as the result of your function (and throw if anything goes wrong). For an example, let’s fetch the IDs of the first 5,000 people that someone follows:

let following = try client.send(request: TwitterFollowingRequest())

To perform the batching, the Twitter API uses the concept of cursors. To get the first batch, you can leave off the cursor, or pass -1. Each request returns a new next_cursor, which you give back to Twitter when you want the next batch. This concept of cursors fits nicely into Swift’s free function sequence(state:next:). Let’s examine this function’s signature:

func sequence<T, State>(state: State, next: @escaping (inout State) -> T?) -> UnfoldSequence<T, State>

This function is generic over two types: T and State. We can tell from the signature that we need provide an initial State as a parameter, and we also provide a closure that takes an inout State and returns an optional T. inout means we can mutate the state, so this is how we update the state for the next iteration of the sequence. The T that we return each time will form our sequence. Returning nil instead of some T ends the sequence.

Because the Fibonacci sequence is the gold standard for stateful sequences, let’s take a look at using sequence(state:next:) to create a Fibonacci sequence:

let fibonacci = sequence(state: (1, 1), next: { (state: inout (Int, Int)) -> Int? in
    let next = state.0 + state.1
    state = (state.1, next)
    return next
})

The state in this case has type (Int, Int) and represents the last two numbers in the sequence. First, we figure out the next number by adding the two elements in the tuple together; then, we update the state variable with the new last two values; finally, we return the next element in the sequence.

(Note that this sequence never returns nil, so it never terminates. It is lazy, however, so none of this code is actually evaluated until you ask for some elements. You can use .prefix(n) to limit to the first n values.)

To build our sequence of Twitter IDs, we start with the state "-1", and build our sequence from there.

let lazyFollowerIDs = sequence(state: "-1", next: { (state) -> [Int]? in

})

We need to send the request in this block, and return the IDs from the result of the request. The request itself looks a lot like the TwitterFollowingRequest from above, except it’s now for followers instead.

let lazyFollowerIDs = sequence(state: "-1", next: { (state) -> [Int]? in
    let result = try? self.client.send(request: TwitterFollowersRequest(cursor: state))
    return result?.ids
})

Right now, this request never updates its state, so it fetches the same page over and over again. Let’s fix that.

let lazyFollowerIDs = sequence(state: "-1", next: { (state) -> [Int]? in
    let result = try? self.client.send(request: TwitterFollowersRequest(cursor: state))
    state = result?.nextCursor ?? "0"
    return result?.ids
})

For the last page, Twitter will return "0" for the next_cursor, so we can use that for our default value if the request fails. (If the request fails, result?.ids will also be nil, so the sequence will end anyway.)

Lastly, let’s put a guard in place to catch the case when Twitter has shown us the last page.

let lazyFollowerIDs = sequence(state: "-1", next: { (state) -> [Int]? in
    guard state != "0" else { return nil }
    let result = try? self.client.send(request: TwitterFollowersRequest(cursor: state))
    state = result?.nextCursor ?? "0"
    return result?.ids
})

(If we added a little more error handling here, it would look almost identical to the actual code that Beacon uses.)

This sequence is getting close. It’s already lazy, like our Fibonacci sequence, so it won’t fetch the second batch of 5,000 items until the 5,001st element is requested. It needs one more big thing: it’s not actually a sequence of IDs yet. It’s still a sequence of arrays of IDs. We need to flatten this into one big sequence. For this, Swift has a function called joined() that joins a sequence of sequences into a big sequence. This function (mercifully) preserves laziness, so if the sequence was lazy before, it’ll stay lazy. All we have to do is add .joined() to the end of our expression.

To get our mutual follows from this lazyFollowerIDs sequence, we need something to intersect the followers and the following. To make this operation efficient, let’s turn the following IDs into a set. This will make contains lookup really fast:

let followingIDSet = Set(following.ids)

We make sure to filter over the lazyFollowerIDs since that sequence is lazy and we’d like to iterate over it only once.

let mutuals = lazyFollowerIDs.filter({ id in followingIDSet.contains(id) })

This reads “keep only the elements from lazyFollowerIDs that can be be found in followingIDSet”. Apply a little syntactic sugar magic to this, and you end up with a pretty terse statement:

let mutuals = lazyFollowerIDs.filter(followingIDSet.contains)

Off the disk

A similar technique can be used for handling batches of items from the database.

Vapor’s ORM is called Fluent. In Fluent, all queries go through the Query type, which is type parameterized on T, your entity, e.g, User. Queries are chainable objects, and you can call methods like filter and sort on them to refine them. When you’re done refining them, you can call methods like first(), all() , or count() to actually execute the Query.

While Fluent doesn’t have the ability to fetch in batches, its interface allows us to build this functionality easily, and Swift’s lazy sequence mechanics let us build it efficiently.

We know we’ll need a function on every Query. We don’t know what kind of Sequence we’ll be returning, but we’ll use Sequence<T> as a placeholder for now.

extension Query {
	func inBatches(of batchSize: Int) throws -> Sequence<T> {
		
	}
}

First, we need to know how many items match our query, so we can tell how many batches we’ll be fetching. Because the object we’re inside already represents the query that we’re going to be fetching with, and it already has all the relevant filters and joins, we can just call count() on self, and get the number of objects that match the query.

extension Query {
	func inBatches(of batchSize: Int) throws -> Sequence<T> {
		let count = try self.count()
		
	}
}

Once we have the count, we can use Swift’s stride(from:to:by:) to build a sequence that will step from 0 to our count with a stride of our batchSize.

extension Query {
	func inBatches(of batchSize: Int) throws -> Sequence<T> {
		let count = try self.count()
		stride(from: 0, to: self.count(), by: batchSize)
		
	}
}

Next, we want to transform each step of this stride (which represents one batch) into a set of the objects in question.

extension Query {
	func inBatches(of batchSize: Int) throws -> Sequence<T> {
		let count = try self.count()
		stride(from: 0, to: self.count(), by: batchSize)
			.map({ offset in
				return (try? self.limit(batchSize, withOffset: offset).all()) ?? []
			})
	}
}

Because .all() is a throwing function, we need to handle its error somehow. This will be a lazy sequence, so the map block will get stored and executed later. It is @escaping. This means that we can’t just throw, because we can’t guarantee that we’d be in a position to catch that error. Therefore, we just discard the error and return an empty array if it fails.

If we try to execute this as-is, the map will run instantly and fetch all of our batches at once. Not ideal. We have to add a .lazy to our chain to ensure that that each fetch doesn’t happen until an item from that batch is requested.

extension Query {
	func inBatches(of batchSize: Int) throws -> Sequence<T> {
		let count = try self.count()
		stride(from: 0, to: self.count(), by: batchSize)
			.lazy
			.map({ offset in
				return (try? self.limit(batchSize, withOffset: offset).all()) ?? []
			})
	}
}

The last step here, like the Twitter example, is to call .joined() to turn our lazy sequence of arrays into one big lazy sequence.

extension Query {
	func inBatches(of batchSize: Int) throws -> Sequence<T> {
		let count = try self.count()
		return stride(from: 0, to: self.count(), by: batchSize)
			.lazy
			.map({ offset in
				return (try? self.limit(batchSize, withOffset: offset).all()) ?? []
			})
			.joined()
	}
}

When we run this code, we see that the our big Sequence chain returns a LazySequence<FlattenSequence<LazyMapSequence<StrideTo<Int>, [T]>>>. This type is absurd. We can see all the components of our sequence chain in there, but we actually don’t care about those implementation details. It would be great if we could just erase the type and be left with something simple. This technique is called a type erasing and it will hide these details. AnySequence is a type eraser that the Swift standard library provides for this exact purpose. AnySequence also will become our return type.

extension Query {
    func inBatches(of batchSize: Int) throws -> AnySequence<T> {
		let count = try self.count()
        return AnySequence(stride(from: 0, to: count, by: batchSize)
            .lazy
            .map({ (offset) -> [T] in
                return (try? self.limit(batchSize, withOffset: offset).all()) ?? []
            })
            .joined())
    }
}

We can now write the code we want at the callsite:

try User.query().sort("id", .ascending)
	.inBatches(of: 20)
	.forEach({ user in
		//do something with user
	})

This is reminiscent of Ruby’s find_in_batches or the property fetchBatchSize on NSFetchRequest, which returns a very similar lazy NSArray using the NSArray class cluster.

This is not the first time I’ve said this, but Swift’s sequence handling is exceptionally robust and fun to work with. Understanding the basics of Swift’s sequences enable you to compose those solutions to tackle bigger and more interesting problems.

When working with Swift on the server, most of the routing frameworks work by associating a route with a given closure. When we wrote Beacon, we chose the Vapor framework, which works like this. You can see this in action in the test example on their home page:

import Vapor

let droplet = try Droplet()

droplet.get("hello") { req in
    return "Hello, world."
}

try droplet.run()

Once you run this code, visiting localhost:8080/hello will display the text “Hello, world.”.

Sometimes, you also want to return a special HTTP code to signal to consumers of the API that a special action happened. Take this example endpoint:

droplet.post("devices", handler: { request in
	let apnsToken: String = try request.niceJSON.fetch("apnsToken")
	let user = try request.session.ensureUser()
    
	var device = try Device(apnsToken: apnsToken, userID: user.id.unwrap())
	try device.save()
	return try device.makeJSON()
})

(I would like to detail what goes into niceJSON in a future blog post, but for now, ignore it.)

This is a perfectly fine request and is similar to code from the Beacon app. There is one problem: Vapor will assume a status code of 200 when you return objects like a string (in the first example in this blog post) or JSON (in the second example). However, this is a POST request and a new Device resource is being created, so it should return the HTTP status code “201 Created”. To do this, you have to create a full response object, like so:

let response = Response(status: .created)
response.json = try device.makeJSON()
return response

which is a bit annoying to have to do for every creation request.

Lastly, endpoints will often have side effects. Especially with apps written in Rails, managing and testing these is really hard, and much ink has been spilled in the Rails community about it. If signing up needs to send out a registration email, how do you stub that while still testing the rest of the logic? It’s a hard thing to do, and if everything is in one big function, it’s even harder. In Beacon’s case, we don’t have don’t have many emails to send, but we do have a lot of push notifications. Managing those side effects is important.

Generally speaking, this style of routing, where you use a closure for each route, has been used in frameworks like Flask, Sinatra, and Express. It makes for a pretty great demo, but a project in practice often has complicated endpoints, and putting everything in one big function doesn’t scale.

Going even further, the Rails style of having a giant controller which serves as a namespace for vaguely related methods for each endpoint is borderline offensive. I think we can do better than both of these. (If you want to dig into Ruby server architecture, I’ve taken a few ideas from the Trailblazer project.)

Basically, I want a better abstraction for responding to incoming requests. To this end, I’ve started using an object that I call a Command to encapsulate the work that an endpoint needs to do.

The Command pattern starts with a protocol:

public protocol Command {

	init(request: Request, droplet: Droplet) throws
    
	var status: Status { get }

	func execute() throws -> JSON
	
}

extension Command: ResponseRepresentable {
    
	public func makeResponse() throws -> Response {
		let response = Response(status: self.status)
		response.json = try execute()
		return response
	}
    
}

We’ll add more stuff to it as we go, but this is the basic shell of the Command protocol. You can see see just from the basics of the protocol how this pattern is meant to be used. Let’s rewrite the “register device” endpoint with this pattern.

droplet.post("devices", handler: { request in
	return RegisterDeviceCommand(request: request, droplet: droplet)
})

Because the command is ResponseRepresentable, Vapor accepts it as a valid result from the handler block for the route. It will automatically call makeResponse() on the Command and return that Response to the consumer of the API.

public final class RegisterDeviceCommand: Command {

	let apnsToken: String
	let user: User

	public init(request: Request, droplet: Droplet) throws {
		self.apnsToken = try request.niceJSON.fetch("apnsToken")
		self.user = try request.session.ensureUser()
	}

	public let status = Status.created

	public func execute() throws -> JSON {
		var device = try Device(apnsToken: apnsToken, userID: user.id.unwrap())
		try device.save()
		return try device.makeJSON()
	}
}

There are a few advantages conferred by this pattern already.

  1. Maybe the major appeal of using a language like Swift for the server is to take advantage of things like optionals (and more pertinently, their absence) to be able to define the absolute requirements for a request to successfully complete. Because apnsToken and user are non-optional, this file will not compile if the init function ends without setting all of those values.
  2. The status code is applied in a nice declarative way.
  3. Initialization is separate from execution. You can write a test that checks to that the initialization of the object (e.g., the extraction of the properties from the request) that is completely separate from the test that checks that the actual save() works correctly.
  4. As a side benefit, using this pattern makes it easy to put each Command into its own file.

There are two more important components to add to a Command like this. First, validation. We’ll add func validate() throws to the Command protocol and give it a default implementation that does nothing. It’ll also be added to the makeResponse() function, before execute():

public func makeResponse() throws -> Response {
	let response = Response(status: self.status)
	try validate()
	response.json = try execute()
	return response
}

A typical validate() function might look like this (this comes from Beacon’s AttendEventCommand):

public func validate() throws {
	if attendees.contains(where: { $0.userID == user.id }) {
		throw ValidationError(message: "You can't join an event you've already joined.")
	}
	if attendees.count >= event.attendanceLimit {
		throw ValidationError(message: "This event is at capacity.")
	}
	if user.id == event.organizer.id {
		throw ValidationError(message: "You can't join an event you're organizing.")
	}
}

Easy to read, keeps all validations localized, and very testable as well. While you can construct your Request and Droplet objects and pass them to the prescribed initializer for the Command, you’re not obligated to. Because each Command is your own object, you can write an initializer that accepts fully fledged User, Event, etc objects and you don’t have to muck about with manually constructing Request objects for testing unless you’re specifically testing the initialization of the Command.

The last component that a Command needs is the ability to execute side effects. Side effects are simple:

public protocol SideEffect {
	func perform() throws
}

I added a property to the Command protocol that lists the SideEffect-conforming objects to perform once the command’s execution is done.

var sideEffects: [SideEffect] { get }

And finally, the side effects have to be added to the makeResponse() function:

public func makeResponse() throws -> Response {
	let response = Response(status: self.status)
	try validate()
	response.json = try execute()
	try sideEffects.forEach({ try $0.perform() })
	return response
}

(In a future version of this code, side effects may end up being performed asynchronously, i.e., not blocking the response being sent back to the user, but currently they’re just performed synchronously.) The primary reason to decouple side effects from the rest of the Command is to enable testing. You can create the Command and execute() it, without having to stub out the side effects, because they will never get fired.

The Command pattern is a simple abstraction, but it enables testing and correctness, and frankly, it’s pleasant to use. You can find the complete protocol in this gist. I don’t knock Vapor for not including an abstraction like this: Vapor, like the other Swift on the server frameworks, is designed to be simple and and that simplicity allows you to bring abstractions to your own taste.

There are a few more blog posts coming on server-side Swift, as well as a few more in the Coordinator series. Beacon and WWDC have kept me busy, but rest assured! More posts are coming.