Xamarin Show: Realm Mobile Database & Platform

This week, James is joined by friend of the show Adam Fish, Product Manager at Realm, who introduces us to the cross-platform mobile reactive database, Realm for .NET. Realm offers developers a crazy fast, mobile optimized, easy to use, open source, and totally free database. Adam walks us through setting up your first Realm database and then expands on it to create an offline-first, real-time, and reactive mobile app leveraging the Realm Mobile Platform. You don’t want to miss this episode!

---

Intro (0:00)

My name’s Adam Fish, I’m the Senior Director of Product at Realm. I’ve been with the company for about two years now. Realm is a database company, and is about five years old. It got started as a typical Silicon Valley start-up. The founders are Danish and came over from Copenhagen, raised some capital, and built the database from scratch.

During the first three years, it was a high-speed, in-memory, server-based database. We quickly realized, with its unique capabilities coupled with the growth of mobile, that there was a great opportunity to basically build a replacement for SQLite or any of the derivatives. Two years ago, we launched it for iOS, and since then we’ve added Android, Xamarin, .Net support, and React Native. It’s really designed to be a cross-platform, and easy to use mobile database.

What is Realm? (2:19)

It’s pretty common to forget that the only database that anyone uses on mobile is SQLite. There’s always different wrappers that people put around it, Core Data being a major one for the iOS platform, but at the core, it’s SQLite.

With Realm, the biggest argument to build something new was that at the end of the day, when you’re building mobile apps, you’re working with objects. There’s always this mismatch between the object layer and the SQL layer. A lot of effort and Core Data goes into trying to manage that, with a ORM, but you’ve always got some level of mismatch than can happen. With Realm, we decided to build it from the ground up as something new.

At the core level it is a table interface, but the actual columns can link to other tables, so it inherently supports links. You don’t have that actual mismatch. We’re able to then provide an object interface in whatever the native language is that you’re using for the platform, and it just becomes a very fluid development experience.

Coupled with that, there’s been a real focus on performance. It’s a column-based database, so it can have really high performance and it uses memory mapping. You’re basically interacting with the data directly on disk and there’s no in-memory copying that’s happening.

Before I worked at Realm, I was working on my own start-up, we had an iOS app, and we were spending an inordinate amount of time fighting with Core Data; your typical threading issues and things that you’ll run into. When it launched, it was the simplicity of Realm that attracted me. I remember going to the website, and I saw the code snippets, and thought, “Aw, yes, that makes a ton of sense.”

The moment you start playing with it and you realize that not only is the interface really easy, but I’m not having to pay any performance penalty for that ease of use. It just became a no-brainer. We re-wrote the whole app using Realm. We started to do some really cool stuff, and contributed some open source projects that use Realm. I became an avid user, and the company ended up saying, “Why don’t you come and join us?”

Since then, speaking with developers, they echo the same thing. It’s the ease of use and performance. It’s the two that come hand in hand that really sells it.

Every month, there’s more Fortune 1000, Fortune 500 companies, and major brands that are moving to Realm and using it in their applications on all the different platforms. It has really taken off. It is fully open source.

During the first three years of the company, when it was more server-oriented, the core database was written in C++. When we first launched, just the bindings that wrapped that C++ layer into the native language of the platform were open source. Last summer, we open sourced the core. The whole stack of the actual database is fully open source on GitHub. Coupled with the open source nature and the fact that it’s been five years since it’s been built, we think it really is battle-tested at this point.

I have to hand it to a lot of the developers on our team, I became a better developer just watching how the Cocoa team at Realm had actually built that version of it. With a lot of the Objective-C run time and stuff, they do some really amazing work. Being able to watch that in real time, it’s amazing as a developer.

Realm and Cross-Platform (7:48)

The easiest place to get started with Realm is realm.io, that’s the number one place. You can just jump right in. We try to get you into the code, right on the homepage, so you can see a little bit about how you would use the database and some of the newer features that we recently released. If you go over to the docs, you dive into the platform that you’re looking for. If you look on the Xamarin docs, there’s a quick getting started. The package is available through NuGet.

The code is on GitHub. There is Xamarin iOS, Xamarin Android, and Win 32 support. We’re trying to work on spreading that over time to all the other platforms.

While there are some edges of the SDKs that are platform specific, the general concept of how you work with Realm objects and write transactions, is all very similar. Someone that is fluent in .Net could look at a Java code snippet and immediately adjust it.

Sample App(9:45)

There is a sample app that we include in the repo called Quick Journal. It’s basically a task tracker app. The first thing really to dive into is that looking at the data model of this application, we are creating an object called a JournalEntry. That’s what’s going to be represented in the table.

When you work with Realm, this is kind of like the key starting point of understanding how Realm works. You sub-class a Realm object, and that sub-class is actually the definition of your data model or schema. With the JournalEntry, we’ve got a string property for Title, another for BodyText, and then we’re demonstrating linking to another object, where we have another Realm object called EntryMetadata.

This is the nice part about how Realm works under the hood; at the core level those links are represented just like a reference. What’s really cool is that if you actually build an app that has a really complex object graph, when you query and you’re working with these Realm objects, you don’t actually have to pull all those objects into memory. You can actually traverse the object graph lazily. It does it for you. As you go to new objects, you basically don’t have to use as much memory. There’s no foreign key look-ups or anything. It is just working with almost totally native objects.

If you create a Realm JournalEntry object initially, its state is unmanaged. It’s basically just an in-memory object. It becomes managed when you actually add it to the Realm. We can show what that looks like, but that’s what isManaged is demonstrating.

The Realm is actually a single file. You can actually introspect the file. We’ve got a little browser application. It’s a single Realm file that is the database, so when you take an unmanaged object, and you add it, it’s going to write that and persist it on disk as well.

Transactions (13:41)

The view looks like a standard XAML. The add entry command is what we’re binding to. In the view model there is nothing Realm specific in that. In the application, if I click “add,” I can just give my journal entry an ID. If we look at the view model code, when I actually am adding that journal entry into the Realm, this is what it looks like.

private void AddEntry()
        {
            var transaction = _realm.BeginWrite();
            var entry = _realm.Add(new JournalEntry
            {
                Metadata = new EntryMetadata
                {
                    Date = DateTimeOffset.Now,
                    Author = AuthorName
                }
            });

            var page = new JournalEntryDetailsPage(new JournalEntryDetailsViewModel(entry, transaction));

            Navigation.PushAsync(page);
        }

The key aspect of working with Realm objects is that every interaction or write that you’re doing with Realm, or any change to the object, needs to be done in a write transaction. There is a very big secondary benefit of having this transaction-based model, which is you know the database is basically acid compliant, and it also is crash-safe.

Acid compliant means that you have a stable view of the database no matter which thread you’re on. It’s just basically a number of different properties the database has that prevents it from having corruption.

The database has what’s called a multi-version concurrency control system. Basically, what happens is the database in itself is essentially just a B-tree, if you want to visualize it. When you create a new transaction, there’s a new root of the tree that’s added, but any other thread that’s doing reads, because the writes are not blocked by reads, continues to look at all the data that’s in the previous root. You can add all your changes in that transaction, and then it’s actually going to write it down to the disk, and call an F-sync command, so that it is actually fully on the disk.

Only after that has happened, do we advance to the new root of the tree. The key point is that, if suddenly your phone died, or something happened and that other transaction was partially complete, the original root of the tree is still there, so you’re never going to get corrupt data. It’s a little in the weeds, but using write transactions actually is very beneficial in terms of the durability of the database itself.

The _realm is how you actually interact with the database itself. The thing with Realm is that it is all accessors, as I would call it. Whether you’re working with the database as a whole, which is what this Realm.GetInstance is,

_realm = Realm.GetInstance();

or you’re working with any sort of Realm object that you’ve gotten from the database, all of those are thread-specific. If I was on a background thread, I would also call Realm.GetInstance, and I would have an object specific for that thread. You never want to pass Realm objects across threads.

We actually do offer the capabilities to do that, so you can. There are ways to pass objects across threads, but you can’t just take an object and directly move it across.

When you actually commit the transaction, and it finishes, then it’s written to the disk.

The transaction gives you the, in the acid, the atomicity of being able to say that these two changes need to always be applied together. That’s another benefit of having a transaction system. We would recommend to use larger transactions than to do a bunch of really small ones. It depends on the circumstance. If you’re looping over, we’d say put the loop in the transaction itself.

Querying (20:16)

It’s pretty straightforward interaction in this application. There’s a lot of other things that are more hidden here, because this is a forms based application. The querying that’s happening in terms of reading the data is a little bit more hidden from view.

On the JournalEntryDetails page, when you’re typing, you are just writing to the Realm object, setting it right on the property itself. Then you save, and this is where the commit happens.

You created the entry, and you passed that entry to this page. You’re doing live data binding to that entry, and then when you hit save, then it commits it. You’re not setting any properties, you’re just typing to it, and it just knows.

That’s all from the binding itself. We tap into that automatically. That’s going to be setting it for you, and so as you commit it, that’s when it’s actually going to hit the disk. When you’re using Xamarin forms, the data binding is plugged into automatically.

We actually support a number of different abilities to listen for changes. If you’re not using a form-based application, we have a number of different APIs where you can basically query the database.

You could query all the person objects in the Realm, and then, whenever that query changes, it tells you. We can even pass in a changes object that’ll say, “Here are the ones that have been inserted, deleted, or modified.”

If you’re working with an iOS or Android app directly, you can pass this into list view or a UI table view to do the animations, if you’re doing this more manually. We also support object notifications, so you can get when properties specifically change.

This is what’s beneath the hood with Xamarin forms, which is doing this automatically for you, but if you’re not using that, you can still tap into that same sort of reactive architecture. Once you’re done completing the transaction, you commit it, and there’s nothing you need to do.

We have a complex query capability. We tap in and use link with that, so you can use a similar link query syntax to work with the database. You can sort objects and use a number of different query capabilities.

The really interesting thing about this, is this lazy loading aspect of the database. When you perform a query, you’re not having to actually pull all those objects into memory. If you imagine you had 10,000 objects that match your query, it does not have to actually move all those into memory. Instead, only when you start accessing the individual objects in the query, do you actually start interacting with them. The memory performance for applications that have tons of data is much lower than you would find with SQLite.

When you perform the entries = _realm.All command, it’s not having to move all those objects into memory. Instead, if you had the screen where the objects were much larger than what could fit on the screen, as you’re scrolling and trying to call out to get the data, that’s when it’s actually interacting with it, so it’s extremely efficient.

We don’t support limit, because it’s irrelevant. You don’t have to paginate when working with this.

The vast majority of times, you can do it all on the UI thread. Now I know that’s kind of a very controversial statement. We’ve designed the performance with the memory mapping, lazy loading, and all these capabilities, that it really can be used on the main thread. Now there’s always going to be an edge case. You can very easily switch to background thread, get a new instance of the Realm for that thread, and work with the objects on the background thread. Then utilizing those notification capabilities, if you say, “were doing some sort of background synchronization on another thread”, and you make changes, your UI thread can subscribe to those notifications as those changes come in and react to it.

It’s a very simple model in terms of being able to listen for changes that are happening across threads. Even more interestingly, this works across processes. If you have an iOS, widget of some sort, where you’re using one of the notifications, original to apple watch app, you have different processes. Another process can be working with the Realm, making changes, and the other process can actually listen for those changes.

Realm Mobile Platform (28:38)

There was a master plan with Realm all along, and it builds off of this idea that the Realm objects are live and auto-updating. As you’re making changes, they’re being synchronized across threads.What we wanted to do was basically say, imagine it’s not the change coming from another thread that’s updating the object, what if that change is coming from another device, or a server? For the past couple years, we’ve been working on what is called the Realm Mobile Platform, and it includes a Linux-based application called the Realm Objects Server that automatically synchronizes changes. The really slick thing is, you pretty much already know how to use it if you know how to use Realm. There’s not much else more to it in terms of the client experience.

Every time you make changes to the local database, they get persisted locally on disk, and then the change itself is sent to the object server and it handles sending it wherever else it needs to go.

It is extremely low-latency. We aren’t sending the state of objects around, which is like traditional REST API, requesting the the json. Instead, because we control the whole stack, we know the specific changes that have actually occurred, and we send the operations.

It’s designed to also be offline first, because from a client mobile developer perspective, your source of truth is the database on disk. You’re making changes to it, and then asynchronously, those changes get sent. On the server application, we used what’s called operational transformation. It’s basically a way to make sure that all the changes get ordered correctly regardless of whether the network has sent them in order.

Realm Browser (31:06)

There is another app included that you can play with for the Realm Mobile Platform. It’s another task app. You just need Realm and Realm database. The key difference is there is a name space sync that includes a number of different APIs. If you don’t want to use the mobile platform if you’re not using the server, you don’t have to; it doesn’t change the database itself. It’s just other APIs that you can tap into if you want to use it.

We put a little more effort into the UI for this app, but it’s a task application where you can have lists, and then you can click in and add different tasks here. It’s also using Xamarin forms.

The key thing to show off is that, we have this mac application. We are working on a cross platform version of this, but today the Realm browser is just a mac application. It’s really cool because you can go in here and you can open up, local Realm database files. You can use this even on device, when you’re developing, you can go get the Realm file off the device and explore it if you need to.

If you download the zip file, the package for the Realm mobile platform, we have a free version, which is called the developer edition. You can run it locally on your mac, which is mainly just for debugging. If you’re going to use it for production, there’s a whole Linux installation. I’ve got it running just a single command. You start it up, and it runs in the terminal. After you start it up, it’ll open the dashboard. It’s a very primitive and basic dashboard at the moment.

You can see all the synchronized Realms that you’re working with. You can see all the users that are actually in the server itself, and even check out the logs.

I’ve got the server running locally, and so I have connected in the sample app, Realm task. I’ve logged into this local server. The synchronized Realm I’m connecting to is connecting to the local host and opening up the Realm file.

Beneath the hood, each application has its own copy of the data, and so the browser has its own copy of the Realm. The simulator has its own copy of the Realm, and the server, running in the background, is going to automatically synchronize the changes. If I make a change, you’re immediately going to see it. I can then undo the change, and call this task two.

The nice part about this, because it is a Xamarin form app, it is basically doing implicit transactions. Every time there is a property update, it is tapping into the inner workings of the data binding mechanisms to be able to create a write transaction on every small change, and so that’s how you’re seeing, it.

Specific character-by-character changes happening.

As a developer, you’re still just writing to make changes to a Realm object. Beneath the hood is the async networking that’s just doing everything for you. That’s really the productivity benefit. I’d rather not have to write json parsing and making network requests and handling any sort of the error scenarios. I just want to work with synchronous local APIs, and make the UI beautiful or build the unique capabilities in the app, rather than all the data handling that, we continually have to rewrite.

An individual Realm is the encapsulation of the data being synced. The unique nature is that, a Realm itself is very lightweight. In the previous app, we had the default Realm and we were just putting all the data into a single Realm. With the Realm mobile platform, we recommend to think about it a little differently. You might have several different Realms with different datasets. That’s how you can synchronize data at different times. One part of the app, like the homework feature of the app might only use the homework Realm if the user goes to that area, and if not, it’s not actually doing any sort of synchronization.

Authentication (38:59)

The permissions for how you work with the platform is all around Realm users. The object server has built-in authentication for username and passwords. It also supports a number of federated or external identity systems, and so, we’ve recently added active directory support.

To actually go and get it implemented, you have to do a couple of things. When you start the server, there’s a configuration file. There’s a section called providers, and this is where you can enable the supported third-party providers. For example I’ve got the Azure provider with my attendant ID, because I’m using the Azure active directory cloud offering.

You have to set your client ID, and your redirect, but basically you create a credentials object, and there’s various different methods that we support. You can create a credential that’s a Facebook credential, a Google credential, or Azure active directory credential, etc.

We’re basically using the Azure STK to go out and get the access token, and then once the client has the access token, it passes it into Realm’s APIs to send that access token to the Realm server and say, “This is my identity through Azure, please verify it.” If it’s verified, then you become a Realm user.

If you just want a username and password type login, its done for you automatically. Instead of creating, a credential where you’re using Azure, you’d be just creating a credential with the username and password. There’s register and login methods associated with it.

Conclusion (41:45)

The number one place is to go for information is the website. You basically have two options. You can go download it locally on your mac, and get it up and running really quickly. Alternatively, you can go over to the Linux install. We support a couple different flavors of Linux: Red Hat, Enterprise, CentOS, and Ubuntu. We also have AWS/AMIs available, but the easiest way to do it is to get your Linux VM up and running, and then we just go through a couple of.

I think that sums it up. If you’ve got any questions, we are very active with support. You can go to the GitHub repo, and file an issue if you have any sort of problems that arise. Our documentation is all on realm.io, that’s just the best place to start if you’re interested.