Sequences, LINQ, Rx, & Reaqtor Part 5: Remotable Expressions

To understand Reaqtor, it is necessary to understand Rx (the Reactive Extensions for .NET). And to understand Rx, it is necessary to understand how C# works with sequences of items. In this series I am outlining the ideas at the heart of Reaqtor, and how they are handled in C#. In the preceding part, we saw how Reaqtive adds extra 'dimensions' to Rx. In the final part we'll see how this comes into play in fully-fledge Reaqtor as a Service, but first, we need to look at one of the most distinctive aspects of the Reaqtive world.

One of the most important features added by Reaqtor, and its supporting libraries Nuqleon and Reaqtive, is the ability to serialize expression trees. We've already seen how with IQueryable<T>, it's possible to translate a LINQ query into something like SQL and execute it remotely. But with the ability to serialize expression trees, it becomes possible to enable remote execution of queries without needing to translate them to anything else: you can just ship an entire LINQ query to another process and run it there.

That capability is at the heart of Reaqtor.

First, there's the obvious fact that if you can send expressions over a network, this opens the door to being able to offer "Rx as a service". Whereas with classic Rx, everything is inherently in-process, remotable expressions provide a way to send a description of a subscription to an external process. Slightly more subtly, serializable expressions also help enable persistence—if you have a serialized description of a subscription, you can store that somewhere persistent, and then reload it into a new service instance if the one it was previously running in stops running for any reason. (In practice, persistent subcriptions require more than just the ability to serialize their defining expressions: many Rx operators maintain state, which must also be persisted, but the serialization of expressions is still a crucial element.)

It isn't strictly necessary to understand the exact details of how expressions are serialized—in fact the Reaqtor codebase offers a couple of different formats. Normally, a JSON-based format is used, but there is also code in there for serializing to a binary representation. The fundamental capabilities are more interesting than the exact format details. So it's worth knowing that serialized expression trees ('Bonsai trees', as they are typically known) are able to represent more or less any expression that a .NET expression tree can, but they are not necessarily tied to .NET.

It's important to understand that Bonsai trees are not compiled code—we are not shipping IL (.NET's equivalent to Java bytecode) over the wire. They are a description of the original structure of the expression. Also, they are typically self-contained: there is normally no need to distribute any particular extra compiled code to the receiving service to enable it to understand the tree. (Contrast this with some systems that enable remote invocation of user-defined functions implemented in Java: Spark supports user-defined functions, but to be able to run inside the processing nodes of a Spark cluster, the compiled version of a function needs to be accessible to those nodes. So you typically end up distributing .jar files across the cluster. This is unnecessary with Bonsai trees because they can be self-contained. After a service converts one back to a .NET expression tree (applying any necessary security checks, and whatever transformations it requires), it can compile it into runnable code at runtime.)

Bonsai does support embedding very .NET-specific information if you want to. (E.g., a Bonsai tree could indicate that an expression invokes the Where operator defined by the Queryable class in the System.Linq namespace in a specific version of the .NET System.Linq assembly). But it doesn't have to work this way.

When used with Reaqtor, Bonsai trees often replace such specific references with more abstract ones. It is common to rewrite expression trees so that particular methods defined by certain types are replaced with a placeholder identifying the relevant method by a URI. So although your code might refer to Queryable.Where, the Reaqtor client library would arragne for the Bonsai representation to replace this with an unbound parameter named, say, rx://operators/filter. This ability to substitute URI-based names for particular members of specific .NET types has the useful effect of decoupling Bonsai trees from any particular versions of library components, or from any particular version of .NET.

In fact, expression trees don't necessarily have to originate from .NET. In principle, any language can generate a Bonsai tree, although it's likely to be a lot more convenient if your language has support for something similar to .NET's Expression<T>, in which compilers can be induced to produce code that builds a description of an expression instead of simply compiling an expression into runnable code.

In the next and final part of this series, we'll see how the various Nuqleon, Reaqtive and .NET features described so far come together to provide a reliable, persistent high-performance, event-based service called Reaqtor.