Webserver Back End with Orleans Actor System, Dotnet Core and Server-Side Redux

The RROD code

Microsoft Orleans Basics

This project is about modern application architecture, based on Dotnet Core. Microsoft Orleans is a large codebase, that is used for running some big projects inside and outside Microsoft. The current version runs on the .NET 4.5 Framework. It makes heavy use of binary serialization and it uses Windows API’s to manage memory and CPU load. This makes it non-trivial to create a cross-platform Dotnet Core version. But the people working on the Orleans project are very capable and work on the Dotnet Core-compatible version 2.0 has progressed steadily. The second pre-release came out recently. This version is used in RROD.

Setting up a solution to use Microsoft Orleans requires a project structure consisting of a minimum of four projects:

• OrleansHost — this project buids an executable that starts the Orleans runtime (the “Silo”) hosting the Grain objects. You can start multiple hosts, they will start working together and share the load.
• Webapp — the web server application project. This project connects as a client to the OrleansHost process(es), over TCP.
• GrainInterfaces — this project contains interface definitions of all Grains, and is referenced by both OrleansHost and Webapp.
• Grains — this project contains the implementation code of all Grains and is deployed with OrleansHost.

During development, the OrleansHost and Webapp processes can simply be started and debugged together on the developer machine. In production, they will usually run separately on a cluster consisting of multiple machines, offering reliability and distributed resource management. Of course, a small application could still run on a single machine, without the reliability but still benefiting from the real-time architecture and high performance.

In the introduction I said I didn’t want to be bothered with caching anymore. However, several AspNet Core libraries need a distributed cache. In a scale-out architecture, you’d normally use the Redis cache implementation that is provided as the default. As an illustration on how one can use Orleans, here is my complete distributed cache implementation for Dotnet Core, using Orleans instead of Redis:

The GrainInterface class declares four functions needed to implement the cache (Get, Set, Clear, Refresh). The Grain implementation just wraps the cachedata in an Immutable (a performance optimization to avoid copying) and keeps a reference to it in a member variable.

The OrleansCache class then implements the IDistributedCache interface that Dotnet Core defines in Microsoft​.Extensions​.Caching​.Distributed. It uses GrainFactory.GrainClient.GetGrain<ICacheGrain<byte[]>>(key) to get a reference to a CacheGrain identified by the cache key and then stores and retrieves data as a byte array. Grains can be generic types, so we can directly use the byte array with the CacheGrain. Registering this cache service at startup by calling services.AddSingleton<IDistributedCache, OrleansCache>() gets you a fully scalable, production ready distributed cache for Dotnet, that other services (such as Session State) will automatically use.

This kind of code is easy to understand and trivial to debug. There is no need to install and manage Redis, think about redundancy, manage an extra connection string, learn another API, learn another query language if the cache needs to be smarter. It’s all built into the technology stack.

Server-side Redux Actor

I am exploring client-side and server-side technologies together. To find out how an Orleans based back-end could work together with a modern javascript front-end, I created a server-side version of the “Counter” sample that is often used to demonstrate Redux. This demo simply increments and decrements an integer value by dispatching Redux Actions, displaying the current state. My version has the following additions:

• When a button is clicked client-side, an AsyncAction is kicked off that dispatches the Action client-side (updating the view) and also posts a request to update the state at the server using a Web API call.
• The web server receives the request, gets a reference to a CounterGrain object (identified by the web session ID) on the Orleans Server, and calls the equivalent method there.
• The CounterGrain is subclassed from ReduxGrain, which is a common base class for grains in this project. It uses an Azure Storage Table to store every Redux Action it ever got. On activation of this Grain, the stream is replayed to get the current state, after which new Actions are processed, added and stored.
• The user can refresh the browser anytime. When this happens, the page is rendered server-side, using the current state from the grain. Orleans keeps Grains in memory, once activated, for two hours by default (or until the Orleans runtime decides it needs to free up more memory). This means that reloading data on the client does not result in database hits on the server; the grain is in memory already and it always has the latest state.
• Updates after the initial page render happen client-side, using the same React based component.
• The Server can initiate updates at any time. In a more complex scenario another user might cause the updates. To simulate this, I added “Start” and “Stop” buttons that operate on a server-side timer managed by Orleans. When started, the server will increment the counter every 3 seconds.
• When a timer tick happens, an IncrementCounter Action is generated on the server. It is processed by generating and dispatching the action to the Store, and saving it to storage. On grain activation we registered a subscription on the Store that will Publish the new State to an Orleans Stream “ActionsToClient”.
• Every client keeps an open WebSocket that subscribes to server updates for its session. The corresponding SocketHandler on the web server subscribes to the Orleans Stream “ActionsToCient” with the id of the session. Actions can be published to this stream from any Orleans Grain. They get passed to the WebSocket and dispatched to the client-side Redux State, resulting in an automatic update of the view.

Here is a schematic overview of the flow:

And here is what this looks like to the user:

Web API

Below is a snippet from the AsyncAction that gets called when the user clicks the “Increment” button. Note that I’m adding a header to protect against cross-site scripting (someone better than me at javascript would intercept the call and add the header auto-magically).

The code below also shows the ES2015 / “modern javascript” syntax using async and fetch(), and references typescript interfaces and string constants generated from C# server code by Typewriter (in the Server import), as discussed in my previous post on client-side javascript.

The MVC Controller receives the /incrementcounter request, gets a reference to the Counter Grain for the current session and calls the corresponding method, awaiting the result.

The Redux reducer that processes the action server-side in C# is almost the same as the one in typescript working client-side: it processes the action and returns a new resulting state.

The ReduxGrain base-class takes this reducer function as a constructor parameter, together with an injected storage handler of the correct state type, to implement the Dispatch() and WriteStateAsync() functions.

The Dispatch() method takes an action and updates the State, using the Reducer. The WriteStateAsync() function is functionally equivalent to how normal Stateful grains store data in Orleans. This makes it easy to change normal Orleans grains to Event Sourcing grains. ReduxGrain implements its own version of WriteStateAsync() that appends its internal list of unsaved actions to an Azure Storage Table using a partition-key based on the grain id and a row-key based on the timestamp.

The methods inside CounterGrain are the equivalent of the AsyncAction in javascript Redux. Dispatch() executes the reducer which should be a pure function without side effects. Grain methods can dispatch actions and execute side-effects, such as saving to storage, publishing events to a stream and starting timers. The architecture could be taken even further by moving those methods into another type SagaGrain that would manage long processes.

In the StartCounter case, we start an Orleans Timer, after which the registered method will be triggered every 3 seconds.

If you look into the source code of the RROD project you might notice that I experimented with different ways of sending actions between the browser, the web server and the Orleans server: using WebSockets or Web API, using Orleans Streams or direct RPC, passing Command objects to a generic Process method or using separate Web API calls with typed data. All these methods work; I eventually settled on using plain Web API calls when going from Client to Server, the controller method will just forward a request by calling the corresponding interface method on the Orleans Grain. The await (Promise) in javascript will resolve only after the Web API is fully processed in the Orleans Grain. This makes it easy to handle errors.

Data flowing back from the server to the client uses Action messages that flow through a WebSocket connection that is opened when the client starts. The flow is made fully asynchronous, using Orleans Streams. Making secondary messages asynchronous in Orleans avoids deadlock scenarios that can otherwise happen easily in Actor systems.

WebSockets and Dotnet Core

I used SignalR in the past with great success and I wanted to use it here too. However, a cross-platform AspNetCore version of SignalR is not available yet. It is currently being built by the AspNetCore team. I first tried to use a prerelease version, but could’t get this to work together with currently released versions of Dotnet Core. Then I found this WebSocket manager middleware for AspNet Core (by Radu Matei) that worked, so I went with that. There is no fallback for browsers that don’t support websockets, and it’s somewhat low-level, but it works well enough for passing simple json objects. When SignalR becomes available, it can easily replace the current code.

Testing it

As in the previous installments of this series, you can play with this yourself. Clone https://github.com/Maarten88/rrod.git, open it in Visual Studio 2017 and run both OrleansHost and Webapp projects. Start the Azure Storage emulator first. Alternatively, on Linux or OSX, you could use VS Code and run dotnet restore, dotnet build and dotnet run on those projects. The Azure Storage emulator is Windows-only, so you may also need to create and configure a “real” Azure Storage connection in the appsettings.config file. Automatic typescript model generation for C# models only works in VS 2017, install the Typewriter Add-in and change any model class to see it working. The generated typescript files are checked-in so if you don’t use Visual Studio you can still run the project, but you’ll have to synchronize javascript and C# models manually if you change something in them.

The resulting Counter page works as designed: Updates can be initiated from the client and from the server. Page rendering works server-side and client-side, and the page is updated automatically when the state changes. The complete action history is saved in an Azure Storage Table.

If you start at the Home page, navigate to the Counter page, increment the counter you’ll see that the Counter page is rendered client-side. View-source will show an incorrect count. Do a refresh while on the Counter page and you’ll see that the page is re-rendered server-side, and you see the correct current count in the html source.

Deleting the session cookie and refreshing the page will generate a new session, resetting the counter to its initial value of zero. Don’t worry if you start a timer and then leave the grain behind: Orleans will let the grain run for some time and then deactivates it automatically when it doesn’t receive input anymore. If you want a grain to persistently reactivate, you can use an Orleans Reminder.

Limitations

The only limitation of an Orleans-based architecture that I ran into is that it provides no built-in way of searching for data, or doing interactive analysis on data. Modelling actors in a smart way can often avoid the need for an index, but sometimes you just need to search through a big pile of static data and loading all of it in memory as actor objects is not feasible. This can only be efficiently implemented with an index. I used Elastic Search or Azure Search as a addition to my Orleans projects to fill that gap, for instance to provide an faceted search and navigation feature. If you need powerful reporting then PowerBI is another good addition to Orleans.

At Microsoft Research, where Orleans is developed, they apparently ran into this limitation too, and smart people there have been working on built-in actor indexing. They tell me their code will be released sometime soon, just like support for distributed transactional actors, a limitation that I didn’t encounter myself yet. So even these limitations may go away eventually.