I stayed close to home, I already was in a System Integration project for my paid job, where we used Apache Camel. I was quite content with its elegance and simplicity.
However, it is in Java and I don’t know of any similar product in the .Net world. So I thought it might be an interesting experiment to create something similar, and I’ll keep/change things to my likings.
FsIntegrator
FsIntegrator is a DSL in FSharp that provides a framework to implement Enterprise Integration Patterns (EIP). What you actually create is middleware, which implements a workflow between different enterprise systems.
For example, if you are creating a service-order backend application, you may want to connect the inbox to an incoming order channel (ie. the backend of an E-Commerce website), send a message to a billing system to start billing process and send a service-call to your business-platform to activate the service you are selling. One system may work with a Soap interface, another system is connected via queueing software and the third system only processes plain CSV text files. This is what EIP is about.
Such a system-to-system flow can be defined by routes, which start with a listener on an endpoint, after which the processing is defined. If you listen to a queue, you can process the incoming XML message. If you are listening on a filesystem folder, you can process the content of incoming files. A route somewhat follows the actor model, where one route implements a static actor. So a route is just the machine, processing incoming data on the endpoints. A route is isolated, so only one message is processed, before the next message is accepted.
FsIntegrator is still immature. It currently only supports endpoints for: filesystem folder, FTP folder, ActiveMQ and Subroute (shared internal). There are a few logical functions to support the Content Based Router pattern. And there is some error-handling, where exceptions can be diverted to a different route, or equipped with custom logic before terminating the process. So this blog is not about a fancy end-product, but about interesting learning points while developing this.
Modular Setup, seamless integration
In these days people really like to include what they want, and leave what they don’t need. Keeping it small so to say. So I created a module for ActiveMQ. If not needed, it should not be included. If needed, you can include it, and from then on it should not be hard to find. I rely on IntelliSense to achieve these goals.
The namespace “FsIntegrator.Producers” has the “From” type, under which all endpoint listeners are available. If the FsIntergrator-ActiveMQ dll is not referenced, the IntelliSense shows the following:
Hiding Implementation Details
I have already written a blog aboutImplementation Hiding in F#. The goal of implementation hiding is to increase user experience for our DSL, keeping them distant of boring details.
Hiding comes in this case with a little challenge. The FsIntegrator Core dll has a Route Engine. Routes are added to this engine, and this engine does some management, start/stop routes, validate collisions such that no two routes are listening to the same endpoint, etc.
Now if the ActiveMQ module is built in a different dll, it does not know (much) about the route-engine, which is in the Core. But it should provide “hooks” to which the external RouteEngine can connect. These hooks are typically details that I want to hide.
The FsIntegrator ActiveMQ listener obscurely implements an interface, via which the hooks are provided. The obscurity lies in this. The implementation of the ActiveMQ listener class implements the interface defining the hooks. But in the corresponding signature file, this interface is not mentioned. An outsider who only looks at the surface will never know about the hooks.
You get compiler warnings for this, which you can suppress, and runtime you can still dynamic-cast the type to the hook-interface it implements.
Component Options via Intellisense
In FsIntegrator, I wanted to use IntelliSense as to present which options you can configure on an endpoint. So these are statically typed, and thanks to Discriminated Unions we can even provide complex option/configuration values. At the DSL level, an endpoint is configured with a list of options, which are specific for that endpoint.
Monad Mini Language
For FsIntergrator I wrote my first monad. I implemented a few CustomOperations as mentioned in the cheatsheet, to provide a mini-language for FileSystem and FTP system operations. Nothing with quotations (yet).
Sometimes, after processing a file, you do not just want to move the file to some backup location and quit. Sometimes you want to delete a non-empty tree on the FileSystem, or in some cases even create such a tree at that moment. Or perhaps you receive a package of files which belong together, but your listener is only triggered by xml files, and when ready, the package needs some post-processing.
Examples
I’ve created a few test scripts, to test FsIntegrator, manually. These scripts can also be considered as (hopefully) self-explanatory tutorials to use FsIntegrator. Well, this is the current status.
Future plans
A couple of things still on my todo-list are:
• File and FTP components currently only work with Xml files, it should also work with other content-types;
• All components should get more configure options, to improve route customization;
• Components which require network should get extra attention for security (logins, certificates);
• Create new components for Rabbit-MQ, Redis, etc….
• Automatic documentation generation should greatly improve;
The repository can be found on GitHub. I used Scaffold, FAKE and Paket, such that this can be build and run under Windows/VS2015 (tested) on Linux (tested) or perhaps OSx (not tested).