Embrace change

I've spent the december with evaluating Googles new programming language Go. The first comments on the language in the net haven't been very positive: nothing new, no clear paradigm, my language (insert your favorite language here) is better due to (insert your favorite feature here). So I'll start with a rant too: the name Go isn't really good. Just try to google it, you get too many misleading links. *smile* So it's very often called or tagged golang. A different name would be better.

The language itself really seems to be nothing new. Since 1996 I mostly used Java, Python, Smalltalk, and Erlang/OTP. And I find many aspects of those languages in Go. So everybody expecting something revolutionary will be disappointed. Instead Google just put many useful features of those languages above - and others - together, useful in the sense of productivity. So I've been really astonished how fast I've been able to reimplement some of my libraries inGo. My favorite features of the language are:

• Even if it is a static typed language it behaves almost like a dynamically typed language. Implicit declarations, no line ending, no braces for the if- and the for-statement, simplified for statement, implicit visibility.
• Garbage collection absolves the developer from memory management, nothing new for many VM languages, but Go produces statically linked binaries.
• The binding of methods to types works like object-orientation without inheritance. Some may interpret this as a lack, but it enforces simple types and the usage of aggregation and composition.
• Like in other languages interfaces allow the definition of a set of methods that has to be defined for own types to work with special functions, e.g. for sorting.
• The empty interface allows variables and arguments with undefined types (type can be determined during runtime).
• A special parameter allows functions with an arbitrary number of arguments, which alows nice, powerful functions.
• Goroutines and channels are a simple and powerful instrument for the creation of concurrent solutions.
• Go contains anonymous functions, that's allways fine.
• Maps as a built-in data type, no extra library needed.
• Easy access to arrays, slices, maps, and channels using the for and range statements.
• The switch statement can work with case expressions, not only values.
• Functions may return multiple values. Naming them already in the declaration simplifies the return.
• Simple and powerful reflection, not only for evaluating but also for the dynamic dispatching of method calls.
• Without reflection a simplified switch based on a type can be made.
• Very fast builds and - after some first benchmarks - very fast execution.
• Rich library for this early state of the language.
• Tools like gofmt, godoc, and gotest - beside others - support the developer, like the well prepared makefiles do.

As said above it's all nothing new, but it's a good combination. Nevertheless there are some disadvantages. I'm missing several features compared to Erlang/OTP, other features are not yet optimal implemented.

• There's no polymorphism. Sometimes it can be simulated using interfaces, but it's not possible to implement the same function or method in the same scope more than once with a different number of arguments or different types.
• Tuples can be simulated with slices of empty interfaces, but that's not really comfortable.
• Together with polymorphism and tuples I'm missing guards, e.g. for separating functions and methods based on values. These are very powerful instruments.
• Go contains no exceptions. Here the multiple return values shall be used, but already the packages of Google show inconsistencies. Smalltalk and Erlang/OTP show how exceptions can be used without contamination of the whole code. Additionally Erlang/OTP shows how both concepts can live side by side.
• Even if the empty interface should enable the code to handle any data the same way the basic types behave different. So no real generic collection libraries are possible. One way could be the enhancement of the empty interface handling, another one the adding of generics. I would like the first one.
• Goroutines can't be monitored, e.g. for restarting them, channel receives can't be done with a timeout, and there's no transparent communication over the network using channels. Here I'm definitely pampered by Erlang/OTP. *smile*
• The handling of maps, arrays, and slices could be easier. If there are len(), cap(), and copy(), why are there no del(), grow(), or contains()?
• A little embedded key/value database for the storage of nested structs in tables would also be nice, together with packages for searching, e.g. via map/reduce. For larger projects I would use CouchDBor RIAK, but smaller ones don't need this.

And now? How to go on? Continue with Go or continue with Erlang/OTP? Or even with both of them? A hard decission. On the one hand I like Erlangs clean model together with the above mentioned features. But on the other hand the productivity I've reached with Go really impressed me. And what's more important, experimenting with clean single paradigm languages or the fast realization of the list of projects that's in my mind in a language which borrowed all its features from other languages? There have been moments I decided to end the evaluation and continue with Erlang/OTP. But each time I found a quick solution, an easy workaround, a different way to handle my tasks. So I think I'll stay a little while longer with Go, realize a small isolated project - I already got one in mind - with it and then write a second blog entry with more results of my Go evaluation.

Loading mentions Retweet

Yesterday I've introduced the new architecture of the Tideland EAS - formerly known as "Events and Services", now the "Erlang/OTP Application Server". One of the new concepts is the definition of agents for the control of business processes. Those agents are typical callback modules working like finite state machines. The state change is done by internal code, by the returning from services calls, or by the arguments of an external continuation after reaching a waypoint.
 
So here's one example of how to use the agent API from a view or a connector. First the agent will be started, then the reply will be handled asynchronously. Both, the agent and the reply call have a timeout. The first one defines the timespan in which the agent should complete its work, the second one is just for the reply call. 

start_agent(Args) -> 
  % my_agent is the name of the callback module. 
  easagt:start(my_agent, Args, {5, hours}). 
 process_agent(Aid) -> 
  case easagt:reply(Aid, {5, seconds}) of 
   {waypoint, foo, Info} -> 
     ContinuationArgs = handle_foo(Info), 
     easagt:continue(Aid, ContinuationArgs), 
     process_agent(Aid); 
   {waypoint, bar, Info} -> 
     ContinuationArgs = handle_bar(Info), 
     easagt:continue(Aid, ContinuationArgs), 
     process_agent(Aid); 
   {finished, error, Reason} -> 
     {error, Reason}; 
   {finished, ok, Reply} -> 
     {ok, Reply}; 
   {error, timeout} -> 
     {error, timeout} 
  end. 

This is a very simplified code fragment, but it shows how a running agent can stop working when reaching a waypoint where external action - or interaction - is needed. The last shown error is the timeout of the reply call, the inner one can also be a timeout, in this case caused by the agent. If the timeouts aren't passed as integers representing milliseconds the time library of the CEL calculates them based on the tuples.
 
The control of state changes inside an agent is done through the result of the current processing.

process(foo, Args, StateData) -> 
  % Do something ... 
  {next, bar, NewArgs, NewStateData}. 
 process(bar, Args, StateData) -> 
  % Do something else ... 
  {call, [{svc_a, req_a, ArgsA}, {svc_b, req_b, ArgsB}], baz, NewStateData}. 
 process(baz, ServiceCallResults, StateData) -> 
  % Do something with the service call results ... 
  {waypoint, baz, Info, yadda, NewStateData}. 
 process(yadda, ContinuationArgs, StateData) -> 
  % Do something with the continuation args ... 
  {stop, ok, Result}. 

It's easy to see how one or more services can be called, the processing will continue when all services replied or after an optional timeout, or the control can be passed to an external waypoint. Additionally simple service calls can directly be done inside the callback functions. 

{ok, Reply} = eassvc:call(svc_x, req_x, ArgsX, Timeout) 

This API isn't stable yet, but it shows the direction. I'll write more about it here.

Loading mentions Retweet

It's time for a change, a deeper change. A change based on the experiences made during the development and testing of the EAS like it works today. The original idea behind the EAS has been an event-driven architecture where services subscribe to different types of events - therefore the name EAS, events and services. Additionally the platform contains a generic complex event processing. This system now works fine, only the timeout mechanism for replies and time based event processing is missing. So far so good.

 But there's still one problem. The idea has been that business process are a flow of events. Each step can raise one or more events that will be handled then. It's a simple idea, but it's hard to handle. The managing of the control flow is hard, especially when a change is needed. That's not worth it. So the idea of changing the design came up. There are still services, which handle requests asynchronously. Additionally there are agents, each one is a kind of state machine representing a business process. The third part is the complex event processing, still based on subscriptions.

Due to the experiences with the EAS the prototype for this new design will be available soon. The persistence of requests, replies, and aggregations together with the dispatching mechanism like it is implemented today supports the scalability using of large number of nodes in parallel. So the new approach can fulfill the requirements together with an easier API for the later implementation of services, business processes, and event processings.
 
One additional thing that helped a lot during the redesign has been my first usage of the Fundamental Modeling Concepts (FMC / http://www.fmc-modeling.org/). It really helps to design and document software-intensive systems in a simple, clear, and esthetic way. Beside static views like above dynamic ones are supported too. So I'll test it more.

Loading mentions Retweet

I thought a bit about my idea for a next generation Tideland EAS and now made a decision. It is really an appealing idea building a new integration platform for event-driven solutions based on technologies like JSON, HTTP, and JavaScript for the service subscription. But that's not my goal. I've got real applications in mind, currently three different ones. And the EAS is just intended to be the backend framework which allows to implement them as event-driven solutions which work reliable in a 24/7 environment. So I'll continue the development as planed. Additionally I've started writing a small article about my motivation for event-driven architectures on http://www.tideland.biz which will be published soon.

Loading mentions Retweet

When I started the development of the Tideland EAS I had a small pure Erlang event-driven architecture in, just as the middleware for my apps. But sometimes ideas assume in an independent existence. OK, the old concept would also benefit from the reliability and scalability of Erlang/OTP, but not as much as it is in the possibilities of this platform.
 
So, while reading about and evaluating CouchDB, my concept changes. I'm currently thinking about reimplementing it as a real middleware for event-driven architectures. This draft gives a better insight:

The black steps (1) to (3) show the subscription process where a service provider sends a JavaScript filtering the subscribed events. The EAS will store some administrative data, like the ID of the subscription and the service provider, and sends the JavaScript to CouchDB as the database server. CouchDB uses the script to create a view for the subscription.
 
The red steps (1) to (6) show the event processing. A raised event sent to the EAS will be stored inside the CouchDB event table and will so also be visible in all matching views. In case of a polling service the new events will be retrieved and sent to the service. It now can process the event and send the reply or new continuing events back to the EAS. Finally - if wanted by the event source - a reply will be sent back.
 
There are still several open aspects I've got to think about. But I think it looks promising. I'll test the concept with a quick prototype.

Loading mentions Retweet

During the last days I've had to stop my development on the Tideland EAS due to the migration of my root server to two new ones. Now my own projects as well as my customers do have a better environment.
 
But after that I've been able to continue the work on my small event-driven middleware for Erlang based solutions. The completion of a test business process helped to make the API for service developers more simple and consistent. Additionally the configuration and the error handling have been improved. Now there's only the configuration - and usage - of functions for the preprocessing of events and postprocessing the results is missing for a beta release. They are intended to allow the transformation of data from one format into another if e.g. two service interfaces doesn't fit.
 
After this extension the functional part for release 1.0.0 is complete. But I still have to add some functionality for reliability and scalability. So the dispatcher shall notify others if a service registers or unregisters. And in case of a restart through the supervisor they shall retrieve the registered services from other nodes. The same is needed for the configuration. So still some work to do, but that's exactly Erlangs playground. *smile*

Loading mentions Retweet

Currently I'm refactoring the Tideland EAS. During the development it got too heavy, too complex, and with too many potential error sources. Additionally I improved my experience with Erlang/OTP and learned more about the OTP design principles. A good basis for a better design.
 
So the result of my refactoring is a better OTP integration as a base for later distribution and reliability in failover situations - here right now only the communication of dispatchers on different nodes is missing. Additionally I can discard lots of code, so the next step will be better maintainable. Through a better integration of worker processes together with pg2 and supervisor the system will be more reliable. And last but not least the configuration changes to external provider modules that will be passed during startup inside the application configuration. This provider module ohne has to define the callback functions init/0, fetch/1, and fetch/2 and will run inside an own process. The returned configuration value can be hard coded inside a module, e.g. for testing purposes, retrieve them out of a configuration file, or use a database backend for a distirbuted configuration.

Loading mentions Retweet

Even after more than 25 years of software development I sometimes forget some of the best practices. This time the principles KISS and YAGNI during the development of my platform for EDA, the Tideland Events and Services. I discovered it during the development of test services and test cases. Due to the adding of a too high flexibility the API got unhandy. So I started to change it and discovered that the internal architecture tries to cover by far too much cases - independent of their likeliness.
 
So now it's time for a larger refactoring with the goal to strip everything down to the original intention. Beside the fact that the API and the usage will get easier the code will also decrease. That's a good result for the future maintainability.

Loading mentions Retweet

Aaaah, Erlang/OTP R13B has been released today. Great! So I'll dl it asap and test how the
Ttideland EAS behaves.

Loading mentions Retweet

Dean Wampler posted a very good article
(http://blog.objectmentor.com/articles/2009/04/20/is-the-supremacy-of-object-oriented-programming-over)
asking if the supremacy of object-oriented programming is over. It summarizes the current
discussion about languages like Erlang/OTP, Scala, F#, or Clojure. Worth reading it.

Loading mentions Retweet