Small features leading to more reliabilty

Since some years I've allways ported one helpful library to my used programming languages. So I've got it in Java, Python, Smalltalk and now Erlang. It is the execution time monitor CELETM. It allows to gain useful informations about how often blocks are called, the minimum execution time, the maximum time, the average time and the total time. The two ways to use it are

Measuring = celetm:begin_measuring({?MODULE, my_block}),
do_this(),
do_that(),
celetm:end_measuring(Measuring).

or

celetm:measure({?MODULE, my_func}, fun() -> my_func() end).

as a convenience function. The accumulated measurings can be retrieved with celetm:retrieve() or displayed in a table in the shell with celetm:io(). Options for sorting and filtering or output to an i/o device exist. So what's the special feature here? My old solutions allways had two central tables, one as a buffer for collecting the single measurings, one for the accumulated measurings. The access to both have been controlled by synchronize blocks or semaphores. With a high load calling processes, e.g. in a server, would have to wait, a typical bottleneck. The operation here is fast enough, so I never had this problem. But it's easy to imagine similar situations with longer lasting central functions. The Erlang solution uses a gen_server as a backend, and celetm:end_measuring() is an asynchronous cast message. So it's a fire-and-forget for the calling process. The only synchronous functions are celetm:retrieve() and celetm:io(). So only they would have to wait. And thanks to the Erlang messaging the implementation has been a really easy job with the shortest code ever.

Future plans

In a comment I've been asked about my plans regarding the Common Erlang Library (CEL). So I'll now write about the current status and my future plans for the Erlang development. Before I talk about the products I've got to say that I'm right now just evaluating Erlang. I've developed lots of stuff in Smalltalk and I still love that language. But as you may know I'm focussed on server-side application and I'm interested in scalability and reliability. So here Erlang is really interessting for my and I may switch. The actual evaluation experiences are really good. And definitely I'll release most of my code as open source when reaching a beta status.

The first project I'll realize in Erlang is the above mentioned CEL. I've got such libraries for almost every language I use as a stack of useful features I need. You allready read about the simple markup language (CELSML), which is a library for the creation, parsing, searching and later manipulation of SML documents, which is a lightweighted alternative to XML. CELSTH is based on this library and contains an extensible SML-to-HTML converter. Another helpful library is the execution time monitor CELETM, I've also implemented this in almost every environment I use. *smile* It allows simple execution time measurings and accumulation of logical blocks. The biggest part of the CEL will be the lighweight message bus CELLMB. It will provide the asynchronous distribution of messages to registered processes, realized using the publish/subscribe paradigm and based on a context, a verb, a noun, and optional metadata. So a request check permission sent in a context authorization will be answered by a registered authorization service process (it may catch all verb-noun combinations or they may be splitted to different processes, this depends on the individual needs), but also listened and logged by an audit service process. So flexible loose coupled architectures are possible. They benefit very much from the Erlang features (processes, messaging, distribution). Last currently known part of the CEL is a set of utilities in the CELUTL module. It provides a ping/pong to monitor processes or a GUID generator.

The first greater application developed simultaneous to the CEL is the Dynamic Content Processor (DCP) Release 2. This is a small content management system containing a blog and some more dynamic features I need. It will be based on Yaws and Mnesia and replace the software behind this blog, www.tideland.biz, and some wikis like wiki.tideland.biz. Here I'll see how Erlang is the right tool for my future projects. Two planned ones - currently I'm compiling the requirements - are the Train of Thoughts (TOT) and a portal for the collection, searching, viewing, and comparing whisky tasing notes. All three, the DCP, TOT, and the whisky portal, need a reliable platform.

Power of Erlang pattern matching

One thing I really like in Erlang is the pattern matching. It allows small and powerful code which would be much more in other languages. I've got one example in my SML code. Here I need a function for the recursive search of matching nodes. A node is a tuple like

{node, Tag, Qualifier, Childs}

where Childs is a list with more nodes or text entries like

{text, Data}

A matching node is, where the given tuple is a node and wheere Tag and Qualifier match their search counterparts or where those are wildcards ("*"). In a simple pseudo code - without casting - this would look like

FUNCTION node_matches(Test, Tag, Qualifier)
IF Test HAS TYPE node THEN
  IF (Test.Tag EQUALS Tag) OR (Tag EQUALS "*") THEN
      IF (Test.Qualifier EQUALS Qualifier) OR (Qualifier EQUALS "*") THEN
          RETURN true
      END
  END
END

RETURN false
END

It works, and is standard. But I think the Erlang way is much more elegant here

node_matches({node, Tag, Qualifier, _Childs}, Tag, Qualifier) ->
  true;
node_matches({node, Tag, _Qualifier, _Childs}, Tag, ?WILDCARD) ->
  true;
node_matches({node, _Tag, Qualifier, _Childs}, ?WILDCARD, Qualifier) ->
  true;
node_matches({node, _Tag, _Qualifier, _Childs}, ?WILDCARD, ?WILDCARD) ->
  true;
node_matches(_Node, _Tag, _Qualifier) ->
  false.

Erlang development continued

The last week I've continued to develop my first project in Erlang. As you allready know it will be the Dynamic Content Processor (DCP) using Yaws and Mnesia. Currently I havn't reached those last two libraries, but my own first one, the Common Erlang Library (CEL), grows.

I've started with the simple markup language, SML. I allready wrote about it. It is a really nice approach how the parser reads the document and sends events in form of messages to a background builder process. And the last message returns the constructed document. In the standard library this is a structure almost like the XML DOM, but much more lightweighted. It really astonished me how easy it is to parralelize typical serial problems.

The next step has been the HTML builder. It uses the same approach like above to produce HTML. No full blown HTML, but fragments to be inserted into templates. It shall be used in the DCP. This library is part of the CEL and can be extended with a callback function. This allows the DCP - or other apps - to extend the SML-to-HTML converter easily.

Beside the processes and the asynchronous messages the pattern matching is really helping. It makes the receiving of the messages and the definition of small and neat handling and poweful callback functions simple.

I'll now continue to develop the converter together with some more unit tests. Once again they help a lot. After that I'll develop something similar to my Smalltalk Lightweight Application Server, based on the Erlang processes and asynchronous messaging. So again, stay tuned.

Whisky tasting season started

Last friday our autumn tasting season started. *hurray* The time to wait has been long enough. So it has been a real great pleasure to meet all my single malt friends again. My first whisky has been the Ingower 1980 26y Signatory. And it has been the best this evening. The Inchgower is a single cask whisky with 55.6% matured in a sherry butt. The - natural - colour is dark amber, the nose very rich with sherry, dark chocolate, some toffee, raisins, and very few peat. The palate is very warm and complex, but not too spicy. It's like a small explosion and contains fruits, like oranges, and caramel, chocolate, and cream. At the end the Inchgower pleases with a long and warm finish. Really, a very good malt whisky. It has been damned hard for my not to buy it - it costs about 120 Euro. *sigh* Maybe next time.

Additionally I've tasted several malts more. But I've had nothing to write down my tasting notes. Currently I'm creating a new, better tasting card. Next time I'll concentrate on a few malts and immediately write my percepts of nose, palate, and finish down.

More Erlang impressions

Since a few days I'm redeveloping the new release of my Dynamic Content Processor based on the simple markup language. I'm using the Erlang/OTP for it, just to learn Erlang. Additionally the documentation and experiences show that Erlang is a very good platform for reliable and performant server applications. The last week I've build a library for SML parsing and processing, a registry for a dynamic process registration and spawning, utilities and unit tests. Not very much, but with positive experiences.

The first one is the usage of concurrent processes. I've started with the development of a structure for my SML data, just like the XML DOM, but more simple. Like in Scheme I've created a set of functions that parsed a string and worked on the structure, synchronously. In most languages you're going this way. Straight and linear. But then I thought about the concept of Erlang and changed everything. I've developed a background process which PID is passed to the parser functions. Now the parsing is in one process, the structure building asynchronous in a second one. Scaling made simple. *smile* A consequent usage of this approach lets Erlang applications scale with the number of SMP cores or nodes. As a next step I've build a process registry to get an indirection for the process creation. So in future the processes can be exchanged via configuration, e.g. processes responsible for authentiction or authorization. The organization of processes is also very simple. The span is from simple one time processes over registered single or multi node service processes up to distributed and pooled processes with message distribution and self monitoring for an automatic restart of failing processes.

The second big step has been the functional programming. OK, I've allready had this in Scheme, but not solely. In Erlang the developer is forced to do so. There are only pseudo variable in patterns, they are constant after the assignment. So the thinking has to change. The advantage is the absence of side effects with only one exception. This helps to make applications more reliable, especially server applications with a hardly predictable behaviour in parallel calling of central processes. By the way, also here the asynchronous message queues are helping. No semaphores are needed and returned structures can be changed without threatening the process, because they are no references. The only side effect is in processes with tail recursion and passed state arguments. When used by different processes this state can change in a way those processes don't expect. But this scenario is easy to handle. Recursive structures using the records, atoms, tuples and lists are a bit difficult to handle, because every change returns a new structure. So the changing needs a traversal of the structures which makes it expensive. Maybe I'll find a hint I've yet overlooked.

The last thing which impresses me is the pattern matching. Multiple function definitions orif and case statements based on patterns and guards are a nice way to structure the code. I've learned it with Prolog and XSL-T and it's really powerful. So the work with atoms, tuples, and lists gets really simple. So as a first recapitulation I'm impressed by the way how fast powerful application can be developed in Erlang. I'm focussed on server applications and it seems that the Erlang/OTP has a very good chance to get an important platform for me. I'll continue to develop my DCP based with Yaws and Mnesia and report here.

SOA Days 2007

This week I've been in Bonn at the SOA Says 2007 Technology Conference. The program was a good mix of key notes, practice reports, and networking. I'll work out an abstract for my colleagues the next days. The most important facts are: The hype is over, in the Gartner hype cycle it's now somewhere between the Trough of Disillusionment and the Slope of Enlightenment. The number of working SOAs is growing, the greatest startet in the late 90s, but without the name SOA. Very often asynchronous message queueing is used instead of web services, service orchestration using BPEL is not allways a topic, and governance and a repository are important when the number of services grows up to several hundred or thousands. So nothing real new, but the talks contained some nice details and ideas.

One quote I've liked the most was by Eclipse Foundation Executive Director Mike Milinkovich: "You know what software development is? Doing the same over and over again." He told this in the context of constantly recurring ideas in new shapes. When I think about the ages of Smalltalk, Lisp/Scheme, Prolog, Erlang, relational and object-oriented DBMS, CORBA, TCP/IP, and several more, I allways feel that there happened nothing real new during the last years. Only variations, most times not better than the original. So it's really important to get the conceptional ideas behind of them, the paradigms and patterns, to make the best out of the technology of today.

Another thing I've done during the train travel and at the evenings in the hotel has been some development in Erlang. I'm still impressed and make good improvements. So my next entry will be about it.

Discovering Erlang/OTP

Since 1984 I'm now developing software, for myself, as a side job, as a part of my job and since 1999 as my main job. And during this time I've twice had the feeling that something very important reached the market. Both times I've been right, even if I've haven't been the first promoter. Now I've got this feeling again.

The first event has been 1989, when I discovered object-orientation for me. I've read about it before but in this year I've had the first chance to work with it. And I quickly realized that this is a very good paradigm to get a natural abstraction and encapsulation of data and behaviour. Until today I've developed most of my software this way. The second event has been 1996, when I wrote my first Java programs. Java as a language is inferior to pure and stringent object-oriented languages like Smalltalk. But in the late 90s Java had the right mix to gain ground. It has been pushed by Sun - only one company, no different ones with incompatible dialects, it could be used for free, the virtual machine supported the development on standard PCs and the usage on Solaris or Linux servers, the large runtime library provided a lot of functionality, and the curly bracket syntax attracted many C and C++ developers. This has been a very good package for the starting internet era.

And now, once again, I've got this feeling that a special technology fits optimal for upcoming challenges. Dual and quad cores in notebooks and desktop PCs, the new CPUs like the Sun UltraSPARC T2 processor and the Tilera TILE64 processor family, large data centers with a very high number of independent computers, or just the Internet show the direction. The computer technology is moving away from standalone high frequency single core systems to networked computers with a high number of cores. And future applications have to live in such environments, in some cases with a centralized infrastructure, in other cases loose coupled and communicating peer to peer. Those applications have to scale, work reliable, recover automaticly after high load or failure situations, and handle dynamic reconfigurations due to their nature running 24/7.

One possible solution for this challenge is an environment with a language that is not only supporting the development of parallel algorithms but is nearly enforcing it. A language, which focuses the usage of processes, which makes communication and distribution easy, and which prevents unreliable behaviour. Such a combination is Erlang/OTP. OTP is the acronym for Open Telecom Platform and shows the roots of Erlang, even if it is not restricted for telecom software.

Erlang is a strong and dynamic typed functional language, compiled to bytecode and executed in a portable virtual machine supporting hot code updates. It's available for Solaris, FreeBSD, Linux, OS X, VxWorks and Windows. Different processor architectures and SMP are supported. The notation is declarative with a powerful pattern matching like Prolog. In fact the development of Erlang has started in 1986 by Joe Amstrong with the experiment to add concurrent processes to Prolog. Those processes are now one of the major features of Erlang. They are extreme lightweighted, share no memory and communicate via asynchronous messages. Each node can handle a large number of processes and they can be distributed over the connected nodes. Processes can monitor other processes, even remote ones, restart them in case of an error or handle a fail-over and migration back for completely broken nodes. They are additionally independent of operating system processes or threads. The Erlang nodes are running VMs on the same machine or distributed over a network. The operating systems of these machines may differ. Started with the right parameter they build a virtual network where processes can communicate the same way independent of their location.

Beside the language, the compiler, and the VM Erlang/OTP contains a large and powerful library and helpful development tools. So e.g. a HTTP client and server, a FTP server, a distributed real-time database, a CORBA 2.0 ORB, and a SNMP agent are parts of the library. The tools contain monitoring, tracing, debugging, coverage, profiling, testing, and lots of stuff more. All have been proven in several high-available systems and applications, mostly telecom or internet. So today Erlang provides a powerful system focusing the development of reliable and scaleable software.

This focus is the main fact why I'm so fascinated. Many languages support threads and processes, support remote procedure calls, and support hot code deployment. And they all allow to develop reliable software. But I've never seen a language which enforces it like Erlang. So I've started my digging developing my first small evaluation project. You, too?

There is nothing more powerful than an idea whose time has come. (Victor Hugo)

Whisky specialist Michael Jackson is dead

The world wide known Whisky specialist and most important author for this topic died on August, 30th, in the age of 65 years after a long illness. That's a very sad moment. *sigh*

Slainte Mhath, Michael.