Concurrency, Parallelism and the World

Since the term concurrency gets more and more into focus people often are confused how it is different from the term parallelism. So there are simple two sentence definitions or longer descriptions. During the last days there also have been some metaphors. I like metaphors, they often helped me to explain ideas and solutions, especially when standing in front of a whiteboard with the colleagues. So now I'll try it now for concurrency and parallelism too.

My scenario is a Citizens Advice Bureau, with a number of clerks for the different tasks and their visitors with different matters. Simply imagine sitting in such a bureau and observe what all those people are doing. There a some clerks working on their own, e.g. ordering and archiving documents or taking phone calls. Others interact directly with visitors, both communicate or exchange papers. And some clerks have nothing to do, they are idling, as well as some visitors whose number hasn't yet been called. But still then they are doing a lot, reading newspapers, checking their e-mail on the smartphone or talking.

Isn't that an interesting bustle? I always enjoy watching those people, how different they are, and how busy. But how does it relate to computers and concurrency? In this metaphor it's pretty simple, all the people are concurrent processes/coroutine/goroutines/actors/objects. And they communicate by talking to each other, fill in forms, writing e-mails or other documents. Some parts of their communication are synchronous, like a direct interaction between a clerk and a visitor. Others instead are asynchronous, like initiating some parts of the work with the first clerk, continue with another part at the second clerk, waiting and getting called by the first clerk, e.g. to receive the new identity card. So the whole business is based on all those individuals, their knowledge, their acting and their communication.

Friends of the good old object-orientation will easily discover the similarities. So are all those programs already concurrent? Sadly not, it also depends on more aspects of the used languages and the runtime systems. It has to support atomic changes of the knowledge - also known as state - and independent acting. That doesn't mean that they are working all at the same time, this is limited by the available resources. In the worst case there's only one processing unit and the scheduler has to assign each process some of its time. But maybe at some time in the future millions of those processes may be able to really work side by side. And here it gets important. The software has to be capable to handle the scaling of resources without any change. So concurrency is a design aspect helped by the language, runtime system and libraries and with the goal to model the real world like described above.

And where is the parallelism? Think back of the example above and how the management may provide an optimal service. So they not only have one clerk for the desk and one for the call center. Instead they have pools of employees doing the same job but in parallel for different visitors or callers. That's all.

Google discontinues the Reader

Since Google announced the discontinuation of the Reader, and other services and products, this week lots if people complained about it. Me too. But now after some days it's easier to see it more differentiated, even positive.

During the last years I concentrated several services at Google - thankfully not all. I didn't payed any money for it, so why complain the closings some may say. But I payed! Instead of money I payed with my individual data, my surf behavior and data of the visitors of my sites (with using Analytics). Google uses this data to sell ads and so make their revenues. It's just a different cyber currency.

So what does this mean to me and my behavior in the future? First of all I've been too careless, my fault. That has to be changed. To save the visitors of my sites I'll remove Google Analytics. Those numbers have been interesting, but there are alternatives. As I've already self-hosted my mail and some static sites I'll now also move my OSS from Google Code and my Tideland blog to this server. Other services will be more distributed to different providers, like Dropbox and Flickr - with no problem paying for it. I only want reliability.

For the Reader as my RSS aggregator - front-end is Reeder - surely alternatives will rise. And that's good. I'm sure RSS itself won't die. To me the discontinuation gave me the final missing hint for an own special project I'm already thinking about since a longer time. So I'll also develop one alternative, as OSS for self-hosting but also as a provided service. It will be no simple Reader replacement, so stay tuned. (smile)

Is Google doomed now? No, I'm only adjusting my personal strategy. I'll keep GMail as a backup and Google+ because I like this platform. I'll do a spring cleanup like Google too, reduce the informations in my profile and make it more difficult for them to analyze me with the above mentioned changes. And that's positive. Thank you, Google.

Beta testers wanted

It's done, all wanted features for the first release of my Tideland Erlang/OTP Redis Client (ERRC) are implemented. So it's now officially available for beta testing. Next steps here are a usage in an own project -- and so doing also bug fixing and optimization -- and also some documentation.

Those who know my Go Redis Client will see that both are working relative similar. The latter is only more convenient in converting retrieved data even into complex structures. Btw, a short information about my Go libraries: they will soon change their location (the package name won't change, don't worry).

So how getting the Erlang/OTP client? For a first look you can visit http://git.tideland.biz/gitweb/, the repository itself can be retrieved with git clone git://git.tideland.biz/errc.

Any feedback is welcome. Enjoy.

Play it again, Sam

I've made my first experiences with concurrent programming a long time ago with green threads, but only to let some independent parts of my applications run in the background. Nothing very complex. This changed with the discovery of Erlang/OTP in 2007. I don't know how I learned about this language and platform, but it sounded interesting. The elegance of functional programming and this simple way of communicating processes fascinated me, also the principle and the power of the OTP. And so I started to implement some modules for general purpose and for event-driven applications.

In 2009 I as you know discovered Go as another interesting solution and due to my side job as author some articles, talks and a book followed. Beside that I developed a set of packages, some are ports of my Erlang modules, some are new. Now I'm also using Go in my job. And so I lost focus on Erlang.

During the last weeks I digged a bit into my old code archive. You already read about Smalltalk and how I like the language and the environment. But it misses the concurrency I want to use in my software. And so I opened my Erlang treasure box. Some of this stuff is not so elaborated, but other parts are a good base for further development. As I have a project in mind for which Erlang is matching I decided to use this platform for it. But for persistency I won't use the build-in modules. Instead I'll port my Go Redis client to Erlang. Sounds like a typical Not invented here (NIH), huh? Indeed, it is. But the development and maintenance of the Go version makes so much fun and I now know most of the protocol internals that I expect to enjoy the port too.

The project itself with the name Wozzot isn't yet fully outlined, so I'll tell you more details about it later. For now just the information that it will be a kind of wiki or content management system with a non-static linking. Instead the links will be automatically generated based on the content and the tagging, but also dynamically changed based on internal usage and external events from different sources. As said, more about it later.

Sublime Text 2 for Erlang/OTP development

As you know I'm using Sublime Text 2 as my favorite editor for Go development. It is very powerful and configurable and can be extended by a lot of additional packages. They easily can be installed with Package Control, an additional package itself. Now I've installed SublimErl and SublimeREPL for my Erlang/OTP development.

SublimErl adds code completion for all Erlang libs and the current project and an auto-indent, it runs EUnit, Common and Dialyzer tests , allows a quick navigation to any exported function in the code and provides a comfortable navigation to the Erlang man pages. Another nice feature is the autmatic compilation of source files immediately after saving. Here it uses rebar, the Erlang/OTP build tool by basho. The package doesn't create a full-sized IDE, but most tasks can easily be done from inside the editor. Only some still have to be started from the shell.

Here Erlang/OTP provides its runtime system erl as a powerful environment. Wouldn't it be nice to have this available from inside the editor too? SublimeREPL is your friend. It offers the start of a lot of REPL tools, also the start of erl. So s lot of quick tests and hacks can be done inside an editor tab.

Having a good time with an old friend

It's obvious, today's processor and computer architectures require  that developers dive into the world of concurrent programming languages. I've started in 2007 with Erlang/OTP, which still impresses me, and 2009 with Go, which I now also use in my job.

But these days I revovered an old friend of mine, a language I still love: Smalltalk. It's simple, elegant, self-explanatory and powerful. The rich class library allow quick solutions, the metaprogramming capabilities help to create powerful code. Additionally the environments - I'm currently using Pharo - provide a large toolset for editing, refactoring and debugging. So I've reactivated some old sources and also started to port a first one of my Tideland CGL packages. It's fun.

The only feature I'm missing is concurrency for larger solutions. But here I gained knowledge about some ideas and plans that sound very promising. So I'll stay in touch with this wonderful world of Smalltalk.

Happy to announce Tideland CGL Release 2.1.0

I'm happy to announce the new Release 2.1.0 of the Tideland Common Go Library. Beside some fixes, especially for the Redis client,  it contains new packages for buildings Atom and RSS clients and to supervise goroutines almost like in Erlang/OTP.

General

• Changed import path prefixes from code.google.com/p/tcgl to cgl.tideland.biz

Cells

• Added EventCollector behavior

Net

• Added new top-level package net for network related packages

Net/Atom

• Added package net/atom for reading atom feeds

Net/RSS

• Added package net/rss for reading rss feeds

Redis

• Changed connection pooling and error handling
• Now errors are better detected and returned and the pool is more flexible and faster
• Removed deadlock error when number of used connections is larger then pool size

Supervisor

• Added new package supervisor inspired by Erlang/OTP supervisors
• Supervisors run goroutines and other supervisors and restart them after errors or panics
• Hierarchical supervisor trees are possible
• Depending on strategy and restart frequency all supervised goroutines may be restarted or stopped after an error
• As Go has not the monitoring/killing abilities of Erlang the goroutines have to implement a defined signature and stop after a signal of their supervisor tells them to do

Web

• Changed content type of gob-encoded data to application/vnd.tideland.gob

As usual you find it at http://cgl.tideland.biz.

Set up an import path alias

Today I set up an alias for the import path of my Tideland packages. How to do so is described in the documentation of the go command. So I registered cgl.tideland.biz and added the needed index and other files per package. They don't contain very much, only the meta tag

<meta name="go-import" content="cgl.tideland.biz hg https://code.google.com/p/tcgl"/>

This way go get know that the real repository is hosted at Google Code. Beside the nice looking import path I can change the repository hoster if this is needed once.

Now I only have to change the import paths inside the code and push it.

Using Sublime Text 2 with GoSublime

In Convinced by Sublime Text I already wrote about Sublime Text 2, my preferred editor I'm using under OS X and Linux. It's already very powerful out of the box. But together with the Package Control plugin and the large number of available packages it gets even better. Searching, installing, updating, removing, everything works like a charm.

Today I want to show you the GoSublime package by DisposaBoy. It provides

• the well-known code completion of gocode by nsf,
• own snippets for code-completion which intelligent - by type-comparison - try to use variables of the surrounding context,
• simple adding and removing of imports,
• direct jumps to imports, e.g. for defining an alias,
• live error detection and highlighting using gotype; a list can be get with Cmd-. Cmd-E,
• a build system simply based on the Go tools; so not only a go build or go test are possible, also variants like go test -i or go test -test.cpu 8 -test.v,
• error cycling using F4/Shift-F4 (beside clicking on the error in the output window),
• simple jump between the errors in different files and back,
• formatting of the source with gofmt automatically when saving or by command and
• directly run/play the current file.

Most of those commands can be accessed via the Cmd-. Cmd-. shortcut. Some or directly part of the plugin and so developed in Python. But a lot is done in MarGo, also developed by DisposaBoy, but in Go. Both tools are not yet done but grow fast. Every few days the automatic update of Package Control shows new changes and the change log is very interesting. Soon there will be

• a system wide jump to the definition of a function or variable,
• a display of Go docs in the editor and
• a share of code from inside the editor in play.golang.org (it's already working in the developer releases of Sublime Text 2).

 So if you're a Go developer looking for a powerful environment try this duo.

Release 2.0.0 of the Common Go Library

I'm happy to announce the Release 2.0.0 of the Tideland Common Go Library. Most important part is the rework of the cells package for event-driven applications. Thanks to the early adopters of this package and their very useful hints. I hope they all now benefit from the modifications and don't have too much problems with the needed API changes. To give all of you a better introduction in how cells is working and has to be used I added a scenario simulating a shop with ordering, its stock, manufacturing and delivering.

Here are the changes:

Asserts

• Better messages for a direct usage inside of editors
• Renamed 'Substring' and 'Implementor' asserts

Cells

• Larger rework with improvements of reliability, performance and convenience
• Cell adding now lazy using a factory
• Additional multiple cell adding and subscribing
• Emits inside a cell now via an own type instead of a channel
• Integrated configuration

Config

• New package
• Configuration is key/value based
• Backends are configuration providers
• Simple map based provider in this release; more to come

Monitoring

• Simplified output of execution times
• No more total time and theoretical operations per second

You'll find the library at http://code.google.com/p/tcgl/.