EuroClojure 2012 Day 1

So there were definitely two big themes in the talks on the first day of the conference.

The first has been about how to use event-based systems to create flexible aggregate data models. All speakers seem to have settled on a reduce or foldLeft approach for creating the aggregate but there have been two models put forward already CQRS and a kind of Aggregate query bus but really it seems that responsibility for accepting event data and allow querying and access to aggregated views seem to be responsibilities in the same system.

The other thing has been creating query systems using logical predicates. The were no less than three generic query systems put forward: core.logic for low-level flexible implementations that identify either data or results and two general query libraries: one from Datomic and the other from Cascalog.

January’s London Clojure Dojo

January meant Battleships. More specifically battling battleships. Five teams created players and duked it out during the dojo with a tremendously narrow margin of victory. So what did we learn?

Well first of all randomly placing ships and shooting is actually a pretty good strategy. This is what the default player does and any deviation from it can be pretty badly punished by it.

One simple thing that people did to start improving over the random start was restricting placement of ships to a single half or quarter of the board. Doing this allowed most teams to start beating the initial strategy.

However clustering your ships is only effective against random shot placement so when people start implementing targeting you actually become more vulnerable. The first effective targeting strategy was surprisingly simple, if you hit something choose an adjacent square as your next target.

The team that squeezed to the top refined this by choosing an adjacent square that hadn’t already been fired at. The next level of improvement would probably be a non-trivial look at the probability that another ship square lay in the adjacent squares by looking at the information surrounding them.

There was a lot of work around the concepts of adjacency and whether the square had been fired at and the teams all seemed to converge towards the clojure.set library (if they were aware of it).

I’m now thinking of what fiendish problem would force and exploration of this library as it seems incredibly powerful for all different kinds of problems.

Why I’m finding Clojurescript underwhelming

I noticed Clojurescript in Github before the big announcement and thought it was an interesting idea. I am a big fan in general about having a Clojure syntax that compiles to Javascript. As a platform it is even more ubiquitous than Java and it would be a great way of simplifying Javascript’s closure and function syntax.

However in practice Clojurescript has been desperately disappointing for me. Firstly there is the weird decision to not have the code run on OpenJDK. This really limits its utility: I don’t seem to have a machine with a compatible setup at the moment despite having various flavours of Javascript interpreters available.

Then while looking for an answer as to how soon this problem is likely to be resolved I discovered this thread which was another level of disappointment. The original post is undiplomatic, perhaps even inflammatory, however the response indicates a level of befuddling clueless-ness.

If you want something to compile into Javascript I think you actually do want it to compile into good idiomatic Javascript unless you have a really good reason not to. You also do want to be able to use really good existing frameworks like jQuery (which really is the defacto standard right now).

The reason I think these are reasonable requests is that Coffeescript seems to manage to do both. Before Coffeescript maybe Clojurescript’s idiosyncrasies would have been forgiveable but being late to the party as well as being less well-mannered makes the defiance in the response seem poorly judged.

I am not sure what Clojurescript is really for (apparently it is aimed at a future community of people that don’t exist yet, which is … helpful). I don’t feel that it is really simpatico with the existing Javascript code that works in the browser and I am not sure it really has a place in the server-side world of Node.js where it might have been a better fit.

I remain open-minded though and would be willing to give Clojurescript a second go once the dust has settled a bit.

Intuitive versus Reasoning Programmers

During the last year I’ve been helping run a monthly series of dojos for the London Clojure User Group. In the course of it I have had the chance to watch a lot of people grapple with functional programming. As a result of this and also looking at the way a lot of my colleagues at ThoughtWorks work I think programmers can roughly be divided into two groups: Intuitive and Reasoning.

To characterise both of them a little, I think that Intuitive programmers tend to use domain language a lot, rely heavily on tests and TDD, often find it difficult to articulate what they are doing in their work, they like small source code files because they like to see everything a file does at a glance, they prefer outside-in problem solving and like the opportunity to go back and revise their work.

Reasoning programmers like to discuss a problem before coding and explore edge-cases and what-ifs, they like to work in the REPL or have in-editor code evaluation, they quickly move from a domain to an abstraction and then work in that abstraction, they don’t mind a thousand line source file as long as it is logically structured (that’s what the search function is there for), they prefer “bottom-up” coding where they distil their abstraction to its essence and having implemented that create the required behaviour by composing their abstractions.

Both sets of programmers can be great at their work but often if they are unaware of the characteristics of how they like to work then there can be massive amounts of tension as the two wrestle back and forth. The Intuitive developers feel angsty when the Reasoning programmers start hacking out code rather than refactoring, Reasoning developers get frustrated at the aimlessness and game playing of the Intuitive’s attempts to generate the minimum amount to make their tests pass.

Reasoning programmers probably feel that in terms of communicating they have the superior style as they are able to advance arguments and logical constructs that can be interrogated. Those articulated models though can feel arid and irrelevant to Intuitive programmers, what does some obscure mathematical formula have to do with trying to make sure the frog doesn’t get run over when it crosses the road?

Intuitive programmers seem to be better at switching contexts and adapting to change as they can quickly see the “outlines” of any problem and their techniques are about honing that initial perception into a functional solution. By contrast a Reasoning program is wary of uncertainty and is unhappy drawing unjustified analogies between different situations.

In FP terms Reasoning programmers have their behaviour emerge as a logical consequence of the operation of lower level abstractions; their code looks like algebra and domain data is passed in to the top of their processing chain. Intuitive programmers on the other hand, fix behaviour in their tests and fill their code with domain language that aims to match the natural language of the organisation, the depth of their function calls is usually smaller and they are swifter to bind calculations to intermediate variables.

I think I am an Intuitive programmer (and therefore I worry that I am miscasting Reasoning programmers through my lack of understanding) and in my work most of my colleagues are as well, it is in the nature of consultancy to have to adapt to constantly changing domains, code bases and expectations. We do have a few Reasoning programmers though who do exactly the same work but do it via thinking deeply through problems and drawing logical inferences.

If there are issues between the two groups then I think it occurs when there are no concessions between the two; common flashpoints being testing, writing defensive/guard code and giving time to discuss problems and problem solving strategies. The two also agree on a lot of things for very different reasons: for example both like to rewrite code, refactoring is a formalised practice for Intuitive developers whereas Reasoning developers often want to apply new insight or learned ideas (“This could all just be monads!”).

An important point is that neither approach is “right” both types of programmer arrive at the same results if their experience, ability and other factors are equal. They are purely styles of working.

Dojos: One conversation in the room not in the driving seats

Bruce Durling and I had a conversational meeting/talk last week about doing a Clojure dojo here in London for the last year. One of the attendees was Nicholas Tollervey the organiser of the Python dojo. In the course of discussing formats there was an interesting discussion about the concept of “one conversation”, which is to say that people stand a better chance of understanding something is there is just one conversation going where everyone is contributed to the shared understanding rather than several conversations where nothing may be getting understood. Nicholas had read the original Paris dojos and interpreted their rules as saying that the conversation is entirely owned by the pilot and co-pilot and that the audience cannot contribute.

My understanding (this may be a Brazilian addition to the Paris method) and the way I would practice dojos is that all the participants are in the conversation but that once a point or question has been raised by the audience then that issue must be addressed before the coding pair continue. So for example if an audience member doesn’t understand a piece of code or a function they should ask what it means and then the coding pair pause and an explanation is explored until the person asking the question feels they understand what is happening. The coding pair then resume.

With that said there are some important points of concern in terms of facilitation. Firstly the coding pair should be shielded from barracking, “Why are you doing it like that?” can be a good a question but is rarely helpful. A better question would be “Are you trying to solve the problem with method A? Would method B be better?”. But the best type of this question is not asking it, there may be a better way to do something but watching people explore an alternative route might be informative in its own right, even if it ultimately confirms your own views.

Questions should be “points of order”, ideally based on facts not opinion and aimed at clarifying understanding. Philosophical points of view are best expressed at natural break points in the coding or down the pub.

Once a question has been asked the focus of the conversation moves to the person asking the question. This often means that the person asking the question feels a tremendous amount of pressure to say that they understand something and allow the focus of conversation to move on. I know I have often been frustrated and embarrassed with myself for not getting a point. However if you don’t really understand something it is important to say so as it is likely that other people in the room feel the same way and that those trying to furnish an explanation have not done so satisfactorily and need to try again. Facilitators need to keep the safety up here if the group struggles to be supportive.

A final point is that every once in a while you get the really good question. The one that is either going to take a massive amount of effort to answer, that is fundamental to the problem or that lacks a definitive answer. Some people regard these questions as rabbit holes and while it is true that you can often kiss the coding goodbye for the rest of the night it is often these questions that lead to the most memorable moments of a dojo. I remember this happening when Ola Bini gave a spontaneous lecture on Ruby’s object lifecycle event hooks. It started with someone using an unusual technique and it ended up being a really enlightening trip through the guts of Ruby.

Dojos are primarily about learning and these side trips are as important as powering down the highway.

Clojure metadata, the compromise solution

One thing I think is really great about Clojure is the deep reflection and runtime manipulation capabilities that are on offer (it is one thing that I really miss in Scala for example). Last night at the London Clojure Dojo (website needs to be created this month I feel) we needed to distinguish a subset of functions. Now ideally we should have put this subset in a different namespace but you know code that is written by many hands doesn’t always make sense and now we are where we are.

We knew it was possible to examine all the functions in our namespace and that the relevant functions would take two arguments with particular names (we are at least consistent in that respect). My initial view was “hell, lets just find every function taking the the right names”, at the other end of the spectrum were people who thought we should explicitly declare the list of valid functions. In the end we settled on adding some metadata to the relevant functions.

This seemed a reasonable compromise, it means that applicable functions are explicitly defined but that in terms of code maintenance we only have to write one function that generates a list of all the valid functions and this will correctly respond to changes in the code as they are made.

This led to another conversation at work about whether a function’s metadata is mutable or not. Now it seems that as map it should be possible to edit it but it’s an interesting question.

Control expressions are functions too

I am kind of simultaneously learning Clojure and Scala, the former through the London Clojure Dojo (come and join us) and the latter through work. It is an interesting experience but I am very glad that Clojure is part of the mix as it helps understand a lot of things that are seem strange in Scala.

In Clojure everything is essentially either a function or a sequence. It is therefore not surprising to see that an if statement works like a function.

(if false 1 2)

Evaluating the above in a Clojure REPL results in the answer 2: if is a function that evaluates its first argument returning the second parameter if the evaluation is true, the third if false.

The same is true of Scala, with the additional wrinkle that if the different clauses evaluate to different types you could be in type hell. Despite its superficial similarity to Java syntax it is in fact quite different and you can compose it just as you would any other function.

1 + (if(true) 2 else 3)

Evaluated in a Scala REPL gives the result 3.

Scala 2.8 seems much better about making everything return a value (and hence act like a function), val and for will both now return Unit, which is not useful but prevents the compiler moaning.

This kind of thing is much easier to appreciate and learn in a pure functional language than in Scala where you never really know whether something is going to operate in a functional or object-orientated way.

Continuous Testing

Continuous testing is one of those things that has crept up on me slowly. About two years ago I was aware of people using a trigger on their TextMate save to run tests and, if green, commit to git. At the time it felt too much effort for too little gain but it was a cool trick.

Now as we forage out into the post-Java world we are starting to get some pretty cool revisions of familiar tools and one of the most engaging for me are continuous build tools. The daddy is clearly SBT, which while simple is also tremendously sophisticated. Adding a REPL to a build is a simple change but has all kinds of nice consequences, my favourite of which currently is the continuous test (~test) target that detects changes in your source and test files, compiles them and runs your tests. SBT cuts out the whole compile-link-run cycle for you, you just make a change, hit save, see the consequences and code again. It’s very fast and far more effective in giving feedback than any of the current IDEs (all of which need to get on this bandwagon fast is they want to stay relevant).

Clojure by comparison has been suffering in this regard with Leiningen becoming an unfortunately early defacto standard despite it standing shoulder to shoulder with the benighted Maven. The key thing that Leiningen does wrong is stay at the command-line and force you to cold-boot a JVM with each new command (the second is dependency resolution, SBT favours Ivy). Fortunately Lazytest by Stuart Sierra can hopefully save us here. Although still alpha Lazytest is an awesome way of developing Clojure and it’s hard to beat that feeling of smug satisfaction as the tests go green.

It is these kind of step change enhancements in development that are going to carry us forward more than shopping list of features that the new languages have or lack.

Metaprogramming with Clojure

So at one of the recent Clojure dojos we had a situation where we had a number of functions to do with moving that essentially involved passing different keys to a map under different symbols that could be used in the REPL.

Well in Ruby this is the kind of thing you would do by metaprogramming abstracting the shared code into a single function that gets mapped under many names.

During the dojo Tom Crayford provided a map function that seemed to correctly generate the functions we wanted but did so under anonymous names. We couldn’t lick the problem during the dojo and it really drove me a little mad as it seemed we were very close. Some people at the dojo were talking about macros but it seemed that was too strong for the short distance we had to cover to complete the task. It also felt like there was an issue with the language if it couldn’t do this.

After four hours, much consulting of the Halloway book and some fierce googling, a question at Stack Overflow put me on the trail on intern. Adding this to Tom’s function resulted in the right functionality. Here’s a simplified example. However when I ran the code into the REPL the functions failed to appear.

Clojure namespaces are very dynamic and it seems they are very amenable to the kind of manipulation I wanted to do. ns-interns gives you a list of the functions that are currently in a namespace so it was possible to confirm that the functions really had failed to appear. Running the code in the REPL did add them though. So the code was correct.

The final piece of the puzzle was the lazy sequence issue that had been mentioned at the dojo. The map function was being immediately evaluated in the REPL but I had to add a doall to make the sequence unwind when it was invoked during the namespace loading.

The relevant code is part of my Clork fork.

The worst thing in trying to make this work is the old school nature of the Clojure documentation. A lot of the functions tells you literally what the function does but not why you might want to use or more critically give examples of the expected usage. Clojure supports documenting metadata but I think it really needs to be used more effectively.

The Java Developer’s Dilemmia

I believe that Java developers are under a tremendous amount of pressure at the moment. However you may not feel it if you believe that Java is going to be around for a long time and you are happy to be the one maintaining the legacy apps in their twilight. Elliotte Rusty Harold has it right in the comments when someone says that there are a lot of Java jobs still being posted. If you enjoy feasting off the corpse then feel free to ignore the rest of this post because it is going to say nothing to you.

Java is in a tricky situation due to competition on all fronts. C# has managed to rally a lot of support. Some people talk nonsense about C# being what Java will look like in the future. C# is what Java would like if you could break backwards compatibility and indeed even runtime and development compatibility in some cases (with Service Packs). C# is getting mind share by leapfrogging ahead technology-wise at the expense of its early adoptors. Microsoft also does a far better job of selling to IDE dependent developers and risk-adverse managers.

Ruby and Python have also eaten Java’s lunch in the web space. When I am working on web project for fun I work with things like Sinatra, Django and Google App Engine. That’s because they are actually fun to work with and highly productive. You focus on your problem a lot sooner than you do in Java.

The scripting languages have also done a far better job of providing solutions to the small constant problems you face in programming. Automating tasks, building and deployment, prototyping. All these things are far easier to do in your favourite scripting language than they are in Java which will have to wait for JDK7 for a decent Filesystem abstraction for example.

Where does this leave Java? Well in the Enterprise server-side niche, where I first started to use it. Even there though issues of concurrency and performance are making people look to things like Erlang and JVM alternatives like Scala and Clojure.

While, like COBOL and Fortran there will always be a market for Java skills and development. The truth is that for Java developers who want to create new applications that lead in their field; a choice about what to do next is fast approaching. For myself I find my Java projects starting to contain more and more Groovy and I am very frustrated about the lack of support for mixed Java/Groovy projects in IDEs (although I know SpringSource is putting a lot of funding into the Eclipse effort to solve the problem).

If a client asks for an application using the now treadworn combination of Spring MVC and Hibernate I think there needs to be a good answer as to why they don’t want to use Grails which I think would increase productivity a lot without sacrificing the good things about the Java stack. Companies doing heavy lifting in Java ought to be investigating languages like Scala, particularly if they are arguing for the inclusion of properties and closures in the Java language spec.

Oracle’s purchase of Sun makes this an opportune moment to assess where Java might be going and whether you are going to be on the ride with it. It is hard to predict what Oracle will do, except that they will act in their perceived economic interest. The painful thing is that whatever you decide to do there is no clear answer at the moment and no bandwagon seems to be gaining discernible momentum. It is a tough time to be a Java developer.