Overview

How to lead a software development team: Three simple things worth noting 
JDK8 lambdas and anonymous classes
Why I like Eclipse and sometimes not
Groovy 2.0 Performance compared to Java (also published on java.dzone.com)
STIters: Smalltalk-style collection iterators in Kotlin
Beyond C/C++: High-level system programming in D

How to lead a software development team: Three simple things worth noting

If someone had the wisdom to explain how best to lead a software development team, there would probably be no problem with software development at all. Inarguably, I do not have that wisdom. But looking back on 20 years of software development I must still say that one IT department boss I once had was till now the way best development lead I ever had. I think I can explain why and I believe it is worth being noted down.

So stay tuned, but first take a seat before you read on: That person had little understanding of IT or software development as such coming from a different discipline. Nevertheless, he was still a ways better development lead than many people that had pursued an entire career in IT. Apparently, other factors than understanding of software development are also decisive. My boss back then did nothing but these three simple easily explained things:

• Figure out the people in your team or department that seem to understand their stuff. If you don't have them, look for them and hire them. Never stick to the same person. Always have 2-3 guys to talk to about almost everything and other people to talk to about specific topics. Look for indicators that can tell you who seems to be capable. My observation from several years in software development is that you can find good indicators even when you haven't studied computer science and haven't grown up in IT.

• Have meetings with those people discussing how to do things. What are the problems? What are the obstacles that could become problems? Work out an approach how to deal with these problems and develop a way how to do things. Whenever you see some new obstacles or ambiguities, have a talk with your best people first. Going for a glass of red vine after work sometimes yields much better results than having a meeting here and now.  Also invite one or the other developer that shows potential or willingness to perform to a meeting with the big guys here and then. That's because you need to invest in them and support them as well. Not only the big guys make the difference (relative differences count, not absolute differences). Always invite the people that are assigned to some specific task that is being discussed, anyway. 

• Communicate to all the other members of the team what the idea is in how to do things and what you have discussed with the big shots. Take time and patience in answering to questions. Make people see that you listen and are trying to look for real answers. Sometimes the ideal approach is not possible. That's fine. But explain why you do things this way. Demonstrate a working attitude that allows developers to talk about things. They need not talk about the current project at all times. If they want to they can also have a talk about Scala, Lua, Ruby, Groovy, functional programming, C++ or Java and what the heck. Just demonstrate that open mindedness and thinking about how to solve problems in general is appreciated. When you are done with that, everybody is going back to work on her/his thing, which is not some single isolated programming task, but the thing s/he is responsible for and wants to a good job in. 

Okay guys, that was it! That's the essence of 20 years experience in software development. And the IT department I worked in back then was quite successful. We were one of the few developement departments delivering the software each time - mostly in time and sometimes with a little delay. If I had asked for money for telling you this, you would have asked me to give me your money back. So what's the point? The point is, people don't do these simple three things and what they do cannot be explained in little time or be reduced to three simple things. I would simply call it the "human factor" and not dive any further into it as not everything that is human is worth noting.

Confidence that the three points mentioned above work well and strength of nerves to stick to them in times of pressure can be developed in many disciplines over the years. It doesn't have to be in IT itself. But if you are developing a banking application or a telecommunication application or whatever and the department boss or project lead knows very well the banking business or telecommunication business or whatever, that's always worth a pot of gold :-).

JDK8 lambdas and anonymous classes

Preview releases of the upcoming JDK8 including the long-awaited lambdas are available for several months meanwhile. Time to have a look at lambdas to see what they are and what you can expect from them.

So today, I downloaded the latest preview release of the JDK8 from jdk8.java.net/lambda to have a look at the upcoming lambdas in JDK8. To my despair, this code snippet did not compile: 

List<Integer> ints = new ArrayList<>();         ints.add(1);         ints.add(2);         ints.add(3);         int sum = 0;         ints.forEach(i -> { sum += i; });

 
The compiler error was: "value used in lambda expression should be effectively final". The compiler complains here that the variable sum had not been declared final. Also see this blog post, that is part of the JDK8 lambda FAQ, which explains the matter (I perpetually insist on having found the issue independently from this post ;-)). So lambdas in JDK8 carry exactly the same restriction as anonymous classes and you have to resort to the same kind of workaround:

	int sumArray[] = new int[] { 0 }; ints.forEach(i -> {sumArray[0] += i;});  println(sumArray[0]);

This works and prints 6 as expected. Note, that the compiler did not complain here about sumArray not being declared final as it is effectively final: "A variable is effectively final if it is never assigned to after its initialization" (see link). This is a new feature in JDK8 as the code below does not compile with a pre-JDK8 if value is not declared final:

	final long[] value = new long[] { 0 }; Runnable runnable = new Runnable() {               public void run() {         value[0] = System.currentTimeMillis();         System.out.println(value[0]);     } };

However, this means that JDK8 lambdas are not true closures since they cannot refer to free variables, which is a requirement for an expression to be a closure:

"When a function refers to a variable defined outside it, it's called a free variable. A function that refers to a free lexical variable is called a closure.". Paul Graham, ANSI Common Lisp, Prentice Hall, 1996, p.107.

The free variable gives the closure expression access to its environment:

"A closure is a combination of a function and an environment.". Paul Graham, ANSI Common Lisp, Prentice Hall, 1996, p.108.

In the end we can conclude that JDK8 lambdas are less verbose than anonymous classes (and there is no instantiation overhead as with anonymous classes as lambdas compile to method handles), but they carry the same restrictions as they do. The lambda specification (JSR 335) also says so explicitly: "For both lambda bodies and inner classes, local variables in the enclosing context can only be referenced if they are final or effectively final. A variable is effectively final if it is never assigned to after its initialization.". Here is also a link to an article where Neal Gafter himself (who was a member of the BGGA team) tried to explain why inner classes are no closures (read the comments section). However, all this is only a little tear drop as the usefulness of closures is preserved to a large extend. An imense amount of pre-JDK8 boilerplate code can be replaced with much more concise expressions now. And in the end, you can anyway still do this:

int sum = ints.stream().reduce(0, (x, y) -> x + y);

Nevertheless, the difference between JDK8 lambdas and closures is worth a note as it is good to know. There is a nice write-up about many the things you can do with JDK8 lambdas in this blog post. Here is some sample code from it: 

	List<String> names = Arrays.asList("Alice", "Bob", "Charlie", "Dave"); names    .mapped(e -> { return e.length(); })    .asIterable()    .filter(e -> e.getValue() >= 4)    .sorted((a, b) -> a.getValue() - b.getValue())    .forEach(e -> { System.out.println(e.getKey() + '\t' + e.getValue()); });

We can also reference a static method:

	executorService.submit(MethodReference::sayHello); private static void sayHello() {         System.out.println("hello"); }

The lambda FAQ says about the restriction on local variable capture explained in this article: "The restriction on local variables helps to direct developers using lambdas aways from idioms involving mutation; it does not prevent them. Mutable fields are always a potential source of concurrency problems if sharing is not properly managed; disallowing field capture by lambda expressions would reduce their usefulness without doing anything to solve this general problem.". 

The author is making the point here that immutable variables, like those declared final, cannot be changed inadvertently by some other thread. A free variable referenced from within a closure expression (but declared outside the closure) is allocated somewhere on the heap, which means that it is not local to some specific stack (hence it is free). Being allocated on the heap a free variable can be seen by all other threads as well. This way a free variable can effectively become a variable shared between threads for which access needs to be synchronized to prevent stale data from happening. So the finalness restriction for JDK8 lambdas helps in avoiding that kind of trouble. Note, however, that this is only true for simple types or objects that are not nested as can be seen in the sample code in the beginning of this text with the single-element-array sumArray: the final variable sumArray cannot be re-assigned, but the single element it holds can be changed any time.

Why I like Eclipse and sometimes not

I learned from the comedy movie Borat that a typical way to turn a statement into a humorous one is to append "not" at the end of it. So I did this as well in the title of this article. Admittedly, the main reason being though, that no one would otherwise read an article titled "Why I like Eclipse" ... ;-).

I often happen to meet people in projects that are really into NetBeans or IntelliJ IDEA and not into Eclipse at all. These people don't understand why someone like me would work with Eclipse (I also use IntelliJ IDEA quite a bit). The problem is here that explaining why I like Eclipse results in a long talk which is first about Eclipse background knowledge that demands a lot of patience and distracts people for too long time from their work. Secondly, a long talk is followed about why I feel very productive when good code browsers as in Eclipse are at my disposal. So I'm trying to explain it in this little article once and for all for the benefit of the world (eventually, you need to append again "not" at this place). Don't worry, it's not going to be one-sidedly as I will also talk about the things in Eclipse that are not that amusing. It's merely about code browsers and their differences than specifically about Eclipse.

Code Browsing

The real reason I like Eclipse is its powerful code navigation and code browsing capability only comparable to the code browsing its big idol which is the excellent Smalltalk environment. I'm willing to sacrifice a lot of other things as long as I have that. Let me quote Bertrand Meyer: “Smalltalk is not only a language but a programming environment, covering many of the aspects traditionally addressed by the hardware and the operating system. This environment is Smalltalk’s most famous contribution”. [1] This statement includes that Smalltalk not only is a language with an IDE on top, but a computing environment as such. This coherence has gone lost with Eclipse which has good and bad consequences. But this is a different topic, too long to talk about it here as well. The people that have worked with Smalltalk understand what this means. But the people that have not, only gaze fixedly at you for a moment and then continue working. So my contribution in this article is aimed at explaining what this is about. Earlier, people often have heard about Smalltalk and are willing to listen for a while. Nowadays, you have to say something like "Smalltalk is the system that had closures from the beginning already in 1980 from which later this clone was made starting with a 'J'". Otherwise people would not even stop coding for a second. Or you have to say something like "Smalltalk is the system Steve Jobs was looking at when visiting Xerox Parc (see also this article about Xerox Parc) when he said that this is the way he wants the user interface to be on the computers he is producing" (user interface with icons, moveable windows that can be collapsed, and a mouse).

What's the catch about excellent code browsing capability then? Problem is that when your code starts to grow, you somewhen reach a point where it is hard to keep oversight. Well, that is was structured programming is for: you can structure your code and then there is abstraction, information hiding, modularity, inheritance, polymorphism and more. But somewhen you can't remember any more in which class what method was placed and it is sometimes still hard to keep oversight even with abstraction and all that. I have seen people that are nevertheless able to understand their code very well using simple development tools only. Therefore, I agree that you don't necessarily need to have an IDE with good code browsers. For some people it's a necessity. For others it's a matter about comfort and maybe also developer productivity.

Eclipse’s heritage from the Smalltalk IDE

Eclipse was developed by the people of a company named OTI in Ottawa, Canada, that used to develop and market the other big Smalltalk development system at that time in the market (besides ParcPlace Smalltalk, now Cincom Smalltalk) which was first Envy/Developer and then OTI Smalltalk (I’m not sure about the name here). When the development of Eclipse started OTI was already acquired by IBM as IBM wanted to sell OTI’s Smalltalk system as IBM Smalltalk as a replacement for their failed CASE tool strategy. The product named IBM VisualAge for Smalltalk was also very successful (especially in the finance sector) at a time where there was only C++ and Smalltalk for serious production quality OO development. Later Java came along and IBM abandoned its Smalltalk system, sold it to Instantiations and jumped onto the Java train developing IBM VisualAge for Java. VisualAge for Java was very much like the Smalltalk IDE only the programming language being Java: It was an interactive development environment where almost any statement could be selected and executed at development time. You could look at your data in inspectors in which you also could send messages to objects dynamically at development time. From what I have heard VisualAge for Java itself was developed in IBM Smalltalk, but I cannot provide evidence for this. This was IMHO a very productive development environment and everything was fine as long as your application only consisted of the code you were writing. But then web development came along and this was no longer true as now, beside source code files, a plethora of all other kinds of file types came into play: html, jsp, xml, css, jar, war, ear, and much more and they all have to be bundled together. The latter was as much a problem for Smalltalk’s/VA Java’s approach to create a runtime package as the former. So VisualAge for Java was abandoned and Eclipse was developed. If you managed to get to this line, the bits of Eclipse history I had to provide are now behind you ;-).

Code Browsing in Eclipse

So far I have not mentioned why code browsing in Eclipse is so fantastic (let’s say it is better than in many other IDEs at least). There are different browsers for different things. If you are working on code files only you can use the "Java Browsing" perspective. You see the packages and their classes of your project at a glance and everything else is removed. You can still have the "Java" perspective  where your Java code and all the other types of files are visible at once. You can have all the browsers you work with side by side each in a window of its own. Select Window > Prefrences > General > Perspectives > Open a new perspective and select "In a new window". From now on every perspective you open will open in a new window of its own. Most people working with Eclipse I have seen don't know this feature at all. But this is the usual way the Smalltalk IDE was intended to be used. Then Eclipse has an equivalent for the Smalltalk class hierarchy browser. It is also not activated by default. To do so you have to go to Window > Prefrences > Java > When opening a type hierarchy and select "Open a new Type Hierarchy Perspective". I always found this browser to be very useful when working on an abstract class and some of its concrete subclasses, because you can really concentrate on just that what matters in that regard.

I once had a situation where Eclipse ran out of memory, which was probably caused by the memory demands of the OSGi implementation when building from within Eclipse. But because of me using several browsers at the same time in Eclipse as I used to do it earlier when developing with Smalltalk, some colleague was absolutely sure that having that many browsers open consumed too much memory. When I switch between perspectives that are displayed in the same window, memory remains allocated for all of them just the same way as when they are opened in their own window. No way you could switch between perspectives that quickly, otherwise. But that argument just didn't fly. Some people are that used to just working with a single window IDE that anything different appears simply weird to them.

And why I sometimes don’t like Eclipse

Eclipse provides a solid base to place an IDE on top of it for all kinds of things. Its Java plugin is also very useful. But it is not always as good at specific tasks such as code completion (IntelliJ IDEA is IMHO awesome here), refactoring (needless to say that "Refactoring was conceived in Smalltalk circles” [2]), “intra-file navigation” (jump from some JSF xhtml statement to the underlying Java code, etc.). It does not have an excellent Swing IDE builder such as NetBeans. When you develop a web application all the plugins that come into play are not nicely integrated and concerted as in IntelliJ IDEA. Also MyEclipse does not do much about this in the end. The weakness of Eclipse in short is that it stops after providing a plugin platform and a Java plugin. From then on every one is left to his own devices. A lot of nice people have developed very respectable plugins for all kinds of things, but they miss “calibration” with related plugins rendering them isolated solutions.

Then Eclipse has become sluggish and sometimes irresponsible. I’m not amused how often Eclipse is irresponsible and I have to wait till it’s responsive again. I don’t know exactly what the reason is in every case. Maybe just someones plugin is not well written and is causing this. Whatever, as already said, other IDEs don’t have this problem as all the plugins that come into play are inter-coordinated and tastefully furnished. 

Last but not least, at the time of writing (21st October 2012) Eclipse has still no support for JDK8 lambdas and default methods.This is because Eclipse’s Java compiler is built into Eclipse and cannot be easily separated (you can define a custom builder for your project which will call the javac of the JDK you defined. But the JDT will still not be able to deal with JDK8-style lambdas). So the whole thing has to be exchanged. This is probably some heritage from Smalltalk as well where the whole thing was a single system. Earlier at that time, this was unmatched coherence (compared to piping in a myriad of little Unix tools). Nowadays it's considered inflexible and monolithic. I use IntelliJ IDEA 12 EAP for my current little spare time JDK8 lambda project. So far there was never a problem to get any lambda expression compiled and to run. Simply amazing.

Last and least, I really wished NetBeans and IntelliJ IDEA had a class browser in addition like the one in Smalltalk or something like the “Java Browsing” perspective in Eclipse. When you are working on code only and no html, xml, css, or whatever files are part of your application, IMHO, there is nothing like it. But in todays world there is no way to develop an application without any xml (or json nowadays), for example. But I'm convinced there is a way to merge the best of Eclipse/NetBeans/IntelliJ IDEA with the best of the Smalltalk IDE.

1. Bertrand Meyer, Object-oriented Software Construction, Prentice Hall, 1988, p.439.

2. Martin Fowler, Kent Beck (Contributor), John Brant (Contributor), William Opdyke, don Roberts, Refactoring: Improving the Design of Existing Code, Addison-Wesley, 1999, p.6.

 

 

Groovy 2.0 Performance compared to Java

Also published on java.dzone.com, see link.

End of July 2012 Groovy 2.0 was released with support for static type checking and some performance improvements through the use of JDK7 invokedynamic and type inference as a result of type information now available through static typing.

I was interested in seeing some estimate as to how significant the performance improvements in Groovy 2.0 have turned out and how Groovy 2.0 would now compare to Java in terms of performance. In case the performance gap meanwhile had become minor or at least acceptable it would certainly be time to take a serious look at Groovy. Groovy is ready for production for a long time. Let's see whether it can compare with Java in terms of performance.

The only performance measurement I could find on the Internet was this little benchmark measurment on jlabgroovy. The measurement only consists of calculating Fibonacci numbers with and without the @CompileStatic annotation. That's it. Certainly not very meaningful to get an overall impression, but I was only interested in obtaining some rough estimate how Groovy now compares to Java concerning performance.

Java performance measurement included

Alas, no measurement was included in this little benchmark how much time Java takes to calculate Fibonacci numbers. So I "ported" the Groovy code to Java (here it is) and repeated the measurements. All measurements were done on an Intel Core2 Duo CPU E8400 3.00 GHz using JDK7u6 running on Windows 7 with Service Pack 1. I used Eclipse Juno with the Groovy plugin using the Groovy compiler version 2.0.0.xx-20120703-1400-e42-RELEASE. These are the figures I obtained without having a warm-up phase:

	Groovy 2.0 without @CompileStatic	Groovy/Java performance factor	Groovy 2.0 with @CompileStatic	Groovy/Java performance factor	Kotlin 0.1.2580	Java
static ternary	4352ms	4.7	926ms	1.0	1005ms	924ms
static if	4267ms	4.7	911ms	0.9	1828ms	917ms
instance ternary	4577ms	2.7	1681ms	1.8	994ms	917ms
instance if	4592ms	2.9	1604ms	1.7	1611ms	969ms

I also did measurements with a warm-up phase of various length with the conclusion that there is no benefit for neither language with @CompileStatic or without. Since the Fibonacci algorithm is that recursive the warm-up phase seems to be "included" for any Fibonacci number that is not very small.

We can see that the performance improvements due to static typing has made quite a difference. This little comparison does not make up for an only little ambitious performance comparison. But to me the impression that static typing in Groovy in conjunction with type inference has led to significat performance improvements, the same way as with Groovy++, has become very strong. With @CompileStatic the performance of Groovy is about 1-2 times slower than Java and without about 3-5 times slower. Unhappily, the measurements for "instance ternary" and "instance if" are the slowest. Unless we want to create master pieces in programming with static functions, the measurements for "static ternary" and "static if" are not that relevant for most of the code with the ambition to be object-oriented (based on instances).

Conclusion

The times where Groovy was somewhat 10-20 times slower than Java (see benchmark table almost at the end of this article) are definitely over whether @CompileStatic is used or not. This means to me that Groovy is ready for applications where performance has to be somewhat comparable to Java. Earlier, Groovy (or Ruby, Closure, etc.) could only serve as a plus on your CV, because of the performance impediment (at least here in Europe).

New JVM kid on the block: Kotlin

I added the figures for Kotlin as well (here is the code). Kotlin is a relatively new statically typed JVM-based Java-compatible programming language. Kotlin is more concise than Java by supporting variable type inference, higher-order functions (closures), extension functions, mixins and first-class delegation, etc. Contrary to Groovy, it is more geared towards Scala, but also integrates well with Java. Kotlin is still under development and not officially released, yet. So the figures have to be taken with caution as the guys at JetBrains are still working on the code optimization (see KT-2687). Ideally, Kotlin should be as fast as Java (see this post). The measurements were done with the current "official" release 0.1.2580.

And what about future performance improvements?

At the time when JDK1.3 was the most recent JDK I still earned my pay with Smalltalk development. At that time the performance of VisualWorks Smalltalk (now Cincom Smalltalk) and IBM VA for Smalltalk (now owned by Instantiations) was very well comparable to Java. And Smalltalk is a dynamically typed language like pre-Groovy 2.0 and Ruby, where the compiler cannot make use of type inference to do optimizations. Because of this, it always appeared strange to me that Groovy, Ruby and other JVM-based dynamic languages had such a big performance penalty compared to Java when Smalltalk had not. Well, coming to think about it: Hot Spot runtime optimization in Java was taken from Smalltalk, anyway (see this article). Nothing beats the arrogance of a Smalltalk developer, even not a Mac enthusiast... Seems like creating a JVM-based language is easier than optimizing it's byte code. From that point of view I think that there is still room for Groovy performance improvements beyond @CompileStatic.

STIters: Smalltalk-style collection iterators in Kotlin

Kotlin is a new kid on the JVM block. It is a relatively new statically typed JVM-based Java-compatible programming language much in the spirit of Scala. It is more concise than Java by supporting variable type inference, higher-order functions (closures), extension functions, mixins and first-class delegation, etc.  (see the Kotlin FAQ). It compiles at least as fast as Java which was one of the main reasons to abandon Scala besides making Kotlin way simpler than Scala. Being developed by JetBrains it inherits the excellent IntelliJ IDE (for which a Kotlin plugin is included in the free community edition) making you very productive writing Kotlin code.

I like the approach in Kotlin where the language designers first strive for a solution to add new functionality to the library if possible and only add it to the language in case adding a library function delivers unsatisfactory results. This is IMHO a good approach to keep a programming language coherent and clear. Basically, this is why I stopped looking into Scala some time ago and then was glad when I discovered Kotlin as Kotlin retains many of the good things in Scala.

Lately, I sat down to add my dearly missed Smalltalk collection iterators to Kotlin as a little Kotlin programming exercise (see stiters).This was good fun as extension methods in Kotlin are intuitive as they should be (basically same thing as adding a method to a class) and with the use of closures simple things remain simple as well, e.g.:

public inline fun <T> Collection<T>.select(condition : (T) -> Boolean) : Collection<T>

{

    // for brevity no source provided for newInstanceNotNull(); look at stiters for more details

    val result = newInstanceNotNull()   

    for (item in this) {

        if (condition(item))

            result.add(item)

    }

    return result

}

It is easy to see, that in the extension method above a method named select is added to the Collection class. Whereas for example in C# and Ceylon the same method would roughly look like this:

public static Collection select(this Collection input)

{

    // method body omitted for brevity

}

Here the receiver object, which is a collection, is handed in as a parameter (denoted by the this pointer in the argument list). Hence, receiver object and method parameter are swapped compared to "regular" instance methods (and this is not really intuitive).

This is how to use the select method in Kotlin:

fun main(args : Array<String>)

{

    val ints = HashSet<Int>()

    ints.add(3)

    ints.add(7)

    ints.add(12)

    ints.add(23)

    var selectResultSet = ints.select{ it < 12 }

    println(selectResultList)  // prints [3, 7]

}

Check out the stiters page for more details about this little library written in Kotlin to bring Smalltalk-style collection iterators to it.

Beyond C/C++: High-level system programming in D

I spent a lot of time the last months thinking about what language to have a look into besides Java. There are so many other interesting languages like Ruby, Groovy, Scala, Clojure, Ceylon, Kotlin, etc. And whatever choice you make has quite some implications.

Now I got fascinated by a language very different from those just mentioned, which is intended for "high-level system programming". It is simply called "D". As the name suggests D is intended to be a successor language to C/C++. The language creator Walter Bright has spent much time of his career developing C and C++ compilers (for Datalight, Zortech, Symantec) and has therefore a very good understanding of issues with C/C++. Out of that experience and understanding he sat down to create a language claimed to be a "modern C/C++". D is some kind of Scala for system programming. It has almost all the features of Scala and other JVM-languages such as generics, metaprogramming, closures, true mixins (called template mixins), actors, functional propgramming extensions. But it is made for high performance and is not overloaded with, IMHO, too many features like Scala. Since D does not carry the burden of header files like C/C++, build times are also quite fast (I heard different things from Scala).

I recommend the book "The D Programming Language" by Andrei Alexandrescu which is a very good read where he explains what is the rational behind the various language constructs in D. The book is written in a very objective and competent way (without any stupid C/C++ bashing): this guy is a true engineer. I don't know whether I will be able to make any time for some spare time programming in D. (Doing some D programming is something I have added to my to do list for the day I win in the lottery and can do the entire day just what I would like to do). But reading the book by Alexandrescu has improved a lot my understanding about things in C/C++ and system programming as such, which has surely raised my horizon about programming in general.