Jan Willem Tulp’s blog

Last week 2 colleagues of mine and myself gave a Scala workshop at Logica. One of the participants was asking whether getDate was a function or not. His question led to an email discussion today about what functions are, and how they relate to the Command Query Seperation (CQS) principle. The definitions coined in Eric Evans book about Domain Driven Design were helpful in defining the definitions.

After several exchanging several emails, we finally came to the following definitions:

• Query Functions: functions that retrieve information about the system. They are not pure functions, because applying these kinds of functions to its arguments may result in different values. GetDate() would be a Query Function, because everytime it is applied, a different value is returned.
• Command Functions: functions that alter the state of the system. They should not return a value (void in Java, or Unit in Scala). setName(String name) in Java would be example of a Command Function. Command functions therefore have side-effects.
• Pure Functions: functions that always return the exact same value for the exact same input. These are mathematical functions and don’t have side-effects. In Scala the following function is a Pure Function: def inc(x: Int) : Int = x + 1. For the input 4, this function will always return 5, no matter how many times you invoke it, and it doesn’t change the state of the system.

  What’s your opinion about these definitions?

  • 

Most Java developers know static code analysers like Checkstyle, Findbugs, or FXCop for the .Netters amongst us. And these tools are extremely helpful for pointing out the spots in your code that may need some improvement.

However, people must be motived to use such tools, and in real-life this is not always the case. After all, quite often people are not really interested in code quality. Especially since for most non-programmers code quality is something that happens ‘behind the scenes’. “We will worry about code quality when we have time for it… now we’ve just got to get our app up and running, right?”. Wrong!

Quality matters, it will make you a happier programmer both in the short run and in the long run. You will spend less time working on high quality code than on low quality code. It makes your application more durable. Software that lasts, that is easy to understand, flexible to use. But not everybody has this high quality bling-bling mindset. So, how do you motivate people to care about quality?

The answer is: let emotion kick in!

The lessons that can be learned from good presentations, enjoyable movies, and successful marketing strategies is that emotion makes people care. Remember your greatest movie. Isn’t it true that you can really identify with the main characters? Didn’t you forget the time when you were watching that great presentation? Read for example Made To Stick to learn about 6 elements of why some ideas survive and others die.

So, how to make code quality emotional? Should we all be crying? Laughing? Perhaps… but there are more options…

One of the things that works extremely well on the project I am currently working on is using competition. We use Hudson as our continuous integration server. Hudson allows you to use plugins to enhance the capabilities of Hudson. One of these plugins we use is the “Continuous Integration Game plugin“. This plugin gives you positive scores for successful builds, passing unit tests, fixing problems, etc., and it awards you negative points for breaking the build, failing unit tests, introducing warnings etc.

Since we have been using this plugin all team members want to win! They want to get the highest scores, so everybody is looking for opportunities to improve the code, to fix that warning before checking in, or get rid of that duplicate code. And of course, getting the highest score may seem to be the goal for some people, at least a very positive side-effect of this emotional involvement is that code quality does actually get better and better! It is also very good for the team spirit, believe me, lots of laughs!! And you may be wondering, how is this emotional? Well, since all team members get personally involved and are praised personally for high scores, people are emotionally involved.

So my advice: try to use some emotional aspects like a competition in your team so that people really do get motivated to work for example on quality. And even though people may be far more interested in the score, you still have the side-effects of lots of fun and producing high code quality in your team.

Every now and then I run into code where use of abstract classes are not always as optimal as can be. The thing at hand here is reuse of common functionality. There are at least two common ways to reuse common functionality:

1. calling methods in an abstract superclass
2. calling methods that are delegated to some other class (not a superclass)

Even though both options may have the same final result, I believe that one option is better suited for some situations, and the other for other situations. Notice that when you are calling methods in an abstract superclass, the direction is up: from the subclass to the superclass. In many cases the subclass does the heavy work, and reuses some common functionality from the superclass.

Now the power of abstract super classes is to use the Template Method design pattern. Instead of doing the heavy work in subclasses, you do the heavy work in the abstract class itself, you define some abstract methods that must be implemented by subclasses. These abstract methods are the specifics that differ for each subclass. This way, the direction is more or less down: from the abstract class you want to get something from a subclass that is specific for this subclass, even though the abstract doesn’t have to be aware of the exact subclass. The subclass just has to comply with the signature of the abstract methods defined in the abstract class.

When you just need some reusable functionality, I prefer using delegation instead of using an abstract class. This way the classes become more loosely-coupled, and the class with the delegated functionality can be used by more classes than just a subclass of some abstract class.

So, recap:

• let abstract classes do the heavy work, and have subclasses implement specifics that will be used by an abstract class
• delegate common behavior to a separate class if you just want to reuse some functionality

Tip: a ‘code-smell’ that may indicate that you’d better use delegation instead of an abstract class: if your abstract class does not contain any abstract methods, your abstract class is probably only there so that subclasses can reuse its functionality. In that case: use delegation!

One of the recurring patterns I find myself doing that successfully help in creating flexible component based application, is using technology as a driver for my component subdivisions. I usually try to create a domain model component that is completely technology agnostic. By technology agnostic I mean that my domain model should not know anything about HTTP, Web Services, LDAP, Databases, HTML, etc. It should be just an implementation of the business domain (only POJO’s or POCO’s). And usually I end up with a lot of interfaces and abstract classes in my domain model, so that components that are technology-aware are able to implement these interfaces. The important thing here is that the domain component is unaware of any technology or any of its implementors.

Then I have some technology aware components that implement the interfaces of my domain model. So for instance, I have an LDAP component that implements some interface of my domain model, using LDAP technology to fulfill it’s goals. This way the implementation of my domain becomes swappable, so that I am free to replace this component with another implementation, for instance with a SQL implementation. The power of this approach is that my domain model contains the business logic of my application, and requires some information or processing from it’s implementors, but it is unaware ‘how’ this is done. This will give you enormous freedom and flexibility.

Then usually I need final component that wires up everything together. This final component is aware of both the domain model and its implementors. This final component is responsible for wiring up the domain model with the correct implementations. A dependency injection framework could be used for this, or just hard-coding the wiring my also be an option (compile-time validation of your configuration may be a benefit here).

Another benefit of this approach is that you are creating a durable application. I have seen numerous situations where software applications are rebuilt from scratch because the legacy application is to complex to modify, extend or improve. So quite often the decision is made that it is cheaper to rebuild it than continuing working on the current application.

By creating the right dependencies, the right modules or components, you create durable software. Although the there will probably be changes in the business domain through time, in general you business domain can now outlive the implementations it uses. Upgrading to a new technology that is incompatible with the current technology you’re using for a certain implementation now only means creating a new implementation component, re-wiring the application with the new component, and everything keeps working. No need to trash the entire application.

The important principles to keep in mind:

• create technology agnostig domain model components
• create technology aware components that implement implement the domain
• use a single component that wires up your entire application
• think of composing your application from small components instead of one big moloch that does it all, so: composition

According to several EU directives, light bulbs, white good, cars and even homes must have energy labels. These labels indicate the energy efficiency in a range from A to G, where A is the most efficient and G is the least efficient. Yesterday I was discussing my current software project with a colleague, and we concluded that they way we’ve divided our project into components is a very flexible one. And because it is so flexible, our software has become very durable, because you can replace one component with a new one, without affecting all the others in any way, as long as the functionality of the software is not changed. And because durability is a hot environmental topic today, we thought that energy-label-like quality ratings for software would not be such a bad idea.

I believe that today there is no such thing as a standardized labeling system for software indicating the quality of the software.Wouldn’t it be a nice thing for customers to see that their software suppliers deliver A quality (durable) software instead of G quality (unmaintainable spaghetti code)? And I believe that one of the key factors in creating durable software is managing your dependencies well. As soon as one component becomes entangled with another one, you’ve already lost some of your flexibility.

I’ve read the book Agile Software Development, Principles, Patterns, and Practices by Robert C. Martin quite some time ago. He explains really well how to handle package or component dependencies, but it wasn’t until my current project that it really got through to me.

The following are my own set of guidelines for deciding how to separate my project into components. It works really well for me:

1. Create one or more domain components
These components should have NO dependencies on frameworks, implementations, XML, or what so ever. These components will most likely contain interfaces, and domain logic. These components should capture the (abstraction of) the functionality of your application. And most important: no dependencies to implementations or frameworks!!

2. Create well isolated implementation components
Implement your domain logic in separate components. And don’t put ALL of the implementation in 1 component. For example, if your application has a view and some sort of data storage that should be used by your domain logic, create a separate view implementation component and a separate data source component. If you would have put them in the same component, your view depends on the data source implementation which is not a good thing.

3. Don’t fall back on shortcuts, it will never help you in the end
Once you use shortcuts or the fastest solution instead of the best solution, software durability and flexibility drops, especially if you create dependencies between components. If you think “oh well, lets just implement the quickest way to pass this extra information” while this information is not related to this component, you mix up responsibilities, create unwanted dependencies, and eventually you’ll end up with a highly entangled product. So be aware of shortcuts!

4. Think: “can I replace this component without affecting something unrelated”
This is a thought I keep in my mind constantly. For our front-end application we used to have only 1 component. Now we’ve separated this application into components with their own responsibilities and their own dependencies. This increased the durability of out application immensely. As long as the functionality of our application does not change, one can decide to use a different front-end framework, while the rest of the components can remain.

5. Think composition
You’re almost always better of composing your application out of small components than creating one bulky component that does all. Being able to compose your application is the path to flexibility.

Creating a highly durable product is fun and very satisfying. Summarized I would recommend: no shortcuts, separate responsibilities, domain logic with no dependencies, application composition.

This is going to a somewhat technical post, be warned

Currently I am working on a project where I have to build a web application that allows users to search a large database with user data (an LDAP database). This web application is actually composed of 2 separate applications that communicate with each other using a Web Service. One of them you can call a front-end application, which contains the web pages, the front-end logic (like sorting results) and a client for the Web Service. And you can call the other one the back-end application, which hosts the web service, the domain logic and the logic for communicating with LDAP. In this post I want to focus on layering in the back-end application.

When I write software, there are always good software design principles in my head, like: keep methods short, make your code readable, seperation of concerns, minimize the side effects, etc. Now, in deciding how to package my back-end application into deployable components, I learned some great lessons from the design principles by Robert C. Martin (Uncle Bob). And especially the fact that different parts of the application may evolve at different speeds helped me to decide to define the components.

Now in this application I actually had 3 components:

• a Web Service component that exposes the domain logic to the outside world
• a domain component that contains all the domain logic of search for user data
• an LDAP component that implements the domain logic using LDAP specifics

And to have maximum flexibility, I wired the dependencies like this:

• the Web Service component only knows about the the domain component, since it’s only purpose is to expose domain logic to the outside world. The Web Service component doesn’t know anything about how the domain component does that, so it could be an LDAP database, a SQL database, text files, some over enthusiastic employee typing in responses in real-time, etc.
• the domain component only knows about itself. It even doesn’t know anything about external frameworks
• the LDAP component only know about the domain component, since the purpose of the LDAP component is to implement the domain logic using LDAP
• all these components are wired using Dependency Injection, which means that these components are connected to each other by means of an external configuration file

This setup makes the design extremely flexible, and these components now can evolve separately from each other without breaking (or just re-deploying) the other components. It also allows for swapping components, like using a REST Web Service instead of a SOAP Web Service to expose the domain logic to the outside world without touching the other ones. And using a SQL database instead of the current LDAP database without touching the other components.

Though I am quite content with this separation of concerns, this flexibility, and this independence, yesterday I ran into a point where I decided to create another component, namely, a specific LDAP component. The reason for this is the following: the current LDAP database we’re using contains inconsistent data. There are many parties responsible for delivering their own data to this LDAP database, without complying to certain standards (at least they appear not to be strict enough). The result is that the LDAP database is cluttered with attributes that are filled for one person, and left out for another person. In order to deal with these inconsistencies, I had to write some logic that deals with this.

But the point is this: this logic that deals with these specific inconsistencies, is not representative for all LDAP implementations. You can see this kind of logic as some sort of anti-corruption layer. The general purpose LDAP logic is now located in the other LDAP component, and specifics for dealing with inconsistent data is now located in this new LDAP component. With regards to flexibility this decision means that you can swap from one LDAP implementation to another without touching the original LDAP component. Or it could also mean that if the organization manages to fix the data so that it is consistent again, this new LDAP component is the only one that needs to be modified (or it may even become obsolete).

Even though I am quite satisfied with this decision, more components also means a need for more decisions: “where should I put this piece of logic, is it general purpose LDAP or is it specific LDAP?”. The direction of my thinking is to make the general purpose LDAP component implement the domain logic as much as possible, and then only put the exceptions in the specific LDAP component.

Anyway, it’s really nice to think about layering, responsibilities and dependencies, and trying to make it as flexible and independent as possible.

Currently I am creating an Excel sheet that I am using to monitor the progress of the project I am currently working. Working on this Excel sheet made me aware of some common mistakes people often make while making estimations. Here are 3 of the most common mistakes:

1. Forgetting that you will probably NOT be working full time on your project.
When you work 8 hours a day, it is often the case that you won’t be working 8 hours full time on your project. People disturb you, you have some meetings you need to attend, you spend some time answering email, you’ve got a brainstorm session, you need to go out to meet the customer, etc. Not working full time on your project means that your focus factor is not 100%. If you call everything else a ‘disturbance’ of effective work time, it might be interesting to figure out how much of your time is spent on disturbances and how much of your time is effectively spent on your project. The project I am currently working on will eventually turn out to have a focus factor of around 80%, which means that 20% of my time at work is spent at meetings, etc.

2. Forgetting the number of hours you have available
If you’re working in a team where some of your team members are part time employees, you will have less available hours to get the work done than a team consisting only of full time employees. Although this sounds logical, available hours is something that is quite often not taken into account.

3. Forgetting that productivity / speed will change over time and can be hard to estimate in advance.
If you’re starting a project with a new team, it will be difficult to estimate how fast your team will work. Once you have worked together for some time, you will (if you measure it) gain some knowledge of how much work your team can complete in a certain period of time. Without this experience as a team, you have to base your estimation on wild guesses and experience with other teams that may be different than this one. It is also very hard to estimate how long something will take when you’re going to do something that is entirely new for you: will it be easy? Will it take lots of study time to master it? Also, you will probably get better at the tasks you’re about to be doing. This means that an estimation of 4 days in the beginning of a project could be only 3 days once you’re familiar with the tasks at hand. Making relative estimations can be a solution to decouple time from size. If you estimate in absolute or fixed days, you are actually ignoring the fact that you’re team will get better and faster along the way.

Example:
Say, you estimate that some task takes 4 days. Now, if you have a focus factor of 80%, this means that the work will take 5 days to complete. Now, if this person only works 4 hours a week, this will mean that if he starts on Monday, has a focus factor of 80% and 1 part time day off, then the task won’t be finished until Monday next week in the end of the day.

So remember that you always take into account that you won’t be working full time on a project, and that you have a (limited) number of hours available for your project. Also take into account that estimations at the beginning of a project may be unrealistic further in time. Remember this when determining a deadline, or deciding which parts of the project will be skipped in order to make it to the deadline.

When you’re working on a project, one of the things you always must achieve is flexibility. Why? Because you never know what the future may bring you: functionality may be added, priorities may change, the deadline may be set earlier than originally was agreed upon, team members may get sick, etc. In other words, you must be able to cope with an uncertain future, while still being able to deliver a useful product or service. The way you break down you project into smaller parts highly influences the amount of flexibility you will have, so good project work-break-down is crucial!

In a current project I am working on I was given a predefined work-break-down of the project. The thing with this work-break-down that bothered me was that it was a break-down based on components, instead of User-Stories, behavior or functionality. The project was divided into components like a GUI component, a core back-end component, interface with other applications component, etc. What’s problematic here is that in order to have the smallest working application possible (the most critical parts), a little bit of all of the components was needed. So either I developed the most critical application while none of the defined components were finished, or I could develop all of the components but then I would be doing much and much more than just the most critical application. There are several problems with this approach:

• prioritizing is hard, because everything seems to have an equal priority since all components seem to be needed
• it is hard to change priorities because all components seem to be needed
• you are unable to drop one of the components because all components seem to be needed. After all, just having a GUI component without a core back-end component for example makes it quite a useless application
• it becomes difficult to finish a component since they contain both critical parts and less critical parts, so all components seem to be needed. (if the GUI component contains both critical parts as well as eye candy, then this way the eye candy must be completed as well in order to finish the GUI component).

Conclusion: all components seem to be needed! And that doesn’t really sound like flexibility!

So, what should you do in order to be more flexible, be able to prioritize more effectively? You should: break down your project based on User Stories, behavior or functionalities.

This allows you (or actually a product owner, end-users, clients or whoever is responsible for it) to prioritize behavior or functionalities of the application. The guiding principle behind this approach is: deliver the simplest, most critical working application as soon as possible, and then add features to it to make it better. This will probably mean that the part with the highest priority contains a little bit of GUI, a little bit of core back-end application, a little bit of interface with other systems, etc. This part should be finished first, and then other parts should be developed in order to make it better, more beautiful, more user-friendly, more whatever.

For example, if I were to build some sort of search application, I would build the most critical part of the search functionality first (highest priority). This would include some GUI with input fields, a software component that has the searching logic, a component that contains an implementation that connects to some data source, as well as a GUI part that displays the result. Now lower priority functionality would be pagination of search results (back-end logic as well as GUI), filtering search results (back-end logic as well as GUI), etc. In other words, once the basic search application is ready (tested, accepted, demoed or whatever), you start adding additional User-Stories, functionalities or behavior.

The great advantage of this approach is that you can easily change priorities (”sorting is more important than filtering, so do that first”), you can drop functionalities while still have a working, useful application (”we’re not going to make it to the deadline, so let’s drop the sorting functionality for now, we’ll release that after the deadline”), and you can truly finish individual parts of the project (”sorting is now finished and accepted, so we remove this from our todo list”).

If you were given 10 tasks and asked how long it would take you to complete these tasks, the most common thing to do is to estimate the duration of each individual task. You may divide the task into activities and estimate those, you may discover some dependencies and once everything looks great you commit to your estimation. Now, as you start working on the tasks, your project eventually fails: you run out of time, the deadline has to be moved further into the future or you want more people on your team in order to deliver on time.

Sounds familiar?

Now most of the time when people make estimations, they usually estimate in absolute days or hours. But there are several danger with traditional estimations:

• if your planning is based on activities rather than features, you’re risking the fact that activities don’t finish, at least they don’t usually finish early.
• if there are many dependencies between activities, lateness is passed down the schedule.
• if your tasks are not prioritized by their value to the users and customers, but by the convenience of the development team, you have no option to allow users and customers determine the order and sequence of the work to be done.
• if you ignore uncertainty and assume that initial requirement analysis led to perfect specifications, you will be surprised by reality.
• if your estimates become commitments, which is often the case, you are mistaking the fact that an estimation is a probability, and you can not commit to a probability.

A good planning process and better estimations allows: reducing risk, reducing uncertainty, supporting better decision making, establishing trust and conveying information.

One way of making better estimations is to make relative estimations. With relative estimations you estimate size and you derive duration, instead of estimating duration. So for example, a user-story of 10 points is twice as big, complex or risky as a user-story estimated 5 points. There is no unit for relative estimates, no days, no hours, etc.

An iterative approach is probably the best way to deal with uncertainty. During each iteration you can determine how many story points you’ve been able to complete, this is the velocity, the derived duration. For example: 3 stories of 5 points in 1 iteration means a velocity of 3 * 5 = 15. The velocity corrects estimation errors, and is the way to determine the speed of your team and to make predictions about the future.

The benefit of relative estimations is that you separate duration from size. This allows you and your team to deal with the fact that during the project you are learning and your understanding of the problem domain increases (in other words: you get more work done later in the project than early in the beginning = increase in velocity). Also unforseen things that may happen, will be a change in velocity while the relative size of the user-stories are not changed.