Tuesday 10 June 2008

To Be(*) Or Not To Be(*) That Is The Question

(*) Included in an API bundle.

There's been a lively debate on the OSGi mailing list over the past couple of weeks surrounding the issue of whether an API should be packaged with it's implementation in one bundle or whether it should be split up into two bundles (one for API and one for implementation).

I think it's fair to say there are a range of opinions on the subject. What is clear however is that there is no one size fits all answer to this question. As a developer or architect you need to consider the use case your API and implementation are going to be put to in order to make the "correct" decision.

The fundamental issue that everyone involved in the discussion agreed on is that using a bundle should be as simple as possible. The difference of opinions comes in depending on what measure of simplicity you take to be most important. I think there are three main themes that have so far been discussed:
  • installation simplicity - minimizing the number of bundles needed to run an OSGi application
  • runtime simplicity - minimizing the connectivity between bundles in a running OSGi application.
  • architectural simplicity - minimizing duplicated packages between bundles used to build an OSGi application
These three goals are often (but not always) in conflict and it depends on your use case which is the most important for you. In order understand where the conflict arises it is important to understand how OSGi classloading works especially with regard to installation and resolution of bundles.

I'll try to give a quick overview.

A bundle can contain classes (like any standard Java jar). However it can also import classes from other bundles via a number of different import mechanisms. Importing classes allows bundles to be modularized such that different bundles provided different sets of functionality but still declare their dependencies in order to run. In order to use classes from a bundle all import requirements must be satisfied. In order to satisfy an import requirement a bundle exporting that requirement must be installed in the OSGi runtime.

So if BundleA depends on classes exported from BundleB you have to install both BundleA and BundleB in order to use BundleA. If BundleB depends on classes from another bundle you need to install those bundles too. You can probably see that this problem can rapidly diverge from being a single bundle install to being a large complicated set of bundle installs.

Another important detail to be considered is that in OSGi it is possible for many bundles to export the same API package and only one will be picked by the runtime to actually provide the classes.

When we discuss splitting API and implementation we are saying that one bundle will export the API classes and another bundle will import that API and provide an implementation for it. When we talk about grouping API and implementation we mean that the API will be exported from the same bundle that provides the implementation. Many implementations may export the same API but this is OK in an OSGi framework.

My own advice would be to start by assuming that API and implementation are packaged in separate bundles. The reasoning behind this is based on the following criteria:
  • In general an implementation is likely to depend on more packages than it's API
  • You can always collapse back to one bundle later if you use import-package vs require-bundle
  • If you use a provisioning mechanism such as Newton or P2 (when it's ready) downloading two bundles vs one is handled automagically
The benefits of splitting API and implementation are the following:
  • If you are considering making your application distributed or want it to run in a constrained environment you can install the API without having to resolve the implementation dependencies (possibly a big deal in a client/server architecture)
  • If you want to restart or reinstall the implementation bundle then this doesn't automatically mean the restart of all client bundles that are using the API from that bundle
  • OSGi export-package has a uses attribute - to specify classes that the exported API has gained from imports. It is possible for combinations of exports and imports to cause bundles to be mutually exclusive - such that it is impossible to close the graph and install all bundles in the same OSGi runtime at the same time. Limiting the connectivity in the graph via separating API from implementation reduces the risk of running into this problem.
If you start by assuming the API and implementation are separate then you can use the following logic to assess whether you can condense them back to one bundle for the purposes of your architecture:
  1. Start by designing your API to depend on as little as possible.
  2. Make your implementation depend on API packages and any other dependencies it needs to function.
  3. If after doing this the implementation depends only on API consider whether the implementation is ever likely to get more complicated.
  4. If it isn't then you can probably collapse back to one bundle.
Of course this can always be done prior to building any bundles if you are good at modelling in you head or on paper etc.

Hopefully that's helped some people understand the issues.

Laters

Sunday 8 June 2008

Software Versioning Is Complex Or Am I Imagining It?

So it seems everyone and his dog are talking about versioning at the moment. Specifically the proposed version numbering systems used in OSGi and JSR 277 and their ability to coexist (or not).

For my own part I personally favor the OSGi scheme because to my mind it is simpler and better defined.

The OSGi scheme says that there are three numbers used in a version, major.minor.micro -> 1.2.1. The micro version is incremented when ever there are bug fixes. The minor version is incremented when there are backwards compatible additions and the major version is incremented when there are any non backwards compatible changes. It is also possible to add postfix elements to an OSGi version i.e. 1.2.1_20080608 (todays date appended). These postfix elements are considered lexicographically to mean increments/improvements which do not warrant a version update - i.e. nightly builds etc.

When a module consumer wishes to specify a version they wish to use they specify either a single version number.

1.0.0

or a version range

[1.0.0,2.0.0)

The first version means any version greater or equal to 1.0.0. The second means any version between 1.0.0 and any version prior to 2.0.0.

However anyone who has worked with software development for any non trivial amount of time will know there are still inherent problems in versioning schemes in that they require developers to correctly markup when changes to their code effect the compatibility of that code.

This is a big deal as depending on the consumer of the code even changes as innocuous as a java doc comment update can effect the version number. There's a really good discussion of the issues surrounding this here.

As we all know developers are mostly human and so; make mistakes, are lazy or are just plain forgetful so it is inevitable that version numbers will sometimes not be updated when they should have been.

After release these badly versioned modules (poison pills) will sit around in software repositories for potentially a very long time causing instabilities in the systems that use them.

Stanly Ho's suggestion for combating this issue is to allow consumers of modules to specify complex lists which select certain version ranges but exclude certain known bad versions.

Like many other aspects of his proposal I think this is going to be overly complex and unmanageable in any practical environment. It effectively means build managers have to rebuild their module every time they find a bad module which is a lot of extra work, let alone how the build manager then propagates his/her updated module to all clients.

OK so what's the point I'm getting to? Well during discussions with colleagues it occurred to me there is a relatively simple extension we could make to version numbering schemes that nips off a lot of the problems at the start of the development life cycle vs at the end.

This is a good thing as if it is easier for developers to handle version numbering there should be fewer bad versions.

I'm not saying there will be no bad versions as there are always the unintended API changes which slip past quality control etc but I would argue these are the exception vs the rule in a well managed project. Any bad versions that do slip out could be highlighted and deleted from the collective repositories by administrators. But given a smaller number of mistakes this problem should be tractable.

I'll prefix the rest of this blog entry by saying my idea is a little crazy so please bare with me, pat me on the head and say there, there back to the grind stone you'll get over it :)

So enough pontificating, the idea:

It occurred to me that software versions should really carry an imaginary coefficient (i.e. square root of minus one).

[Sound of various blog readers falling off their seats]

...

What the hell does that mean I here you ask.

I said it was crazy and I'm only half proposing it as a real (chuckle) solution. However to my mind it seems more natural to label versions intended as release candidates or as development code as having an imaginary coefficient.

The reasoning behind this is that whilst development code is tending towards a real API or implementation there are always going to be aspects to it that is not quite finished and so can be considered imaginary. I think a useful way of visualising this is that as software approaches release the imaginary coefficient gets smaller, i.e. there is a rotation in the software from the imaginary plane to the real.

Ok so just because it has a nice analogy doesn't make it valid, how does this help in real world software development?

Firstly this scheme does not effect existing versioning semantics, if you want to work purely in the real plane then that is perfectly acceptable. So it doesn't break existing OSGi bundle versions in the wild today. However, it does give developers a way to markup whether they want their code to interact with code that is still in development.

Imagine a case where software producer A (providing module A) is building release candidates and software consumer B is building module B that depends on module A:

ModuleA:1.0.0_RC1
ModuleA:1.0.0_RC2
ModuleA:1.0.0_RC3
etc.

ModuleB:import ModuleA:1.0.0

In the current scheme there is no way to distinguish release candidates from final code and incremental patches so when producer A builds his release candidate consumer B sees each release candidate as actually being more advanced than the final 1.0.0 release.

This is clearly wrong.

As software engineers we tend to manage this internally by having a succession of 0.9, 0.9.1, 0.9.2, etc releases prior to the 1.0.0 RC releases so minimizing the number of RC candidates that ever get released and making it relatively simple to tidy up any that do accidentally make it into live environments.

However this breaks the simple rules set up in the OSGi scheme in that there is a discontinuity between 0.8 and 0.9 that may be non backwards compatible. We then have to employ all sorts of artificial rules during development to say import [0.0.0,0.9.0) but this needs to be applied to all modules after we have decided to make 0.9 the non backwards compatible internal release.

Instead I propose that when producer A starts work on a set of changes to module A he increments the appropriate version element (depending on the scale of the changes he/she is making) early and adds an imaginary coefficient to mark it in development.

Therefore from the previous example we have

ModuleA:1.0.0_3i (RC1)
ModuleA:1.0.0_2i (RC2)
ModuleA:1.0.0_1i (RC3)

As an external consumer of module A we are then able to use [1.0.0,2.0.0) to mean all compatible increments. An internal consumer prior to release can say [1.0.0_2i,2.0.0)
to mean all releases of module A after RC2. Importantly this will continue to work after release with no need to update existing imports.

But (I here you say) how do we know how far away from zero to start when starting the imaginary coefficient. I would argue that this can be handled in the same way as real versions i.e. 1.0.0_0.1i is closer to release than 1.0.0_1i and 1.0.0_0.0.1i is closer still. So this issue is likely project or problem specific - start where ever you like.

We could come up with a scheme where by the major.minor.micro elements of the imaginary coefficient denote degrees of confidence as to how closely the code matches the proposed design - i.e. major = zero -> developer tested, minor = zero -> system tested, micro = zero -> beta tested etc.

The notion of complex version numbers applies equally to the JSR 277 four number version scheme which to my thinking is a completely pointless addition to a spec which does nothing to address the actual problem and breaks lots of existing code that was previously working fine.

I'd be very happy for someone to come along and state why imaginary version numbers are not needed as in general I prefer to reuse existing tools where possible and ideally the simplest tool that does the job.

So if nothing else this is a vote for OSGi versioning but with a couple of notes on how we may be able to improve it.

Laters

Update: 09/06/2008
Apologies I linked to the second part of the eclipse article on version updates by mistake the correct link is here.

Saturday 7 June 2008

A JSR To Far?

So there's a lot of conversation going on around the both the politics and the technological issues surrounding JSR 277. Personally I think the spec is doomed if it doesn't start working much more closely with the OSGi community - and here's why.

The OSGi community now comprises almost all the major software vendors who have been expending significant effort working on the OSGi specifications and OSGi enabling their applications over a number of years. The JSR is currently stalled and taking advice predominantly from Sun internal researchers.

If JSR 277 makes it into the Java 7 release then it seems entirely plausible that the major vendors could choose not to certify their products on Java 7 (especially if the JSR gets in the way of their existing investment in OSGi).

Where would this leave Sun if the likes of Websphere, Weblogic, Spring, etc, etc are not certified to run in an enterprise environment? Also where does it leave Java?

Disclaimer, these are my own interpretations of the situation and I have had no conversations with any of the protagonists leading me to this assessment. I'm just concerned that politics and egos seem to be getting in the way of one of the most important evolutions of the Java eco system which could be a disaster for all involved in Java.

Please guys, set egos aside and come to a sensible decision that allows us all to get on with our day jobs.