OSGi Dev Con 2009 & OSGi Tooling Summit Roundup

So I'm currently sitting in my hotel lobby waiting till we jump in the hire car off to the airport for our flight home. It's been a long week - the time zone difference between San Francisco and London is always exhausting and the mental effort of networking, presenting, and talking to customers is always pretty intensive too.

I think the conference was definitely quieter this year - you couldn't help but feel the shadow of the economic climate hanging over us all - however though volume was down compared to last year, quality seemed to be much higher. My rough market assessment based on talking to people on our exhibition stand was that awareness of OSGi is much improved compared to last year.

Even better for me people really seemed to get what we (Paremus) are about. Last year we were the "RMI guys". This year people we talked to seemed to get genuinely excited about what our product is a really about: a flexible, scalable solution to provisioning and managing dynamic distributed OSGi based applications in enterprise environments.

Pretty much everyone we talked to was very positive on OSGi as a runtime solution to Java modularisation. However, the problem for its adoption in the enterprise is the backwards compatability issues in moving to an OSGi environment (things like static references and Class.forName are a really bad idea in OSGi) and the fact that OSGi development relies on a reasonable amount of domain specific knowledge.

I think good tooling solutions that work right the way though the stack are crucial to help new developers though the pitfalls of the new classloader space. Unfortunately, tooling support for new developers is pretty disjointed.

Hence, the next part of my post...

We (Paremus) recognised that it does us no good if customers are having difficulty building OSGi based solutions. Therefore, my team mate Derek Baum and I have been working on a solution to developing OSGi bundles which works in both an IDE and headless build environment called Sigil. Prior to OSGi Dev Con Paremus chose to licence Sigil under the Apache licence, as we recognise that tooling is an area where we need support from the community in order to help the community as a whole.

On the Friday, after the end of the conference, I and a number of other representatives with interests in the area of development tooling met at an OSGi Tooling Summit hosted by Yan Pujante at LinkedIn's Mountain View offices. The group was pretty large and diverse (as you can see here). The OSGi Alliance's typical role in this respect is to foster communication (via summits, conference calls etc), then generate one or more RFPs which list a set of requirements for the problem domain, and finally put together a set of RFCs which provide a spec to allow different vendors to work together.

The format of the meeting was an initial intro and positioning statements (i.e. what we were there to achieve). We then moved into a blue sky discussion where we tried to think of a perfect world solution to the OSGi development lifecycle - this stimulated some really interesting conversation which I'll doubtless follow up on in the next couple of weeks in future blog posts. Finally, we put together a series of use cases headers that captured the major tasks undertaken by developers when building, maintaining and releasing OSGi applications into the wild.

I had a number of really encouraging conversations with Chris Aniszczyk and Peter Kriens who work on PDE and BND respectively, both of which have a lot of cross over with the work I've been doing on Sigil.

Chris has just twittered that he's thinking of changing the name of PDE to BDE which I think is a great idea - perhaps some merging of BDE, Sigil and BND? I think there is likely some common core classes that are useful to all OSGi developers. We're obviously going to need some domain specific extensions for plugin development, newton development, spring development, etc. but I see these as addons on top of a common OSGi layer.

I guess in a perfect world I'd like to be able to support Maven, Netbeans and IntelliJ users as well. Hopefully I'll be able to update you in the next couple of months on progress in this area.

Laters,

Flocking Behaviours

Just a quick note to say I'll be running a BOF session with Peter Kriens and Chris Aniszczyk on OSGi development tooling at EclipseCon this year as part of the OSGi track. You can find details here.

I guess my main focus for calling the BOF is that I'm very interested in talking to other OSGi developers to see what it is we on the tooling side can do to make our collective jobs easier.

What is it about OSGi development that really frustrates you - and what can tools do to make it easier?

It'll also be really interesting to meet up with other tools developers to see if we can get a bit of reuse going on - likely we won't solve that in one BOF but if we can just put names to faces and get a bit of shared understanding going on we're likely to be more productive in the long term.

Hope to see you there.

Laters,

OSGi tooling

There's a really interesting conversation going on at TSS about OSGi and future directions for Enterprise Java.

I've posted a reply which I thought it was worth reposting here:

I think there are two issues with [the approach of repackaging existing modules as OSGi bundles and simply importing/exporting all packages] which really cause headaches going forward; Module vs API dependencies and complex "Uses" graphs.

Firstly module vs api; in most dependency tools such as Maven and Ivy the developer specifies dependencies at the module layer - i.e.

<dependency org="org.apache.log4j" name="org.apache.log4j" rev="1.2.15" />

But then spring have added an OSGi version of the module which has a different module id.

<dependency org="org.apache.log4j" name="com.springsource.org.apache.log4j" rev="1.2.15" />

How does the tooling know that these two modules are actually compatible? It can't and this tends to lead to false dependency failures. This is the same problem as using module dependencies in OSGi at runtime. Actually the code is not dependent on the module but on the API that the module supplies.

Having been working with OSGi as part of our [1] distributed OSGi platform for almost 4 years now, I came to the conclusion that the tooling should use the notion of package import at build time - just as OSGi does at runtime. This is the approach I've taken in Sigil [2] which we're now using to build our next product release.

Having taken the jump to use API dependencies our build system is now much simpler to manage. Instead of having to manually lookup which module supplies a package, I just type the package into the IDE and our tooling goes off and trawls the OSGi manifest information in various repositories to find a module that supports it (seemless). We've also used this on a number of third party applications (the record being a project with over 600 modules - which we managed to convert to use sigil in less than a couple of hours based on pretty much a simple grep of the java code for java import statements).

Secondly the major thorn in the approach of naively exporting all packages in a module is the complex "uses" information it generates. "Uses" is a flag provided by OSGi on exports to assert that the class space is coherent across multiple bundle import/export hops.

It is currently believed that the "uses" information makes the problem of resolving bundle dependencies NP-complete. As the number of bundles and import/exports increases the number of calculations the OSGi runtime has to do to check that all bundles can sit together goes up at a terrifying rate.

I've referred to this as placing barbed wire around sand castles (in most cases). If the modules were more sensibly designed i.e. only exporting the "really" public code then this problem is much reduced.

I'm also attending the summit Jason refers to and I've called a BOF "OSGi Development Tooling" with Peter Kriens and Chris Aniszczyk at the OSGiDevCon conference just prior to the summit to discuss tooling efforts w.r.t. OSGi. The BOF is open to the general public and it would be great to hear opinions before we sit down at the summit to figure out a path forwards.

[1] http://www.paremus.com
[2] http://sigil.codecauldron.org

Coming up for air

Ok so it's been a while since I last post anything here, mainly because I've been working on a number of different big projects for the last couple of months and this left no time for blog posting.

So having come up for air for a few hours at least I thought it'd be a good idea to blog about them see if I can drum up some interest ;)

The major addition to my task list over the past few months has been the Sigil project. This is a set of tools to help developers build OSGi bundles. It started off as an eclipse plugin but about two or three months ago my team mate came along with some really great code to integrate it with Ivy.

I believe Sigil is the first tool out there to unify OSGi development in the IDE and server side build in a common configuration file (this being a good thing as it saves the messy job of keeping other systems in sync).

The IDE supports tools such as code complete, quick fixes and tools to analyse OSGi dependencies. I've also built in the idea of repositories (currently either file system or OBR but extensible via plugins) which allow the developer to download dependencies on the fly whilst developing by simply adding an import-package or require bundle statement to their code. Oh and the same repositories can be used in eclipse and ivy :)

The other big piece of code I've been working on is of course Newton. There are no big feature announcements for this release as we've been focussing on making the current platform more and more robust. But we've just made available the 1.3.1 release which has a number of internal improvements to reliability and scalability - I'd encourage you to take a look - one game we end up playing in the office is "Wak - A - Mole" with the containers when the fractal demo is deployed. You can take out a container (ctrl-c, ctrl-z, power cycle, cable unplug, etc) and the infrastructure will always recover :)

Anyway that seems like a good amount of detail for the time being. I'll try to blog some more about some of this stuff soon...

Laters,

Is this an application which I see before me?

This post has been triggered by two interesting posts on the topic of what it is to be an application in an OSGi environment. This is something I felt I just had to get stuck in with as its exactly what we've been working on in the Newton project.

The way we've tackled this is very similar to the approach suggested by Mirko, except instead of using Spring-DM as the top level element in the model we've picked SCA.

SCA is a relatively new specification but it gives a vendor neutral way of describing the service dependencies and architecture of an application running in an enterprise environment.

The basic pattern is that a component which has a certain implementation (java, spring, bpel, c, etc) is able to publish a number of services (over a range of protocols - in-memory, RMI, WS, JMS, etc) and consume services from other components via references. The combined unit which is the component, it's services and it's references are then packaged in an entity called a composite.

In Newton we associate a composite with a top level (or root) bundle. This bundle then provides the class space for the composite to instantiate it's implementation, services and references. Importantly the bundle does not have to contain all of the classes that it needs to function but can use OSGi tools such as Import-Package to achieve modularity at the deployment level.

When an SCA composite is installed in a Newton runtime we go through a series of steps:

1. Resolve the root bundle that supplies the class space for the composite. If this is not the first time the root bundle has been resolved we increment an internal counter
   
2. Resolve and optionally download the bundle dependencies required to satisfy the root bundle against a runtime repository (this includes ensuring that we reuse existing bundles within the runtime - if they were installed for other composites)
3. Build a runtime model around the SCA document that controls the lifecycle of the component as references come and go
4. Finally when all required references are satisfied (a dynamic process) we publish the services to be consumed by other components in the enterprise.

When an SCA composite is uninstalled we go through the reverse process:

1. Unpublish the services and release any references.
2. Shut down the implementation and discard our runtime model.
   
3. The bundle root counter is decremented, if the bundle root counter reaches zero then it is no longer required in the runtime and is marked as garbage.
   
4. Finally garbage collect all bundles that are no longer in use, so clearing down the environment.

This pattern then forms the building blocks of our distributed provisioning framework that is able to deploy instances of SCA composites across what we term a "fabric" of newton runtime instances.

To achieve this we group composites together into an entity known as a System. A system is a group of composites which each specify a particular replication strategy and a set of contract requirements. A replication strategy is a dynamic pattern used by Newton to decide how many instances of a Composite should be installed at any given instant. A contract is a pattern used to restrict where a composite should be installed, i.e. a contract may specify a filter such as (&(machine.cpu.count>1)(machine.memory>1G)) i.e. only install on dual cpu or greater machines with more than 1GB of memory available.

So in the Newton world an application is actually a very dynamic entity. It is built from OSGi bundles and wired together on the fly. But it never the less has rules which are defined via a model (expressed in SCA plus our system extensions) which specify:

• how implementations and interfaces are connected
  
• how remote services should interact via bindings
• how they should scale
• where they should install

Hope that's been of interest,

Laters,

A New Hope

A long time ago in a galaxy far far away...

A group of rebels formed the OSGi Alliance, an alliance against the empire (or at least monolithic code empires). The alliance was created to champion an ideal (in CS terms a specification) which described a standardised way for modules of Java code in a single virtual machine to interoperate without requiring the JVM to be restarted.

The specification was simple to understand and many were drawn to it due to the elegance of its implementation. In time many who chose this new life found that despite the simplicity of the specification championed by the JEDi alliance they could nevertheless wield wondrous powers not seen before in the Java world.

Inevitably these powers drew the attention of those still living within the empire. Initially some sought to discredit the Jedi powers, either through misunderstanding or fear. But soon others became envious and wanted these powers for themselves.

Some within the empire chose to take the difficult road and started to convert their legacy methodologies to meet the requirements of the Jedi ideal. But others saw a different path, instead of converting their methodologies to conform to the Jedi ideals they decided to convert the Jedi ideals to fit their own methodologies.

Those in the Jedi council found themselves torn between the short term promises of wealth offered by the empire or sticking to their ideals and holding out for the long term riches of a truely flexible Java virtual machine architecture.

It is at this point in the story that we find our selves. Peter Kriens (Obi Wan?) has recently blogged about the choices facing the Jedi alliance and argues for the purist ideals to be upheld.

Myself I find myself acting as a trader (or possibly a smuggler - gonna argue for Han but you make your own judgements...) between these two worlds.

As a developer and architect of Infiniflow I have been directly involved with building a distributed computing infrastructure that seeks to embrace ways of the Force as championed by the Jedi alliance for a single JVM but across the enterprise.

Our driving goal that led us as a company to the Jedi way of life was acceptance within our architecture that things change, or to put it another way: there is no one rigid model of the enterprise but rather a series of iterations which encapsulate the functions required to service the current business requirements.

Whether you believe a word of this, having walked the boundary between the Jedi world and that of the Empire I am acutely aware of the problems associated with integrating tools built by these two communities.

My own gut feeling is that the Jedi should stick to their ideals and treat integration with the Empire as an edge case - i.e. certainly build up specifications but treat them as boundary conditions vs compromising the core ideals. In this way legacy methodologies can be incorporated but in such a way as to highlight the fact that it is just a stepping stone towards a new ideal.

When it is impossible to integrate a legacy pattern I'd argue that this is a point when we have to admit that Gödel was right - it is not always possible to please all of the people all of the time (I paraphrase but you get the point). You can always delegate legacy cases to a separate jvm and communicate remotely to the old world.

If the Jedi council compromise their core ideals for ill conceived or temporary solutions they risk sending out a mixed and confusing message to those who are new to this technology.

Padawan learning the ways of the Force for the first time are unlikely to appreciate the subtle differences between such abstract concepts as import-package and require-bundle. They will of course go for which ever is the simplest pattern to achieve their short term goals but this could well be at the expense of logical coherency within the OSGi specification.

Once stepping onto the path to the dark side it is very difficult to turn back and ultimately leads to ruin. (Or cool lightning powers - you decide)

I have to give credit to the fantastic blog posts of Hal Hildebrand for the Star Wars theme to this blog entry, whether this will become a common theme for my own posts I'm unsure but it was certainly fun to write.

Laters,

To Be(*) Or Not To Be(*) That Is The Question

(*) Included in an API bundle.

There's been a lively debate on the OSGi mailing list over the past couple of weeks surrounding the issue of whether an API should be packaged with it's implementation in one bundle or whether it should be split up into two bundles (one for API and one for implementation).

I think it's fair to say there are a range of opinions on the subject. What is clear however is that there is no one size fits all answer to this question. As a developer or architect you need to consider the use case your API and implementation are going to be put to in order to make the "correct" decision.

The fundamental issue that everyone involved in the discussion agreed on is that using a bundle should be as simple as possible. The difference of opinions comes in depending on what measure of simplicity you take to be most important. I think there are three main themes that have so far been discussed:

• installation simplicity - minimizing the number of bundles needed to run an OSGi application
• runtime simplicity - minimizing the connectivity between bundles in a running OSGi application.
• architectural simplicity - minimizing duplicated packages between bundles used to build an OSGi application

These three goals are often (but not always) in conflict and it depends on your use case which is the most important for you. In order understand where the conflict arises it is important to understand how OSGi classloading works especially with regard to installation and resolution of bundles.

I'll try to give a quick overview.

A bundle can contain classes (like any standard Java jar). However it can also import classes from other bundles via a number of different import mechanisms. Importing classes allows bundles to be modularized such that different bundles provided different sets of functionality but still declare their dependencies in order to run. In order to use classes from a bundle all import requirements must be satisfied. In order to satisfy an import requirement a bundle exporting that requirement must be installed in the OSGi runtime.

So if BundleA depends on classes exported from BundleB you have to install both BundleA and BundleB in order to use BundleA. If BundleB depends on classes from another bundle you need to install those bundles too. You can probably see that this problem can rapidly diverge from being a single bundle install to being a large complicated set of bundle installs.

Another important detail to be considered is that in OSGi it is possible for many bundles to export the same API package and only one will be picked by the runtime to actually provide the classes.

When we discuss splitting API and implementation we are saying that one bundle will export the API classes and another bundle will import that API and provide an implementation for it. When we talk about grouping API and implementation we mean that the API will be exported from the same bundle that provides the implementation. Many implementations may export the same API but this is OK in an OSGi framework.

My own advice would be to start by assuming that API and implementation are packaged in separate bundles. The reasoning behind this is based on the following criteria:

• In general an implementation is likely to depend on more packages than it's API
  
• You can always collapse back to one bundle later if you use import-package vs require-bundle
• If you use a provisioning mechanism such as Newton or P2 (when it's ready) downloading two bundles vs one is handled automagically

The benefits of splitting API and implementation are the following:

• If you are considering making your application distributed or want it to run in a constrained environment you can install the API without having to resolve the implementation dependencies (possibly a big deal in a client/server architecture)
  
• If you want to restart or reinstall the implementation bundle then this doesn't automatically mean the restart of all client bundles that are using the API from that bundle
  
• OSGi export-package has a uses attribute - to specify classes that the exported API has gained from imports. It is possible for combinations of exports and imports to cause bundles to be mutually exclusive - such that it is impossible to close the graph and install all bundles in the same OSGi runtime at the same time. Limiting the connectivity in the graph via separating API from implementation reduces the risk of running into this problem.

If you start by assuming the API and implementation are separate then you can use the following logic to assess whether you can condense them back to one bundle for the purposes of your architecture:

1. Start by designing your API to depend on as little as possible.
2. Make your implementation depend on API packages and any other dependencies it needs to function.
   
3. If after doing this the implementation depends only on API consider whether the implementation is ever likely to get more complicated.
4. If it isn't then you can probably collapse back to one bundle.

Of course this can always be done prior to building any bundles if you are good at modelling in you head or on paper etc.

Hopefully that's helped some people understand the issues.

Laters