Wednesday 2 December 2009

Hello Tatooine

So in my last couple of posts I've been showing the power of Nimble. You will have noticed that it is primarily a console environment. As such you may be wondering how you can provide your own commands to execute in the Nimble shell - Posh (Paremus OSGi Shell).

Posh is an implementation of the command line interface specified in RFC-147 from the OSGi alliance. If you are familiar with OSGi development you will know that to date every framework implementation has defined it's own particular API for providing command line utilities within an OSGi runtime. This has meant that there is significant duplication of effort when writing commands to work in the various environments.

To this end the alliance proposed RFC 147 in order to provide a common standard that different frameworks could implement such that a command that worked in one framework could work unchanged in another. The initial implementation of RFC 147 was developed primarily by Peter Kriens and donated to the Felix project earlier this year. Since then there have been a number of maintenance releases and it has been included as a component of the Felix Karaf container and the Nimble container from Paremus.

This gives you some background, so now the standard thing for me to do would be to write a trivial hello world application. But that's no fun, so instead of conforming to the norm I thought it would be more interesting to port the Starwars Asciimation work to run in OSGi as an RFC 147 command line interface.




Yep this is very probably the geekiest post you will ever see... :)

I hasten to add that I did not undertake the core task of this myself, instead I took the liberty of contacting Simon Jansen (the author of Asciimation) to ask his permission to "borrow" the ascii text from his player. This he was very kind to do.

The first thing we need to do to define our cli is define a class that implements the core functionality as shown below:

package org.chronologicalthought;

import java.io.BufferedInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.PrintStream;
import java.io.Reader;
import java.net.URL;

public class Starwars {
public void starwars() throws IOException, InterruptedException {
play(67);
}

public void starwars(int frameLength) throws IOException, InterruptedException {
URL res = Starwars.class.getResource("/starwars.txt");
if (res == null)
throw new IllegalStateException("Missing resource");
InputStream in = res.openStream();
try {
InputStreamReader reader = new InputStreamReader(new BufferedInputStream(in));
render(reader, System.out, frameLength);
} finally {
in.close();
}
}

private void render(Reader reader, PrintStream out, int frameLength) {
// ...
}
}

Here the command provides two methods, play and play(int) and prints the individual frames from the "starwars.txt" file embedded in our bundle to System.out.

Wait a minute you might be thinking. Where's the API to the CLI? Well this is one of the neat things about RFC 147 you don't need to write your code to any API. The specification provides a clever utility in the form of a ThreadIO service that multiplexes the references to System.in, System.out, and System.err so the command can interact with the user. It also calls methods on the class reflectively so there is no need to implement a defined interface. Simply declare a method and Posh will attempt to convert arguements supplied from the command line to match the method signature.

The next step is to define an activator that publishes our cli class to the OSGi bundle context.

package org.chronologicalthought;

import java.util.Hashtable;
import org.osgi.service.command.CommandProcessor;

import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;

public class Activator implements BundleActivator {

public void start(BundleContext ctx) throws Exception {
Hashtable props = new Hashtable();
props.put(CommandProcessor.COMMAND_SCOPE, "ct");
props.put(CommandProcessor.COMMAND_FUNCTION, new String[] { "starwars" });
ctx.registerService(Starwars.class.getName(), new Starwars(), props);
}

public void stop(BundleContext ctx) throws Exception {
}
}

This activator publishes the Starwars class with two attributes:

  • CommandProcessor.COMMAND_SCOPE - a unique namespace for our command
  • CommandProcessor.COMMAND_FUNCTION - the names of the methods to expose as commands in the cli

The code is available from here for those who want to take a look around:

Now the final stage is to load our command within nimble. Since my cli has such a trivial set of dependencies I will defer the usage of the full nimble resolution to another blog post. So instead I will use a set of trivial commands which I've defined here. So finally let the show commence:

$ svn co http://chronological-thought.googlecode.com/svn/trunk/starwars
$ cd starwars
$ ant
$ posh
Paremus Nimble 30-day license, expires Wed Dec 30 23:59:59 GMT 2009.
________________________________________
Welcome to Paremus Nimble!
Type 'help' for help.
[feynman.local.0]% source http://chronological-thought.googlecode.com/svn/trunk/nimble-examples/basic-commands.osh
[feynman.local.0]% installAndStart file:build/lib/org.chronologicalthought.starwars.jar
[feynman.local.0]% starwars





WWW.ASCIIMATION.CO.NZ


presents




The final piece of Nimble functionality that it would be fun to demonstrate is stopping the movie. Simply hit Ctrl-C. The nimble shell then sends a Thread.interrupt to the currently running command. For those who want to see the movie to the end and don't want to wait try running:

% starwars 20

To set the frame length as 20 milliseconds.

Enjoy the show.

Laters.

Tuesday 1 December 2009

And for my next trick

Just for fun and to demonstrate the power of the Posh (sh)ell environment I decided to knock together the following trivial script to do a "traditional" OSGi bundle file install from a directory:

// create a temporary array for storing ids
array = new java.util.ArrayList;

// iterate over the files passed
// in as arguement 1 to this script
each (glob $1/*) {

// use the BundleContext.installBundle
// method to install each bundle
id=osgi:installBundle $it;

// store the bundle id for start later
$array add $id;
};

// iterate over our installed bundles
each ($array) {
// use the BundleContext.start method
//to start it
osgi:start $it;
};

To try this out for yourself or to find out more about Nimble you look here once installed you can run the above script using the following command:

posh -k http://chronological-thought.googlecode.com/svn/trunk/nimble-examples/file-install.osh <your bundles dir>

Where you should replace <your bundles dir> with a path to a directory on your local file system that contains bundles.

Hmmm what to blog next...ponders...

Laters,

Nimble OSGi

So I just sent a rather cryptic twitter message with the instructions:

posh -kc "repos -l springdm;add org.springframework.osgi.samples.simplewebapp@active"

I figure it's probably worth a short note to explain what this is doing given the narrowband aspect of twitter communications.

This command is running an instance of the posh (sh)ell which ships with Nimble. There are two switch parameters parsed to the shell:

-c: Tells posh to execute the command passed in from the unix shell in the posh (sh)ell environment
-k: Tells posh to remain running after the command has completed and open a tty session for user input

Now we come to the actual commands:

repos -l springdm: tells posh to load the spring dm repository index into the nimble resolver

add org.springframework.osgi.samples.simplewebapp@active: tells nimble to resolve all dependencies for the spring simplewebapp from it's configured repositories.

The interesting thing about nimble resolution is that it doesn't just figure out the bundles that need to be installed. It also figures out what state these bundles should be in. If you look at the bundles in the nimble container using the command lsb you will see that not only are all the bundles installed but certain key bundles have also been activated:

lsb
*nimble/com.paremus.util.cmds-1.0.4.jar 00:00 59Kb
0 ACTIVE org.eclipse.osgi:3.5.1.R35x_v20090827
1 ACTIVE com.paremus.posh.runtime:1.0.4
2 ACTIVE com.paremus.posh.shell:1.0.4
3 RESOLVED com.paremus.util.types:1.0.4
4 ACTIVE com.paremus.nimble.core:1.0.4
5 ACTIVE com.paremus.nimble.repos:1.0.4
6 ACTIVE com.paremus.nimble.cli:1.0.4
7 RESOLVED javax.servlet:2.5.0.v200806031605
8 RESOLVED com.springsource.slf4j.api:1.5.6
9 RESOLVED com.springsource.slf4j.nop:1.5.6
10 RESOLVED com.springsource.net.sf.cglib:2.1.3
11 RESOLVED com.springsource.edu.emory.mathcs.backport:3.1.0
12 RESOLVED org.springframework.osgi.log4j.osgi:1.2.15.SNAPSHOT
13 RESOLVED com.springsource.org.aopalliance:1.0.0
14 RESOLVED org.springframework.osgi.jsp-api.osgi:2.0.0.SNAPSHOT
15 RESOLVED com.springsource.slf4j.org.apache.commons.logging:1.5.6
16 RESOLVED osgi.cmpn:4.2.0.200908310645
17 RESOLVED org.mortbay.jetty.util:6.1.9
18 RESOLVED org.springframework.osgi.jstl.osgi:1.1.2.SNAPSHOT
19 RESOLVED org.springframework.core:2.5.6.A
20 RESOLVED org.springframework.osgi.commons-el.osgi:1.0.0.SNAPSHOT
21 RESOLVED org.mortbay.jetty.server:6.1.9
22 ACTIVE org.springframework.osgi.samples.simplewebapp:0.0.0
23 RESOLVED org.springframework.beans:2.5.6.A
24 RESOLVED org.springframework.osgi.io:1.2.0
25 RESOLVED org.springframework.osgi.jasper.osgi:5.5.23.SNAPSHOT
26 RESOLVED org.springframework.aop:2.5.6.A
27 RESOLVED org.springframework.osgi.catalina.osgi:5.5.23.SNAPSHOT
28 RESOLVED org.springframework.context:2.5.6.A
29 ACTIVE org.springframework.osgi.catalina.start.osgi:1.0.0
30 RESOLVED org.springframework.osgi.core:1.2.0
31 RESOLVED org.springframework.web:2.5.6.A
32 RESOLVED org.springframework.osgi.web:1.2.0
33 ACTIVE org.springframework.osgi.web.extender:1.2.0
34 ACTIVE com.paremus.posh.readline:1.0.4
35 ACTIVE com.paremus.util.cmds:1.0.4

This listing also demonstates another key feature of nimble. Typing lsb resulted in the following log line:

*nimble/com.paremus.util.cmds-1.0.4.jar                           00:00    59Kb

This demonstrates that the nimble container resolved the lsb command from its repository index and installed it on the fly. In fact if you look at the Nimble download it is only 55K in size. All of the extra functionality is automatically downloaded based on information provided via the nimble index files and traversing package and service level dependencies!

To complete this blog post you can browse the simple web app running from nimble by opening:

http://localhost:8080/simple-web-app/

Nimble is available for download here.

Monday 30 March 2009

OSGi Dev Con 2009 & OSGi Tooling Summit Roundup

So I'm currently sitting in my hotel lobby waiting till we jump in the hire car off to the airport for our flight home. It's been a long week - the time zone difference between San Francisco and London is always exhausting and the mental effort of networking, presenting, and talking to customers is always pretty intensive too.

I think the conference was definitely quieter this year - you couldn't help but feel the shadow of the economic climate hanging over us all - however though volume was down compared to last year, quality seemed to be much higher. My rough market assessment based on talking to people on our exhibition stand was that awareness of OSGi is much improved compared to last year.

Even better for me people really seemed to get what we (Paremus) are about. Last year we were the "RMI guys". This year people we talked to seemed to get genuinely excited about what our product is a really about: a flexible, scalable solution to provisioning and managing dynamic distributed OSGi based applications in enterprise environments.

Pretty much everyone we talked to was very positive on OSGi as a runtime solution to Java modularisation. However, the problem for its adoption in the enterprise is the backwards compatability issues in moving to an OSGi environment (things like static references and Class.forName are a really bad idea in OSGi) and the fact that OSGi development relies on a reasonable amount of domain specific knowledge.

I think good tooling solutions that work right the way though the stack are crucial to help new developers though the pitfalls of the new classloader space. Unfortunately, tooling support for new developers is pretty disjointed.

Hence, the next part of my post...

We (Paremus) recognised that it does us no good if customers are having difficulty building OSGi based solutions. Therefore, my team mate Derek Baum and I have been working on a solution to developing OSGi bundles which works in both an IDE and headless build environment called Sigil. Prior to OSGi Dev Con Paremus chose to licence Sigil under the Apache licence, as we recognise that tooling is an area where we need support from the community in order to help the community as a whole.

On the Friday, after the end of the conference, I and a number of other representatives with interests in the area of development tooling met at an OSGi Tooling Summit hosted by Yan Pujante at LinkedIn's Mountain View offices. The group was pretty large and diverse (as you can see here). The OSGi Alliance's typical role in this respect is to foster communication (via summits, conference calls etc), then generate one or more RFPs which list a set of requirements for the problem domain, and finally put together a set of RFCs which provide a spec to allow different vendors to work together.

The format of the meeting was an initial intro and positioning statements (i.e. what we were there to achieve). We then moved into a blue sky discussion where we tried to think of a perfect world solution to the OSGi development lifecycle - this stimulated some really interesting conversation which I'll doubtless follow up on in the next couple of weeks in future blog posts. Finally, we put together a series of use cases headers that captured the major tasks undertaken by developers when building, maintaining and releasing OSGi applications into the wild.

I had a number of really encouraging conversations with Chris Aniszczyk and Peter Kriens who work on PDE and BND respectively, both of which have a lot of cross over with the work I've been doing on Sigil.

Chris has just twittered that he's thinking of changing the name of PDE to BDE which I think is a great idea - perhaps some merging of BDE, Sigil and BND? I think there is likely some common core classes that are useful to all OSGi developers. We're obviously going to need some domain specific extensions for plugin development, newton development, spring development, etc. but I see these as addons on top of a common OSGi layer.

I guess in a perfect world I'd like to be able to support Maven, Netbeans and IntelliJ users as well. Hopefully I'll be able to update you in the next couple of months on progress in this area.

Laters,

Wednesday 18 March 2009

Flocking Behaviours

I'm speaking at EclipseCon 2009

Just a quick note to say I'll be running a BOF session with Peter Kriens and Chris Aniszczyk on OSGi development tooling at EclipseCon this year as part of the OSGi track. You can find details here.

I guess my main focus for calling the BOF is that I'm very interested in talking to other OSGi developers to see what it is we on the tooling side can do to make our collective jobs easier.

What is it about OSGi development that really frustrates you - and what can tools do to make it easier?

It'll also be really interesting to meet up with other tools developers to see if we can get a bit of reuse going on - likely we won't solve that in one BOF but if we can just put names to faces and get a bit of shared understanding going on we're likely to be more productive in the long term.

Hope to see you there.

Laters,

Thursday 26 February 2009

OSGi tooling

There's a really interesting conversation going on at TSS about OSGi and future directions for Enterprise Java.

I've posted a reply which I thought it was worth reposting here:

I think there are two issues with [the approach of repackaging existing modules as OSGi bundles and simply importing/exporting all packages] which really cause headaches going forward; Module vs API dependencies and complex "Uses" graphs.

Firstly module vs api; in most dependency tools such as Maven and Ivy the developer specifies dependencies at the module layer - i.e.

<dependency org="org.apache.log4j" name="org.apache.log4j" rev="1.2.15" />

But then spring have added an OSGi version of the module which has a different module id.

<dependency org="org.apache.log4j" name="com.springsource.org.apache.log4j" rev="1.2.15" />

How does the tooling know that these two modules are actually compatible? It can't and this tends to lead to false dependency failures. This is the same problem as using module dependencies in OSGi at runtime. Actually the code is not dependent on the module but on the API that the module supplies.

Having been working with OSGi as part of our [1] distributed OSGi platform for almost 4 years now, I came to the conclusion that the tooling should use the notion of package import at build time - just as OSGi does at runtime. This is the approach I've taken in Sigil [2] which we're now using to build our next product release.

Having taken the jump to use API dependencies our build system is now much simpler to manage. Instead of having to manually lookup which module supplies a package, I just type the package into the IDE and our tooling goes off and trawls the OSGi manifest information in various repositories to find a module that supports it (seemless). We've also used this on a number of third party applications (the record being a project with over 600 modules - which we managed to convert to use sigil in less than a couple of hours based on pretty much a simple grep of the java code for java import statements).

Secondly the major thorn in the approach of naively exporting all packages in a module is the complex "uses" information it generates. "Uses" is a flag provided by OSGi on exports to assert that the class space is coherent across multiple bundle import/export hops.

It is currently believed that the "uses" information makes the problem of resolving bundle dependencies NP-complete. As the number of bundles and import/exports increases the number of calculations the OSGi runtime has to do to check that all bundles can sit together goes up at a terrifying rate.

I've referred to this as placing barbed wire around sand castles (in most cases). If the modules were more sensibly designed i.e. only exporting the "really" public code then this problem is much reduced.

I'm also attending the summit Jason refers to and I've called a BOF "OSGi Development Tooling" with Peter Kriens and Chris Aniszczyk at the OSGiDevCon conference just prior to the summit to discuss tooling efforts w.r.t. OSGi. The BOF is open to the general public and it would be great to hear opinions before we sit down at the summit to figure out a path forwards.

[1] http://www.paremus.com
[2] http://sigil.codecauldron.org