One of the other significant points about this fix pack is the move to WebSphere Application Server 7.0.0.11 and, in particular, the fact that this finally has an Installation Manger based install process. This means that you can finally install both the Application Server, ESB and fixes for both, all using the same install technology. Unfortunately you will have to use the old Update Installer to remove any ifixes that you have already installed.

Service Federation Management: I’ve touched on this previously in my post on the product announcement as it is the area on which my efforts have been focussed for this release. The basic thesis is that many enterprises consist of a number of disconnected domains each with their own services. These domains could correspond to lines of business, geographical boundaries, or exist as the result of mergers and aquisitions. What service federation management aims to do is to simplify the sharing of services from one domain to another. A Business Space based console is shipped with the WebSphere Service Registry and Repository Feature Pack. This console allows the definition of multiple domains, each of which has its own registry instance. Existing services in those domains can then be grouped and shared from one domain to another via the console. In addition, a domain may contain one or more connectivity providers (WebSphere Message Broker or WebSphere ESB). When a service group is shared from one domain to another, a proxy can be created in either the provider or consumer domain or both.

Patterns/Proxy Gateway: The existing pattern support in WebSphere Integration Developer has been radically updated. In addition, a new service gateway pattern has been added: proxy gateway. The existing dynamic proxy pattern requires the integration developer to determine what the target service for any given invocation is. The proxy gateway pattern allows back-end services to be defined in an internal registry via a new Business Space widget. Each service is provided with an unique virtual service name that can then be appended to the export endpoint address to identify the target service. A new endpoint lookup primitive is utilized in the flow to retrieve the target service meta-data from the registry. Consequently, a proxy gateway scenario can now be implemented out of the box.

Data Transformation: There have been considerable enhancements to the XML map support. XPath functions can now be accessed directly. “Smart” data type support allows direct mapping between compatible types e.g. date to dateTime conversion. User-defined lookups can be implemented with support for static relationship, CSV, and property file, lookups out of the box. If/else if/else constructs are also now supported. Refactoring support has been added. To name but a few enhancements…

Policy-Driven Mediation: The mediation policy Business Space widget that shipped with the V6.2 Feature Pack is now part of the base product. It is also now possible to attach policies to individual services, service endpoints and operations, in addition to the existing module level policies.

Subflows: Support for subflows (the area I worked on in V6.2) has also be enhanced. It is possible to set the types of the in/out nodes of a subflow to be weakly typed thereby promoting greater reuse. It is also possible to extract part of an existing flow in to a new subflow (although it is then necessary to define the wirings to the in/out nodes explicitly).

Mediation Primitives: No release of the product would be complete without a set of new mediation primitives:

• Message Validator – validates that the message matches the schema
• Trace – prints a trace message to the server log
• SLA Check – ensures that the service requestor has an SLA defined in WSRR
• Flow Order – allows the order of invocation to be defined where a mediation flow branches
• UDDI Endpoint Lookup – enables a service lookup based on business name, service name or find qualifiers
• Gateway Endpoint Lookup – as mentioned previously, allows service endpoint information to be retrieved based on a virtual service name

Mediation Flow Editor: When creating a new mediation flow component implementation you’ll immediately notice a difference to the editor. The operation connections view has gone and, when creating the flow, you are now presented with a set of template options permitting either a one-to-one or one-to-many mapping, or for you just to start with a blank canvas and add callout nodes to references as and when they’re needed. This takes a bit of getting used to for seasoned users but should be more intuitive for the beginner. Then, when editing the flow itself, the request/respones tabs have moved to the top of the editor alongside a new breadcrumb navigation, the combination of which makes it much easier to see which flow you’re currently working on. Another great usability improvement is the ability to click on any terminal in the flow and see the structure of the service message object at that point.

Quality of Service: Event sequencing, previously only supported in WebSphere Process Server, is now available in WebSphere ESB too. There is also a new store and forward qualifier. This can be applied to asynchronous invocations and indicates that, when a given exception occurs (e.g. ServiceUnavailableException), the request messages should be requeued ready to be forwarded at a later point.

Bindings Enhancements: The web service bindings have had support for referenced attachments added. For a JAX-WS binding, it is also possible to modify the response endpoint dynamically within the mediation flow enabling responses to be redirected to a different client application. For the messaging bindings, there is now failed event manager support for all of the binding types. With the move to be based upon WAS V7, the WebSphere MQ related bindings can now also make use of the WebSphere MQ JCA resource adapter. For a JMS import where the response is being sent to a topic, it is also now possible to modify the topic dynamically during the mediation flow. The EJB binding can now be specified on an export as well as an import with support for both EJB 2.1 and 3.0, local and remote interfaces.

WebSphere Service Registry and Repository Integration: In addition to the new SLA check primitive already mentioned above, the existing endpoint lookup primitive has been enhanced. It is now possible to specify the required version dynamically. Lookup for JMS, HTTP and MQ binding types is also now supported. For situations where it is necessary to access WSRR from a custom Java primitive, for example to perform a custom query or to retreive other objects types, it is now possible to use the same client SPI used by the endpoint lookup primitive. This enables custom code to use the same WSRR definitions and caching mechanism used by the provided primitive.

new Service Federation Management across WebSphere Service Registry and Repository and the ESB family to manage service visibility and reuse across the enterprise

Maybe that doesn’t tell you a great deal but I’m not at liberty to say any more at the moment. You could, however, turn to the WSRR announcement which provides a little more detail:

Better manage visibility of service across SOA domains. By using the Service Federation Management console to simplify the process of sharing business services between service domains, you can improve the connections within an organization, and increase reuse of services and information by extending the reach of existing assets across the organization and beyond.

And this might lead you to Greg Flurry and Marc-Thomas Schmidt’s developerWorks article which provides the motivation for this new functionality.

More to come once we reach general availability…

If you’re using a JAX-WS binding then you need to take a different approach. In the server runtime, define a new policy set that contains a HTTP transport policy as described in the Application Server InfoCenter. This policy set has a Connection timeout property (in addition to many other settings). Export the policy set and then import it in to WebSphere Integration Developer (File > Import then Web services > WebSphere Policy Sets). At this point I then had to restart WebSphere Integration Developer to pick up the new policy set. Then select the import on the assembly diagram and on the Properties tab select the Binding > Policy Sets panel. Here you can select the default policy set for the binding.

$AdminTask modifySCAImportHttpBinding -moduleName MyMod -import Import1 -connectionRetries <Import1>3</Import1>

Thankfully the InfoCenter has been recently updated as a result of APAR JR31724 as previously you just had to guess at the syntax! Perhaps I should now try and get them to fix the description of the command which I see states “The modifySCAImportHttpBinding command changes the JNDI name of one or more of the resources associated with an HTTP import binding.”! One suspects someone may have been a little generous on the old cut and paste…

Now that is all very well but it is cumbersome and error prone if you have a large number of properties to modify. I’d also recommend that application deployment, as with any administrative action in anything but a workstation development environment, be performed by scripting. There is no special command for installing mediation modules (they just use the standard AdminApp.install command) so how to specify the module properties? An intial search of the InfoCenter failed to turn up anything useful so I resorted to running the AdminApp.installInteractive command. This showed me the syntax required for specifying the properties on the command line but I still needed to know the name of the corresponding task. I then looked at the wsadmin.traceout file in the profile logs directory where a entry is added showing the generated command. For example:

[20/05/09 16:15:38:000 BST] 0000000a AdminAppClien A   WASX7278I: Generated command line: install 'c:/temp/module.ear' '[ -SIBSCAClientInstall [[, [group1]key1=value1,[group2]key2=value2]]]'

A little bit of massaging and I had the necessary Jython needed to modify properties on application installation:

AdminApp.install('c:/temp/module.ear',[ '-SIBSCAClientInstall', [[', [group1]key1=value1,[group2]key2=value2']]])

It wouldn’t take much to read the property names and values out of a property file to allow the necessary parameterisation for different environments.

The only place I found this task documented was in a very good IBM Education Assistant presentation covering promoted properties. Unfortunately this presentation only covers V6.1 so doesn’t show the syntax required for group names.

Lastly, an alternative approach (although not one that I’d necessarily advocate in terms of following a supported process) would be to have multiple copies of the sca-module-properties.xml file and build a different EAR for each environment. This file contains the default values for each promoted properties in the following format:

<sCAModuleProperty identifier="key1" value="value1" group="group1"/>

1. The syntax of the endpoint URI used for the JMS bindings uses JNDI names to refer to the resources and is therefore not truly dynamic in the sense that JMS destinations need to have been pre-configured for all of the messaging provider destinations to which you might want to send.
2. For a native MQ binding, you can currently only modify the queue dynamically and not the queue manager to which you can connect.

There are, however, two relatively new innovations that make this picture more complex. Firstly, WebSphere MQ v7 brings to the distributed platform the browse-mark behaviour that previously existed only on the mainframe. This basically allows the connection consumer to lock messages as it browses the queue thereby ensuring that, when it decides to consume one, it is still available. Secondly, ASF is now old technology with the J2EE Connector Architecture 1.5 specification providing the preferred mechanism for integration between a JMS provider and application server MDBs. In WebSphere Application Server v7 this manifests itself as the WebSphere MQ resource adapter.

The good news is that, if you have a WebSphere MQ v7 queue manager, then the WebSphere MQ v7 JMS provider will automatically use the new browse-mark behaviour for both ASF and the resource adapter. If you are currently using non-ASF, you should therefore switch away from that to one of these two mechanisms when moving to WebSphere Application Server v7. However, if you still have a v6 queue manager and have previously been using non-ASF then you should continue to do and not switch to the resource adapter as this uses the same underlying technology as ASF.

Thanks to Ben Ritchie, Mark Whitlock and Matthew White for providing insight in to the new behaviour.

The two biggest deliverables from the team here in Hursley are the service gateway and policy support.  The dynamic service gateway support addresses one of the major limitations of the product to date – the fact that the strong typing of SCA interfaces made it very difficult to implement a gateway pattern, particularly for web services. The dynamic gateway implementation provides a very simple interface with two operations: one for request-reply operations and one for one-way operations. The export binding will perform the mapping from the incoming request to one or other of these operations. The general pattern is then to perform some common processing before using a lookup (WSRR or otherwise) to locate the required target provider.

The policy support comes in two parts. A new policy resolution primitive enables the retrieval of policy information relating to a service from WebSphere Service Registry and Repository. The Service Message Object (SMO) has then been enhanced by the addition of a dynamic context. Properties retrieved from the policy are used to populate the dynamic context. These properties then override properties promoted from primitives in the flow. The ability to dynamically modify the behaviour of the flow at runtime, combined with the service gateway support, really opens up the possibilities for creating a meta-data driven ESB with WebSphere ESB.

When you install the runtime you will notice that the Web Services Feature Pack is also installed. The service gateway utilizes the JAX-WS support added by the feature pack and this brings with it support for SOAP 1.2 and WS-ReliableMessaging.

My personal baby is the new concept of mediation subflows. The componentization in SCA already provides for reuse but sometimes this is needed at a much lower level. In many ways, a mediation subflow is just like a regular flow – it consists of a number of mediation primitives wired together. The biggest difference is that, rather than exposing interfaces, the in and out nodes of a subflow specify a message type or may even be left untyped. To promote greater reuse, the subflow can also be defined in a library.

To use a subflow, a new mediation subflow primitive has been added. The input and output terminals of this primitive match the in and out nodes of the subflow with which it is associated and properties promoted from primitives in the subflow become properties of the subflow primitive. A subflow can also contain mediation subflow primitive so nesting is possible. I could say much more about all of this but I’ll save the detail for another day (or a developerWorks article if I get my act together).

While talking about new primitives, there are four additional primitives that allow easy manipulation of MQ, SOAP, HTTP and JMS headers. There is a new primitive that performs filtering based on the type of the message and another that allows a data handler to be used to transform a request to/from the raw wire format. Both of these become particularly useful in the gateway context where multiple different message types may be arriving in a single flow. The message logger primitive has also been enhanced so that it can now be used to perform custom logging (the default being to java.util.logging) rather than just to a database.

On the aggregation front, we now have the ability to perform some asynchronous processing between fan out and fan in nodes. This means that if, for example, the aggregation block contains an asynchronous service invoke, the next leg of the block can start processing whilst waiting for a response.

Two major changes at the module level: a mediation flow component can now be placed in both business and mediation module; and multiple mediation flow components can be placed in the same module. The former means that, if you need to perform pre- and post-processing on, for example, message headers, this can now be done in the same module as the business logic. It is also now possible to specify version numbers for modules and tie this in to service selection.

Lastly on the runtime front, the failed event manager finally comes to WebSphere ESB and has been enhanced to cover many more scenarios such as failures in the bindings.

On the tooling side, I won’t provide quite so much detail but here are some of the new features in WebSphere Integration Developer that I’ve already found useful:

• Java context assist for BO fields when using the SDO API
• Improved automatic wiring layout
• Filtering in the mapping editor
• Ability to view transaction boundaries within and across modules
• Insertion of primitives by drag-and-drop on to a wire
• Wiring unmatched terminal types results in automatic insertion of an XSLT primitive
• Formatted server logs view with filtering
• Improvements to the cross component trace including support for the bindings

Whilst this post by no means provides a comprehensive list of the new function, I hope that it provides a good flavour of the significant changes that have gone in to this release. I certainly wish I’d had even half of these features when implementing WebSphere ESB solutions out in the field!

I started on 5 October 1998, straight out of university (a degree in Engineering and Computer Science from Oxford University). This was a month later than the majority of that year’s graduate intake as I had been otherwise engaged with a leisurely crossing from New York up to Quebec and then across to Vancouver by Greyhound (the coach company that is, not the dog). I began work as a tester on Component Broker, IBM’s CORBA offering and a forerunner for much of what was to become WebSphere Application Server. It was only a few months before I’d managed to wheedle my way across in to the development team, initially fixing defects in the transactions component.

I then had my first brush with the world of messaging working on the MQSeries Application Adapter component. Next up was some work on the common Java client which was to be a sign of things to come. As IBM threw its weight behind Java, Component Broker sank beneath the waves (although I still drink for my CB mug) and WebSphere Application Server became the way forwards. I continued to work on the transactions service and related activity session service before starting to look at the integration of WebSphere MQ with Version 3.5 of the Application Server.

This (along with a part-time MSc in Software Engineering back at Oxford) kept me gainfully employed until someone had the bright idea of writing a new Java messaging provider for the Application Server from the ground up. I was originally slated to work on the mediation component of what was to become known as the Service Integration Bus but somehow that never happened. Instead, having successfully persuaded the powers that be that the J2EE Connector Architecture was the way to integrate the new JMS provider with the Application Server, I found myself owning the resource adapter component.

Once Version 6.0 had safely shipped I felt that, rather than making design decisions in the rarefied atmosphere of Hursley, I needed to get out and meet some customers. This saw me transfer across to the IBM Software Services for WebSphere organisation. I consequently had the dubious pleasure of explaining to customers that, yes, I had designed and coded that particular piece of function in the product and, no, I have no idea at the time how I was expecting people to use it!

My four years in Software Services culminated in a six month assignment to Norway about which you can read more elsewhere on this blog. I was getting travel weary though and, with a young family, sought a little more predictability in my schedule. Consequently I found myself back sitting not very far away from where I had sat when I left development but now working on WebSphere ESB. Two out of the three guys I originally shared a house with when I first joined IBM also work on the same product (the third having left IBM)!

When I joined IBM I had no great plans to still be a part of the company ten years later but I’ve enjoyed the variety of work, the Hursley location and, most importantly, the clever folks that I’ve had the pleasure of interacting with on a day-to-day basis. Who knows where I’ll be in another ten?!