Thursday, 25 October 2012

JDK8 - the Scala killer?

With Java taking a long overdue step toward functional language capabilities, the Java release scheduled for next year (2013) should provide the Java development community with a language feature that results in it providing a serious and compelling alternative to Scala as a JVM language with functional capabilities.

Evolution of Java, as a language, has stagnated somewhat during the last decade. Since the introduction of Generics nothing of major significance has been delivered. However, JDK8 is going to be a game changer, specifically by the introduction of lambda expressions.

Get ahead of the game by getting involved in debate and education around lambda expressions directly with a Java book author (Java Generics & Collections - Maurice Naftalin) and take the opportunity to learn, discuss and debate the new feature. You will notice that Maurice Naftalin has also strong associations with the Oracle Java Language & Tools team.

In order to join the discussion and debate go to Maurice Naftalin's Lambda FAQ

Tuesday, 25 September 2012

C24 and mongoDB - Agile Message Management (Vickery, Porter, Roberts)

AUTHORS

Matt Vickery - C24 Technologies & Incept5                                                  Daniel Roberts - 10gen
Iain Porter - C24 Technologies & Incept5

ABSTRACT
C24 and mongoDB  - Agile Message Management. This article presents and demonstrates a natural combination of two enterprise software products  - C24 Technologies C24 iO Studio and 10gen's mongoDB. Both technologies will be introduced, key feature sets outlined and a practical demonstration of their combined capability provided through working code. The primary driver for promoting these two technologies is that they form a powerful toolset that has underpinned agile software delivery for several significant enterprise applications that required non-trivial messaging and data storage capability.



Monday, 25 June 2012

Spring and C24 iO - Part 2 C24 Spring Integration


The focus for Part 2 of the series (Spring and C24 iO - Accelerating Enterprise Data Services Deployment) concentrates on integration namespace capabilities that have been provided by C24 for building integration flows and, through example, explores the use of C24 iO within those flows.

Some contextual information may be required in order to better understand the example code and configuration. SWIFT is a messaging standard used by most of the world's financial institutions. If you've not come across them before, SWIFT messages look like someone has danced on a keyboard so don't worry about understanding the content. The important aspect is that it's a message type that is standard, well adopted and complex enough that it's difficult to write custom code in order to parse. SWIFT messages (ISO-15022) are usually specified by type using a number, so, in this example you will encounter an MT540 SWIFT message, this is a Receive Free message sent from an account owner to an account servicer. If you don't know SWIFT but you either use C24 iO or perhaps are thinking about it, read on, it's not a major obstacle.

C24 Spring Integration Namespace Support 

C24 Spring Core provides capability to use iO model objects as Spring beans using very simple XML namespace or @Configuration annotated classes to populate the Spring container.

The Spring Integration framework provides capability for Spring Integration Messages (org.springframework.integration.Message) to be handled in a uniform way by integration constructs such as gateways, splitters, aggregators, header enrichers, filters and transformers; that list is not exhaustive. These integration constructs have no knowledge of message payloads, their types or their content.

Although the Spring Integration framework offers a set of constructs that support lightweight messaging, it does not natively support specialist data-binding, transformation and validation as this would require a more intimate knowledge of the message payload as well as tooling around message models, validations and transformations. Spring Integration provides the facility to plugin code that can perform those functions, i.e. via Service Activators, but this means that configuration and distribution of that code is much more difficult, i.e. it would always result in developers having to write and share custom Service Activator (Java) code. This is a requirement that we are really trying to avoid as there is a much more attractive solution available. SpringSource has partnered with C24 to provide native capability around 12 three core aspects of message processing: 1) specialist binding, 2) validation and 3) transformation. 

Extending the semantically rich offerings of Spring Integration constructs by adding C24 iO capability required doing two things. Firstly, a namespace was constructed that specified syntactic requirements for those three types of message processing construct. Secondly, a set of Spring Integration beans have been built that are able to process iO model objects as Spring Integration message payloads. 

This work was undertaken by SpringSource  engineers early in 2011. The partnership between C24 and SpringSource has resulted in native capability within Spring Integration to process iO model objects in ways described in the following sections.

C24 Namespace Constructs

iO Object Unmarshalling-Transformer

The C24 Spring Integration Unmarshalling Transformer construct provides capability for the Spring Integration Framework to receive a message in any supported data format and output it as a C24 iO model object.

By way of example, the Spring Framework may supply a raw SWIFT MT541 message String (as shown in Diagram 3) to the transformer below. The result is that an MT541 model object will be returned. Note that no custom Java code had to be written in order for this to happen, merely simple namespace configuration.

The Unmarshalling Transformer namespace construct to use for this purpose is:

    <c24:unmarshalling-transformer
        id="c24Mt541UnmarshallingTransformer"
        source-factory-ref="textualSourceFactory"
        model-ref="mt541Model" input-channel=”unmarshall-channel”
        output-channel=”enrichment-channel”/>

The source-factory-ref is the name of the parser that will bind the input to this transformer into an object described by the bean reference model-ref.


iO Object Marshalling-Transformer

The C24 Spring Integration Marshalling Transformer construct provides capability for the Spring Integration Framework to pass in a C24 iO model object and output it in an alternative, and configurable, format. In the example below, a message containing an iO model object can be passed-in, a JSON (java.lang.String) representation of the object is returned.


Note that an output format is chosen by specifying a sink-factory attribute on the marshalling element. The Marshalling Transformer namespace construct to use for this purpose is:

              <c24:marshalling-transformer
        sink-factory="biz.c24.io.spring.sink.JsonSinkFactory"
        output-type="STRING"   
        input-channel=”marshall-channel”
        output-channel=”storage-channel”/>

Some of the sink-factory attributes that are currently supported are for: XML, JSON, CSV and Java Source. 

iO Object Transformer

The C24 Spring Integration Transformer construct provides capability for the Spring Integration Framework to pass in a C24 iO model object and receive output as another C24 iO model object.

In most cases the source and target iO model object are different types, but that’s certainly not a prerequisite for transformation. The single most significant parameter is the one that identifies the target or output iO model object type.

The Transformer namespace construct to use for this purpose is:

              <c24:transformer
        transform-class="biz.c24.io.swift2012...MT541ToFlatTransform"
         input-channel="transformation-channel"
        output-channel="reply-channel"
          always-return-array="false"/>

iO Object Validating Header Enricher

The C24 Spring Integration Validating Header Enricher is a type of transformer that provides capability for the Spring Integration Framework to pass in a C24 iO model object and receive output in the form of validation results.

Results are loaded as [K,V] pairs into the Spring Integration header, this means that they can be accessed by other integration constructs or even SpEL expressions directly.

The Validating Header Enricher namespace construct to use for this purpose is:

              <c24:validating-header-enricher
        fail-events="true" pass-events="true"   statistics="true"/>

The result of using the construct in this way will mean that all fail events get loaded into an SI message header named “c24_failEvents”, all pass events into a header named “c24_passEvents”, and statistics for those operations into a header named “c24_statistics”. Default values for that namespace are true for fail events, false for pass events and statistics.


iO Object XPath Transformer

The C24 iO XPath Transformer is a type of transformer that provides capability for the Spring Integration Framework to pass in a C24 iO model object and receive output data of various types including object, string, boolean and list. The data that’s obtained from using this integration construct is derived directly from the iO model object payload.


The XPath Transformer namespace construct to for this purpose is:

      <int-c24:xpath-transformer input-channel="..."
        evaluation-type="OBJECT_RESULT"
        xpath-statement="/MT541Message/Block4/SeqB/Field90aPrice"
        input-channel=”price-lookup-channel”
          output-channel="deal-price-recorder-channel"/>

iO Object XPath Router

The C24 Xpath Router is a type of message router that provides capability for the Spring Integration Framework to pass in a C24 iO model and have messages routed to a specified channel determined by an XPath expression.

This router is much the same as the pure Spring Integration router except that support is added that enables iO model objects to be routed to channels, determined by XPath expression results invoked against that data.

The XPath Router construct to use for this purpose is:

              <c24:xpath-router
        xpath-statement="/MT541Message/Block4/SeqA/.../G/Function"
        input-channel="message-router-channel">
            <c24:mapping channel="new-message-channel"
                value="NEWM"/>
                <c24:mapping channel="cancellation-message-channel"
                value="CANC"/>
                <c24:mapping channel="preadvice-message-channel"
                value="PREA"/>
    </c24:xpath-router>

As with a standard router configuration, you can also specify a default-output-channel that’s used if a single message channel cannot be chosen from the set provided. It is also possible to configure a bean reference that can be used to provide the XPath statement.

iO Object XPath Selector

The C24 XPath Selector is a type of Filter that provides capability for the Spring Integration Framework to pass in a C24 iO model object and determine if the message will continue to undergo processing. That decision is based on the result of a comparison between payload content data and data provided in the namespace construct. If the message is not filtered it will be returned as the payload to the Spring Integration Framework, otherwise it is discarded.


The XPath Selector namespace construct used for this purpose is:

              <c24:xpath-selector
        evaluation-result-type="boolean"
        xpath-statement="/MT541Message/Block4/SeqA/.../G/Function"
        string-test-value="CANC"/>

In this example, only SWIFT messages that have a cancellation function will continue to undergo processing, all others will be discarded. Once again, the XPath statement can also be supplied as a reference to a Spring bean.

C24 Namespace Examples

Given a @Configuration class such as the following, two beans will become available from within the Spring container. Firstly, the textualSourceFactory is the factory object that will provide the instance of the SWIFT parser to any namespace construct that requires it. Secondly, an MT541 model object will also be made available to any namespace construct that requires it, i.e. any namespace construct that performs marshalling/unmarshalling or transformation involving MT541 models.


@Configuration
public class C24SpringConfiguration {

    @Bean(name = "textualSourceFactory")
    public TextualSourceFactory getTextualSourceFactory() {
      TextualSourceFactory textualSourceFactory 
          = new TextualSourceFactory();
      textualSourceFactory.setEncoding("UTF-8");
      return textualSourceFactory;
    }

    @Bean(name = "mt541Model")
    public C24Model getMT541Model() {
      Element element 
        = new biz.c24.io.swift2011.MT541Element();
      return new C24Model(element);
    }
}

Of course, you don't have to use a @Configuration class to populate the Spring container, you can use an XML descriptor to declare the beans instead.

The sample project reads a raw SWIFT message and then progresses it through a series of pipes and filters. The single namespace construct that binds a raw SWIFT MT541 into Java code is the following.

<c24:unmarshalling-transformer 
      id="c24Mt541UnmarshallingTransformer"
      source-factory-ref="textualSourceFactory" 
      model-ref="mt541Model"/>

That's all that's necessary to convert a raw SWIFT message like this:

{1:F01DRESGB2LAXXX0548034693}{2:O5410947040127FRNYUS33AXXX42181834250401270947N}{3:{108:valid}}{4:
:16R:GENL
:20C::SEME//FRTJ123REC2
:23G:NEWM
:16S:GENL
:16R:TRADDET
:98A::TRAD//20000519
:98A::SETT//20000524
:90A::DEAL//PRCT/101,001283
:35B:ISIN GB0987654321
:16S:TRADDET
:16R:FIAC
:36B::SETT//FAMT/4000000,
:97A::SAFE//222S
:16S:FIAC
:16R:SETDET
:22F::SETR//TRAD
:16R:SETPRTY
:95P::SELL//DEUTDEFF
:16S:SETPRTY
:16R:SETPRTY
:95R::DEAG/CRST/456
:16S:SETPRTY
:16R:SETPRTY
:95R::REAG/CRST/123
:16S:SETPRTY
:16R:SETPRTY
:95P::RECU//DRESDEFF
:16S:SETPRTY
:16R:SETPRTY
:95P::BUYR//MGTCDE55
:97A::SAFE//111S
:16S:SETPRTY
:16R:SETPRTY
:95P::PSET//CRSTGB22
:16S:SETPRTY
:16R:AMT
:19A::SETT//GBP4047151,3
:16S:AMT
:16S:SETDET
-}

into a C24 Java object. Once that's done, as a developer you can change SWIFT message values, perform semantic validation or anything else that you choose to do with the Java object.

The sample project contains a number of C24 models and transformations, we've looked at the MT541 model but what about transformations? In order to transform data from one form or type to another we can use a C24 iO transformer. These transformers take data contained in a source model object and transfer them into a target model object. These two models can be the same type but more often than not, the transformation takes place between two distinct models.

The object that is passed into the C24 iO transformer namespace construct is already defined by a model (it can't be a C24 iO object otherwise), so the only other configuration that is necessary to trigger a transformation into a target object is a reference to the transformation model itself. As in the example project, that can be achieved using the following configuration:

<c24:transformer 
  transform-class="..swift2011..MT541ToFlatTransform"
  input-channel="mt541-transformation-channel"
  output-channel="transformation-gw-reply-channel"
  always-return-array="false"/>

The transformation class, automatically generated by C24 iO during deployment of the transformation model, will receive the MT541Message that's passed into it and output a single XML string. This XML string automatically becomes the Spring Integration message payload and will be delivered to the next namespace construct in the flow by the Spring Integration payload.

This is just a very small and specific taster of C24 Spring samples, further examples will be published in future articles.

Tuesday, 19 June 2012

New C24 Spring Whitepaper

In conjunction with C24, I've just released a new C24 whitepaper titled: 

Extending Spring with C24 Integration Objects (iO): http://bit.ly/LvkfAK

Abstract:


Extending Spring with  C24 Integration Objects (C24  iO). This whitepaper reviews the power of combining  two enterprise software products from C24 Technologies (C24) and SpringSource. Working together, these products provide highly robust,  scalable and maintainable Java application and data integration platform tools aimed at enterprise business solution development. SpringSource has partnered with C24 to  extend Spring with native message processing capabilities for specialist binding, validation and transformation. This paper will demonstrate a harmony between C24 and SpringSource products by exploring the offerings from each company and strength  of the natural technical relationship between them.  Through technical insight,  it will become clear  how the combination of these products can help the enterprise gain a key competitive advantage through low-cost, best-in-class, rapid solution delivery.

Thursday, 3 May 2012

Spring and C24 iO - Part 1 C24 Spring Core


During very early stages of data processing software flows, it’s often necessary to take some type of raw data and parse it into a specific application software language construct in order that the application can progress that raw data through the business towards a conclusion. With XML being used extensively throughout software applications, and Java a very popular programming language, a good example of this would be a process that results in XML being available as Java objects. This process is often known as binding but is sometimes referred to as transformation. In an attempt to maintain common ground and maintain clarity, the process shall be referred to throughout these articles as a binding-transform.

The XML/Java binding-transform is an interesting example, popular technologies for performing such actions are JAXB & Castor; amongst a large cast of others. However, the specific value that C24 iO adds in this regard is that it’s not tied to any specific input data type such as XML - in fact iO models are strictly decoupled from input and output data types in order that you can freely choose or interchange between them when appropriate. 


C24 iO models represent business objects. As a developer you can choose a number of input data types to use to populate those objects with business data, CSV, XML, SWIFT, JSON, FpML, Columnar etc. The same is also true for output, you can choose a data format for those business objects when you’re ready to perform a reverse of the bind-transform should you need to dispatch data downstream in a specific format. The facility to do this binding-transform, and the reverse, is provided by C24 iO, out-of-the-box - no additional development is necessary.

In order to build a C24 iO model an understanding of your business object is necessary. The iO model can be built automatically from a sample message in many formats or from an external model such as an XSD. Alternatively, if you wish to maintain your business model using C24 iO, you can use iO Studio to build it from scratch. Model building terminology should seem familiar and intuitive, due mainly to the language that’s used for constituent parts, most people understand elements, attributes, simple and complex types or will quickly learn the difference when using this graphical tool.

The one remaining piece of the puzzle is the mechanism that allows these models to be used in application software at runtime. Once models have been created and tested in iO Studio, a triggering of code generation produces a Jar file that contains everything necessary to read data into the business model (from various source data types), interact with it through intuitively named methods and write it out again (into various target data formats). 

Each business object generated from an iO model is automatically part of a type hierarchy known as a Complex Data Object (CDO). Any standard or custom model that you use or build within C24 iO Studio will generate code that will implement the CDO type.

The c24-spring-core namespace provides capability to use CDOs as Spring beans either within applications using the Spring framework or as Spring beans within applications using the Spring Integration framework.

The namespace is http://schema.c24.biz/spring-core, the XSD is located at http://schema.c24.biz/spring-core.xsdUsing the c24-spring-core namespace within your Spring application means defining an interest in that namespace and location using something like the following configuration:

xmlns:c24="http://schema.c24.biz/spring-core"
xsi:schemaLocation="http://schema.c24.biz/spring-core 
http://schema.c24.biz/spring-core.xsd"


Once the namespace configurations have been completed, a namespace construct can be used to generate a C24 iO bean and use it within a Spring application. Those readers familiar with SWIFT FIN messages will recognise this example as a message model representing a securities trade message, an MT541.

<c24:model id="mt541Model" base-element="biz.c24.io.swift2011.MT541Element"/>


Once the Spring container is loaded with this bean, the developer is at liberty to interact with this business object in the same way that you would when using it outside of the Spring container. The benefit gained here is that it now has a lifecycle controlled by the Spring container and can be used by other beans within the same context, this will be explored further in the next part of the series.


The motivation for Part 1 of this series was an introduction to C24 iO, a brief explanation of some of it's basic capability and an introduction to iO models used as Spring beans.

Spring and C24 iO - Accelerating Enterprise Data Services Deployment


The focus for this series of blog articles is to explore the coupling between SpringSource's Spring Integration framework and C24s Integration Objects (iO) product. The primary driver for this showcase of two technologies is to demonstrate a significant potential time-to-market advantage for building, delivering and deploying Enterprise Data Services solutions.

SpringSource's Spring Integration is a framework and extension of the de facto standard EIP patterns described by Gregor Hohpe & Bobby Woolf [Enterprise Integration Patterns, ISBN 0321200683, Addison-Wesley, 2004]. Spring Integration also provides a veneer (via XML namespacesor Java code) over common Spring Core APIs and significantly, a messaging capability within and between applications.

C24 Integration Objects (iO) is a high-performance integration tool that, through model construction, supports Java binding capability and through transformation construction, supports transformations between models. C24 is a strategic SpringSource partner and well recognised in the finance industry as a leading integration tool provider - due primarily to the vast array of financial messaging models supported, low latency processing and the graphical modelling capability of C24 iO Studio. Furthermore, C24 iO models can be constructed as semantically rich components in order to employ strict validation against customisable rules that go far beyond capabilities of technologies such as the XML Schema Definition (XSD) language constraint model.

Together with C24, SpringSource have built two custom namespaces for C24's iO product that together, provide capability for a closely-coupled integration between Spring, Spring Integration and iO. The first namespace, c24-spring-core provides a facility to use C24 iO models directly within any Spring or Spring Integration application. The second namespace, c24-spring-integration, provides a facility to use C24 iO models within Spring Integration flows. This brings a time-to-market advantage for clients who wish to take advantage of C24 iO and a robust integration framework - in many cases, very little custom Java code is necessary to get operational flows deployed.

The focus of the following series of articles is to present key aspects of Spring Integration and C24 iO as high-performance, key, integration technologies. Additionally, and most significantly, an exploration will be undertaken of capabilities of the two technologies (Spring & iO) coupled together in order to deliver a client integration solution that forms the basis of a financial application installation.

A single sample project is used through the series, mainly to demonstrate code and configuration around a single business theme. This sample project is based on an operational flow built to process SWIFT FIN messages, all of the source code and configuration will be made available in the set of references that will appear as the last article in the series. 

Part 1 in the series provides some background information regarding C24 iO as a tool, it's core capabilities and relationship with the Spring container.

Part 2 details the capabilities that have been provided by C24 iO for building integration flows and, through example, explores the use of C24 iO within those flows.

Part 3 demonstrates a typical Spring Integration based solution that uses C24 iO to provide binding-transform and transformation capability. Because this flow is not contrived, it demonstrates many of the issues that have to be tackled during design and implementation of integration projects that use the Spring Integration framework.

Part 4 is all about testing integration flows. After having used Mockito on several large projects to test Spring Integration flows, an interesting testing strategy is presented.

Part 5 is a short post and is really a container for all of the reference material used and a guide to the sample project available from Github.



Tuesday, 20 March 2012

Git - Simple File Management

Files can be added, committed and removed from git repositories using one or more of the following commands:

Adding a file named "testFile.xml" to the index.

git add testFile.xml

Testing to view differences between the index and HEAD

git diff --cached

produces something like the following:

lizard:l8mdv-si-samples mvickery$ git diff --cached
diff --git a/testFile.xml b/testFile.xml
new file mode 100644
index 0000000..e69de29

This has not been added to the repository yet but remains in the index. Committing the file to the local repository involves promoting changes from the index to the repository, a suitable command could be something like:

git commit -m "Creating content" testFile.xml

Assuming no other files have been created and added to the index, running "git diff -- cached" after performing the commit should result in nothing being returned from the command invocation. The index and HEAD are now synchronised.

The local repository contains the new file, if you're using GitHub you may want to push the new file to the remote repository, this can be done with a command such as the following:

git push -u origin master

Additionally, if you're using GitHub as a remote repository, you can remove files from that repository by using a commands such as the following:

Removal from index, local repositoryremote repository and the local file system:

git rm testFile.xml
git commit -m "Test file removal"
git push -u origin master

Removal from the index, local repository, remote repository but not the local file system:

git rm --cached testFile.xml
git commit -m "Test file removal"
git push -u origin master

Thursday, 23 February 2012

Spring Integration - Robust Splitter Aggregator

A Robust Splitter Aggregator Design Strategy - Messaging Gateway Adapter Pattern

What do we mean by robust?
In the context of this article, robustness refers to an ability to manage exception conditions within a flow without immediately returning to the caller. In some processing scenarios n of m responses is good enough to proceed to conclusion. Example processing scenarios that typically have these tendencies are:
  1. Quotations for finance, insurance and booking systems.
  2. Fan-out publishing systems.
Why do we need Robust Splitter Aggregator Designs?
First and foremost an introduction to a typical Splitter Aggregator pattern maybe necessary. The Splitter is an EIP pattern that describes a mechanism for breaking composite messages into parts in order that they can be processed individually. A Router is an EIP pattern that describes routing messages into channels - aiming them at specific messaging endpoints. The Aggregator is an EIP pattern that collates and stores a set of messages that belong to a group, and releases them when that group is complete.

Together, those three EIP constructs form a powerful mechanism for dividing processing into distinct units of work. Spring Integration (SI) uses the same pattern terminology as EIP and so readers of that methodology will be quite comfortable with Spring Integration Framework constructs. The SI Framework allows significant customisations of all three of those constructs and furthermore, by simply using asynchronous channels as you would in any other multi-threaded configuration, allows those units of work to be executed in parallel.

An interesting challenge working with SI Splitter Aggregator designs is building appropriately robust flows that operate predictably in a number of invocation scenarios. A simple splitter aggregator design can be used in many circumstances and operate without heavy customisation of the SI constructs. However, some service requirements demand a more robust processing strategy and therefore more complex configuration. The following sections describe and show what a Simple Splitter Aggregator design actually looks like, the type of processing your design must be able to deal with and then goes on to suggest candidate solutions for more robust processing.

A Simple Splitter Aggregator Design
The following Splitter Aggregator design shows a simple flow that receives document request messages into messaging gateway, splits the message into two processing routes and then aggregates the response. Note that the diagram has been built from EIP constructs in OmniGraffle rather than being an Integration Graph view from within STS; the channels are missing from the diagram for the sake of brevity.





SI Constructs in detail:


Messaging Gateways - there are three messaging gateways. A number of configurations are available for gateway specifications but significantly can return business objects, exceptions and nulls (following a timeout). The gateway to the far left is the service gateway for which we are defining the flow. The other two gateways, between the Router and Aggregator, are external systems that will be providing responses to business questions that our flow generates.

The Splitter - a single splitter exists and is responsible for consuming the document message and producing a collection of messages for onward processing. The Java signature for the, most often, custom Splitter specifies a single object argument and a collection for return.

The Recipient List Router - a single router exists, any appropriate router can be used, chose the one that closely matches your requirements - you can easily route by expression or payload type. The primary purpose of the router is route a collection of messages supplied by the splitter. This is a pretty typical Splitter Aggregator configuration.

Aggregator - a single construct that is responsible for collecting messages together in a group in order that further processing can take place on the gateway responses. Although the Aggregator can be configured with attributes and bean definitions to provide alternative grouping and release strategies, most often the default aggregation strategy suffices.

Interesting Aspects of Splitter Aggregator Operation
  1. Gateway - the inbound gateway, the one on the far left, may or may not have an error handling bean reference defined on it. If it does then that bean will have an opportunity to handle an exceptions thrown within the flow to the right of that gateway. If not, any exception will be thrown straight out of the gateway.
  2. Gateway - an optional default-reply-timeout can be set on each of the gateways, there are significant implications for setting this value, ensure that they're well understood. An expired timeout will result in a null being returned from the gateway. This is the very same condition that can lead to a thread getting parked if an upstream gateway also has no default-reply-timeout set.
  3. Splitter Input Channel - this can be a simple direct channel or a direct channel with a dispatcher defined on it. If the channel has a dispatcher specified the flow downstream of this point will be asynchronous, multi-threaded. This also changes the upstream gateway semantics as it usually means that an otherwise impotent default-reply-timeout becomes active.
  4. Splitter - the splitter must return a single object. The single object returned by the splitter is a collection, a java.util.List. The SI framework will take each member of that list and feed it into the output-channel of the Splitter - as with this example, usually straight into a router. The contract for Splitter List returns is as its use in Java - it may contain zero, one or more elements. If the splitter returns an empty list it's unlikely that the router will have any work to do and so the flow invocation will be complete. However, if the List contains one item, the SI framework will extract that item from the list and push it into the router, if this gets routed successfully, the flow will continue.
  5. Router - the router will simply route messages into one of two gateways in this example.
  6. Gateways - the two gateways that are used between the Splitter and Aggregator are interesting. In this example I have used the generic gateway EIP pattern to represent a message sub-system but not defined it explicitly - we could use an HTTP outbound gateway, another SI flow or any other external system. Of course, for each of those sub-systems, a number of responses is possible. Depending on the protocol and external system, the message request may fail to send, the response fail to arrive, a long running process invoked, a network error or timeout or a general processing exception.
  7. Aggregator - the single aggregator will wait for a number of responses depending on what's been created by the Splitter. In the case where the splitter return list is empty the Aggregator will not get invoked. In the case where the Splitter return list has one entry, the aggregator will be waiting for one gateway response to complete the group. In the case where the Splitter list has n entries the Aggregator will be waiting for n entries to complete the group. Custom correlation strategies, release strategies and message stores can be injected amongst a set of rich configuration aspects.
Interesting Aspects of Simple Splitter Aggregator Operation
The primary deciding factor for establishing whether this type of simple gateway is adequate for requirements is to understand what happens in the event of failure. If any exception occurring in your SI flow results in the flow invocation being abandoned and that suits your requirements, there's no need to read any further. If, however, you need to continue processing following failure in one of the gateways the remainder of this article may be of more interest.

Exceptions, from any source, generated between the splitter and aggregator, will result in an empty or partial group being discarded by the Aggregator. The exception will propagate back to the closest upstream gateway for either handling by a custom bean or re-throwing by the gateway. Note that a custom release strategy on the Aggregator is difficult to use and especially so alongside timeouts but would not help in this case as the exception will propagate back to the leftmost gateway before the aggregator is invoked.

It's also possible to configure exception handlers on the innermost gateways, the exception message could be caught but how do you route messages from a custom exception handler into the aggregator to complete the group, inject the aggregator channel definition into the custom exception handler? This is a poor approach and would involve unpacking an exception message payload, copying the original message headers into a new SI message and then adding the original payload - only four or five lines of code, but dirty it is.

Following exception generation, exception messages (without modification) cannot be routed into an Aggregator to complete the group. The original message, the one that contains the correlation and sequence ids for the group and group position are buried inside the SI messages exception payload.

If processing needs to continue following exception generation, it should be clear that in order to continue processing, the following must take place:
  • the aggregation group needs to be completed,
  • any exceptions must be caught and handled before getting back to the closet upstream gateway,
  • the correlation and sequence identifiers that allow group completion in the aggregator are buried within the exception message payload and will require extraction and setting on the message that's bound for the aggregator


A More Robust Solution - Messaging Gateway Adapter Pattern
Dealing with exceptions and null returns from gateways naturally leads to a design that implements a wrapper around the messaging gateway. This affords a level of control that would otherwise be very difficult to establish.


This adapter technique allows all returns from messaging gateways to be caught and processed as the messaging gateway is injected into the Service Activator and called directly from that. The messaging gateway no longer responds to the aggregator directly, it responds to a custom Java code Spring bean configured in the Service Activator namespace definition.

As expected, processing that does not undergo exception will continue as normal. Those flows that experience exception conditions or unexpected or missing responses from messaging gateways need to process messages in such as way that message groups bound for aggregation can be completed. If the Service Activator were to allow the exception to be propagated outside of it's backing bean, the group would not complete. The same applies not just for exceptions but any return object that does not carry the prerequisite group correlation id and sequence headers - this is where the adaptation is applied.

Exception messages or null responses from messaging gateways are caught and handled as shown in the following example code:

import com.l8mdv.sample.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.integration.Message;
import org.springframework.integration.MessageHeaders;
import org.springframework.integration.support.MessageBuilder;
import org.springframework.util.Assert;

public class AvsServiceImpl implements AvsService {

    private static final Logger logger
            = LoggerFactory.getLogger(AvsServiceImpl.class);
    
    public static final String MISSING_MANDATORY_ARG
            = "Mandatory argument is missing.";
    
    private AvsGateway avsGateway;

    public AvsServiceImpl(final AvsGateway avsGateway) {
        this.avsGateway = avsGateway;
    }

    public Message<AvsResponse> service(Message<AvsRequest> message) {

        Assert.notNull(message, MISSING_MANDATORY_ARG);
        Assert.notNull(message.getPayload(), MISSING_MANDATORY_ARG);

        MessageHeaders requestMessageHeaders = message.getHeaders();
        Message<AvsResponse> responseMessage = null;
        try {
            logger.debug("Entering AVS Gateway");
            responseMessage = avsGateway.send(message);
            if (responseMessage == null)
                responseMessage = buildNewResponse(requestMessageHeaders,
                        AvsResponseType.NULL_RESULT);
            logger.debug("Exited AVS Gateway");
            
            return responseMessage;
        } 
        catch (Exception e) {
            return buildNewResponse(responseMessage, requestMessageHeaders,
                    AvsResponseType.EXCEPTION_RESULT, e);
        }
    }

    private Message<AvsResponse> 
                     buildNewResponse(MessageHeaders requestMessageHeaders,
                     AvsResponseType avsResponseType) {

        Assert.notNull(requestMessageHeaders, MISSING_MANDATORY_ARG);
        Assert.notNull(avsResponseType, MISSING_MANDATORY_ARG);

        AvsResponse avsResponse = new AvsResponse();
        avsResponse.setError(avsResponseType);

        return MessageBuilder.withPayload(avsResponse)
                .copyHeadersIfAbsent(requestMessageHeaders).build();
    }

    private Message<AvsResponse> 
                     buildNewResponse(Message<AvsResponse> responseMessage,
                     MessageHeaders requestMessageHeaders,
                     AvsResponseType avsResponseType,
                     Exception e) {

        Assert.notNull(responseMessage, MISSING_MANDATORY_ARG);
        Assert.notNull(responseMessage.getPayload(), MISSING_MANDATORY_ARG);
        Assert.notNull(requestMessageHeaders, MISSING_MANDATORY_ARG);
        Assert.notNull(avsResponseType, MISSING_MANDATORY_ARG);
        Assert.notNull(e, MISSING_MANDATORY_ARG);

        AvsResponse avsResponse = new AvsResponse();
        avsResponse.setError(avsResponseType,
                responseMessage.getPayload(), e);

        return MessageBuilder.withPayload(avsResponse)
                .copyHeadersIfAbsent(requestMessageHeaders).build();
    }
}


Notice the last line of the catch clause of the exception handling block. This line of code copies the correlation and sequence headers into the response message, this is mandatory if the aggregation group is going to be allowed to complete and will always be necessary following an exception as shown here.

Consequences of using this technique
There's no doubt that introducing a Messaging Gateway Adapter into SI config makes the configuration more complex to read and follow. The key factor here is that there is no longer a linear progression through the configuration file. This because the Service Activator must forward reference a Gateway or a Gateway defined before it's adapting Service Activator - in both cases the result is the same.

Resources
Note:- The design for the software that drove creation of this meta-pattern was based on a requirement that a number of external risk assessment services would be accessed by a single, central Risk Assessment Service. In order to satisfy clients of the service, invocation had to take place in parallel and continue despite failure in any one of those external services. This requirement lead to the design of the Messaging Gateway Adapter Pattern for the project.

  1. Spring Integration Reference Manual
  2. The solution approach for this problem was discussed directly with Mark Fisher (SpringSource) in the context of building Risk Assessment flows for a large US financial institution. Although the configuration and code is protected by NDA and copyright, it's acceptable to express the design intention and similar code in this article.