BPEL Correlations...as easy as it sounds?

In my previous post I discussed the basics of BPEL correlations. In this post, I want to present the challenge I faced trying to apply correlations to my business process, which unfortunately for me does not align with any vendor tutorials I have seen. Imagine that :)

The portion of my business process which requires correlation involves calling out to a notification web service and supplying an unbounded set of people to which I want to notify. This notification web service then responds with a single notification identifier (notification ID) representing the notification sent. At some future point in time, the people notified will respond (or the notification system will let me know there was no response). These responses can be received in any order at any time, and are received via a web service call into my ESB. The payload of this web service call includes the notification ID and the response of the person. After receiving the response, my business process updates the person information with the response and continues on. This is exactly what correlations are made for. Perfect!

Challenge 1: If I define correlation initialization on the notification web service invocation activity, this will persist 1 instance of the business process. But I have multiple responses coming back. After the first response is received, I have no business process instances waiting.

Solution 1a: I'll just loop around the web service invocation for each person in my set of people and make individual calls out to the notification web service. This will create multiple notification IDs and therefore persist multiple business process instances. So as responses come back, each one will correspond to its own instance. The business process would look something like this:

Problems: First, the business process will always run to completion as the response web service implementation gets invoked after the while loop (i.e. there is no receive task for the process to wait on). So, if we replace the web service implementation with a JMS receive task we can get over that hurdle. All we have to do is create another business process that implements a web service, and within the implementation it uses a JMS send activity to send the notification response to the corresponding JMS queue that this process is monitoring. The new business processes look like this:



Now the problem is that we really only have one business process instance being created. Although it seems like multiple business process instances are being persisted, each one really maps to the same instance. So, once the 1st response is received, the persisted instance is gone and other responses are left hanging.

Solution 1b: So, we remove the loop here and will need to create another business process (we are now up to 3 business processes) that sends messages with a single person to the inbound JMS queue. This will cause a new business process instance to be created for each person, so each response can now be correlated to one of these instances.

Challenge 2: How do we define the correlation set so that it actually works? Our correlation aliases were initially defined as the notification ID returning from the notification web service invocation and the JMS receive message text. Since the message text value (an XML message) does not match the notification ID (an int data type), the correlation is not working.

Solution: This was a difficult one. I went through many iterations with problems....err challenges encountered with each one. Here they are:

(1) Add an unmarshal task so that we can correlate on the resulting notification ID value. This will require using correlation on the Unmarshal activity. Since the JMS receive activity, which occurs prior to the unmarshal, causes the BPEL engine to retrieve an instance from persistence, and no correlation set is defined for this task, the engine grabs any instance. So essentially there is no real correlation going on.

(2) Try using the JMS message header correlation ID. This involved mapping the response notification ID to the correlation ID of the JMS message header prior to placing this on the response queue (in our second business process). The issue here was the correlation ID message property is defined as a string, while the notification ID is an int. So CAPS will not let you define an appropriate correlation key/correlation set that will work. The data types of each alias need to be the same, which makes sense. Also, since my business process has 2 JMS receive tasks, the BPEL engine gets confused when trying to create the instance identifier using the correlation ID of the JMS message header.

(3) Call for help. I discussed this problem with someone I know from Sun and he was able to provide a pre-release of some documentation that provided a good amount of insight into how to deal with correlations. The result of this new found knowledge was the creation of one more business process (for a total of 4) that actually had 2 JMS receive tasks. I could then create a correlation set based on the JMS message header correlationID and use correlations on each of the JMS receive tasks. The following images show 3 of the 4 business processes (the one not shown is simply a loop that places messages on the inbound JMS queue for the notification web service invocation process).

Notification Web Service Invocation

Notification Response Web Service Implementation

Combined Business Process Using Correlation

First, a message is placed on the inbound queue of the notification web service invocation process. This calls the external web service and uses the response to place a message on the correlation.queue JMS queue. Also, the notification ID that is the response from the web service invocation is set to the JMS message header correlationID property.

Recall that the Combined Business Process has a correlation set defined to be the correlationID message header property. The 2 JMS receive activities are set to use correlations. The first receive task initializes it and the second uses it.

So, once the first message is received, an instance of the Combine Business Process is created and then persisted (since it is now waiting on the second JMS receive). Now, at some future point in time, a person responds to the notification (or the service responds with "no response"). The response includes the notification ID and the response. The Notification Response Web Service Implementation maps the notification ID to the correlationID message property and places the response on the response.queue.

The second JMS receive task on the Combined Business Process is using this queue. This causes the BPEL engine to retrieve the business process instance that corresponds to the notification ID contained within the persons response. The process instance is retrieved (along with its state), and the business process continues to completion. This implementation of correlation finally worked.

From initial design to final implementation took longer (and was more painful) than it sounds. As a result I got a great understanding of how correlation works within CAPS and I am confident that this knowledge will transfer to other products as well.

BPEL Correlations introduced using Sun CAPS

It has been a while since my last post, so let me start with a quick update on my real world SOA journey. As I had mentioned previously, I was able to get some Sun Java CAPS training and was temporarily putting Cape Clear on the back burner. Well, I have been working with CAPS now for about a month and I find it very interesting. Fortunately, some of the issues I ran into with Cape Clear are no different on CAPS. In reality, they are more SOA issues than product issues. For example, invoking a web service that uses SOAP encoding (as discussed here). That being said, I could reuse much of the experience I gained with Cape Clear during my CAPS implementation.

Enough history (you can read my previous posts to catch up), this post is supposed to be about correlations. If you are familiar with JMS, correlation is probably nothing new. Basically you are associating one message with another message via some identifier. In JMS you typically use the correlationID message header attribute and create the appropriate logic in your application. CAPS (and Cape Clear, as well as others) provides a correlation mechanism within a Business Process (using BPEL). BPEL correlation allows a process to be persisted while waiting for a correlating response. This is usually required when dealing with asynchronous exchanges.

A typical example is a purchase order and corresponding invoice (related via the PO number). A business process may receive a PO, do some processing, and then wait for the associated invoice. Once the invoice is received, the business process does some more processing using the PO and Invoice data, and eventually completes. While waiting for the Invoice, the process can be persisted either to memory or some more permanent store to survive server reboots, etc. The magic here is retrieving the correct business process instance once the corresponding invoice is received. This is accomplished via the correlation set. A correlation set is defined different ways for different products, but it essentially represents some combination of message parts that are available to activities within the business process. So in this example, the PO number in the purchase order message and the PO number in the invoice message would comprise this correlation set. These message parts are typically called aliases.

Within CAPS, you define what is called a correlation key, which is made up of the aliases described above. Then you define a correlation set based off of the correlation key. I'm not really sure why the extra step of creating a correlation key, but that is the mechanics involved. As shown below, this correlation key/correlation set is defined on the properties sheet of the business process.


In this example, the correlation key is made up of a single alias, the message correlation ID. In reality, the alias can be made up of one or more message parts that exist in the business process.


Once defined, the correlation set needs to be applied to activities within the business process. One activity should initialize the correlation set, while another process should use the correlation set.


The difference being the value for the "Initialize Set" field. Also, depending on the type of activity for which you are using correlation sets, you may get the following dialog:


On this dialog, you need to select where the application of the correlation set should take place, at the input of this activity, the output, or both. I'm not really sure what it means to use "both", but I hope to experiment with this sometime in the future.

In order to have the business process use the correlations, there must be a receive activity somewhere in the midst of the business where we are waiting for a message. So in the typical example above, we may be waiting for the invoice to arrive on a message queue somewhere after the PO has arrived in our business process.

So here is how the process works, using a simple business process shown below:

A PO comes in via JMS (PO number 123). The business process is instantiated. This receive activity is configured to initialize a correlation set as shown previously. Execution of the various activities and business rules will continue until the next receive activity is encountered (JMS.receive Invoice). At this point the business process is persisted using the identifier specified in the correlation set. In our example this would be the correlation message header property of the inbound JMS message (123). At some future time, an invoice shows up on the appropriate message queue for PO 123. This invoice has the correlation Id of the JMS message header set to the purchase order, so the BPEL engine looks for a business process instance with a correlation identifier which includes the name of the correlation set (corrSetCorrelationID) and a value of 123. If this is found, the business process instance is retrieved (with its state intact) and the process continues to completion.

This is a real simple example which uses the JMS message header as the correlation set alias. This requires the applications creating the messages to populate this field prior to sending the message. A more elegant solution is to use actual message values to correlate on, but the goal of this post was to introduce the basics of correlation.

You can find tutorials that show how easy this is to do on many of the SOA product suites. Tutorials are great, but they hardly translate to real world problems. Here is where my real world "challenge" came into play. The CAPS tutorials that I found really didn't explain the entire correlation configuration and mechanics. In my next post, I'll discuss the challenges I encountered applying correlation to my business process (described here) and the final implementation. Although this will present correlation relative to CAPS, I believe this will translate well into other product implementations of correlation.

Changing Course....slightly

If you've read my previous posts, you know a couple of colleagues and I are in the process of developing a business process using various SOA product suites. The intent is to go beyond the typical hand held tutorials that vendors provide and understand what it really takes to develop composite applications in an existing enterprise.

To that end, I have already learned a good bit. For instance, WSDLs and Schemas are not always provided for you, sometimes you actually have to create them.... and it is not always drag and drop and whammo, you created a business process... and just because you have a web service doesn't mean it drops easily into a BPEL process. That's just a small taste of what I have discovered so far, and I know there is plenty more to come.

As I have stated earlier, I am using the Cape Clear ESB to implement our business process. Well, over the past couple of weeks, I haven't had much time to progress from Step 2. That is mainly due to some Sun Java CAPS training we received (and being able to attend the SD West conference). As a result of this training, I am going to shift my implementation from Cape Clear to Java CAPS for the time being. Once I complete the implementation in Java CAPS, I do plan to return to the Cape Clear implementation.....and I will still be relaying the things I encounter along the way with both products....in case you were worried :)

Stay tuned....

Step 2: Accessing the CRM System

My excursion into the "real" world of SOA has already been a learning experience. As mentioned in my original post in this series (The Beginning), the intent is to move away from the vendor propaganda and canned tutorials and implement a business process from the ground up using different SOA products. This way I can understand first hand what it takes to service enable existing applications, the implications of various system interfaces that can be encountered in an Enterprise, and the challenges to deal with when trying to build a composite application with pre-existing systems. In order to do this, several of my colleagues and I have created a Virtual Enterprise (described here).

So far, we have encountered the need to introduce a new service into our Virtual Enterprise due to the granularity of some of our Enterprise Applications (specifically the IVR and Email systems). We also began to realize that SOA products don't always support some of the deployed technologies within an Enterprise. As described in my previous post (Receive a JMS Message), I had to workaround the fact that Cape Clear did not support our Virtual Enterprise messaging standard of ActiveMQ. Also, Cape Clear expects all messages to be SOAP messages our Alert system doesn't communicate via SOAP.

This is all good stuff. I am starting to accomplish exactly what I set out to....experience SOA in the real world. Now on to the next challenge....invoking a web service (RPC encoded style) to retrieve a set of participants in my event.

This task seems pretty straightforward. Call a web service, get a set of people back and use this set to send out some notifications. Cape Clear likes everything to be a web service, so what could be easier. Just define the partner in your BPEL process using the WSDL from the CRM web service, drag and drop....and now you can use the CRM system in my process.

Cape Clear has a very easy way to define a partner within a BPEL process. Simply walk through a wizard which requests a WSDL (either file, URL, or Cape Clear Server based), asks you to define the partner type (in this case I will be invoking this partner), and like magic it creates the necessary variables to be used in your BPEL based on the schemas defined in the WSDL. Based on the operations exposed within the WSDL, the appropriate actions also become available to your BPEL process. From there it's good old drag and drop the action onto the BPEL process, use the variables created during the partner definition......and viola, you just included this external system in your business process.

Here is where I encountered my next challenge. Recall that this web service is using RPC encoded style SOAP and returns an array of people. The variable that Cape Clear (and I learned later any current BPEL engine) creates is a simple variable type. Well, I am trying to work with a complex array of people. Something isn't right here. After much research, struggles, research, struggles and a submission to the Cape Clear forum I decided I needed a work around.

While waiting to hear back from the Cape Clear folks, I decided to wrap the CRM systems RPC style web service with a document/literal style web service. Then I could use the document/literal web service in my BPEL process. Sounds like a good bit of overhead, but that is for a later topic.

So, I attempted to create a simple web service that calls another web service. This is great....Cape Clear allows me to create a project for exactly this purpose. So, first I create a schema that will represent the return type of my new wrapper web service. This schema essentially defines an element containing an unbounded number of people elements.

Now, I fire up the Cape Clear wizard to create a web service that invokes another web service. I point at my RPC style web service and.....low and behold....the wizard cannot handle the WSDL. Great....now what. I decide to try create a web service client in a standard Eclipse IDE I have installed (BTW....the Cape Clear IDE is Eclipse with some plugins) and everything worked fine. So now I submit my second post to the Cape Clear 7.0 Beta forum (I was still waiting on a response to the earlier post).

In the mean time, I create a WSDL based on the schema I created earlier for my new wrapper web service. If you never created a WSDL from scratch before....it's a good learning experience. Once I had the WSDL defined, I created a Java Web service to implement the WSDL. I then had to copy in the classes from my external Eclipse project that would allow me to call the RPC style web service into my new Cape Clear project. I now had one project that represented a web service that would call another web service. I deployed this to the Cape Clear Server and was now able make the call out to the CRM web service through my new wrapper web service.

After this was completed and functional, I got my reply from Cape Clear. They informed me that "RPC/encoded style of WSDL is not supported by BPEL. The problem is that according to the rules of SOAP encoding, the messages does not conform to the structure defined by the XML Schema". Their recommendation was to do exactly what I ended up doing...create a web service that wraps the RPC/encoded style web service (although they recommended RPC/literal). They also indicated that "The local call to the new web service will be very efficient, so it shouldn't cause any performance problems". Again, that is for a later topic.

Good thing these tools make it so easy to build a business process :)

First Step: Receive a JMS Message

This blog continues the story from my previous post (The Virtual Enterprise). In this installment I will describe what I had to go through in order for my business process to receive an XML message from a JMS queue (using ActiveMQ as the provider). Recall that I am implementing my business process using Cape Clear.

As a point of clarification, I have yet to receive any formal training in Cape Clear's product and am attempting to implement my business process by referring to documentation and tutorials available on Cape Clear's developer web site (http://developer.capeclear.com/). To Cape Clear's credit, they do have a good amount of documentation available to the public. Just to make it a little tougher, I chose to use Version 7 of their product, which (at the time of my effort) is in beta (some may call me a glutton for punishment).

In Cape Clear's world, everything is a web service communicating using SOAP. Also, as for supported JMS providers, ActiveMQ is not among them. They do support (out of the box) IBM, BEA, and JBoss. As an additional piece of information, my Cape Clear installation is deployed on JBoss (other options include BEA, IBM, Tomcat).

So, here is problem number 1 (or is that challenge number 1). How do I get a JMS XML message (non-SOAP) sitting on an ActiveMQ message queue to kick off a business process (BPEL) deployed to Cape Clear?

Just to be specific, there are actually 2 problems here. First, using ActiveMQ as the JMS provider, and second, handling a non-SOAP message. Looking back (after a little teeth grinding and mental cursing), the solution was "easy".

First, I configured the JBoss server to access an ActiveMQ broker running on another machine. This part was fairly straightforward using the ActiveMQ resource adapter. I followed directions that I found here (modified for my specific versions of the products and to use a remote broker in place of the embedded broker). Once I got this working, the next step was to configure the Cape Clear server to route messages on this ActiveMQ queue to my new BPEL process. This was accomplished by configuring a "transport" on the Cape Clear server. I essentially created a JMS JBossMQ transport, but in place of the standard connection factories and queue names, I used the JNDI names for the ActiveMQ connection factories and queue names that I configured with the ActiveMQ resource adapter. And then, like magic, I was able to receive messages from an ActiveMQ queue.

Now, how do I get the BPEL process to handle a non-SOAP message. Fortunately, as I stated earlier, Cape Clear has a good amount of documentation and tutorials on their site. I was able to find an example of handling non-SOAP XML messages in an Orchestration Web Service (i.e. a BPEL process deployed as a web service). This tutorial was created for version 6 of their product, but works as well on version 7. Essentially what happens is that you define an XSLT transformation to wrap a non-SOAP message in the appropriate SOAP header and body tags to make it a SOAP message. This XSLT is applied prior to invoking the BPEL process by configuring an interceptor for the BPEL process deployed within the Cape Clear server. How much this additional step potentially impacts performance is TBD and will be addressed sometime later in my road to discovery :). I wonder why Cape Clear just doesn't make that a configuration option for a BPEL web service and then perform some sort of optimized operation to receive the non-SOAP message, or skip the SOAP message parsing entirely and just pass the message directly to the BPEL?

Anyway.....there you have it. I can now accept a non-SOAP message from an ActiveMQ queue and invoke my BPEL process deployed to Cape Clear. So simple a monkey could do it ;) ......got any bananas?

The Virtual Enterprise

As discussed in my previous blog (The Beginning), my associates and I are going to implement a single business process with an Enterprise. This Enterprise consists of 5 systems that have defined system interfaces. We call this our Virtual Enterprise. It is virtual in the sense that it does not exist within any Corporate Enterprise, but they are real functioning applications. These applications were designed and built by my associates and I, but done so in an independent manner, as if they grew up in a real Corporate environment. Keep in mind that they are simplified versions of what one might find in a real Enterprise, but the intent is really to create a business process based off of pre-defined system interfaces. In order to provide flexibility into our Virtual Enterprise, the interfaces that each system supports can be changed simply by modifying some configuration parameters. This allows us to modify some of the restrictions of the Enterprise for determining capabilities of various SOA product suites.

Let me introduce the systems:

(1) The Alert System. This system will record information about an event. It is web based and can release the event information via JMS to a defined ActiveMQ queue. The structure of the event is specified in an XML schema.

(2) The CRM System. Here we can define employees, customers, etc. and group them together in some logical fashion. This is our people repository. This system has an RPC encoded style SOAP interface for retrieving groups of people, individuals, and the set of groups defined within it.

(3) The Scheduler. This systems provides the ability to schedule a get together (conference call, meeting, etc.). Provided some information such as number of participants, dates and times, it will provide a logical meeting place (conference call code, etc.). This application is EJB based and can be accessed via RMI.

(4/5) The Notification System. This system is responsible for notifying participants in the get together. The reason I label this 4/5 is that it is really wrapping 2 separate systems (an IVR system and an Email system). When we defined this Virtual Enterprise, we decided that participants in the get together could be notified either by email or voice. As we began to review the Enterprise and look at the Business process we were implementing, we realized that these systems were a bit fine grained for an SOA. We therefore decided to wrap these in what we call a Notification Service. This notification service will provide a more coarse grained interface to the underlying systems. This was lesson #1 for our Real World SOA. The underlying IVR system has a document literal SOAP interface and the Email system has an SMTP interface (shocking). We are allowing the Notification system to be defined using the strengths of any given SOA product we are evaluating. This is really intended to test some of the product suite capabilities for building a service as opposed to the true service enabling of existing applications.

Now that you know the systems involved, the use case is fairly straight forward (ha ha). The alert system will create an event that people need to get together and discuss. Therefore our business process has to determine what people need to get together, where they are getting together, how to contact them, manage their responses over time, and provide status back to the alert system as the process progresses. If a person is not available then contact their alternate. Sounds simple enough, but when you dig deeper there is a good bit going on here.

Subsequent blogs will provide more information as to the issues encountered along the way, potential or actual workarounds, struggles with the products, etc.

Stay tuned........

The Beginning

The intent of this blog is follow the trials and tribulations that I and several my associates at Chariot Solutions (Rod Biresch and Tom Purcell) come across as we implement an SOA solution. We all have experience in the realm of integration, and as the buzz around SOA has increased, so has our interest and understanding of what SOA is (and can be) for an organization.

Granted, SOA doesn't have one definition, but we believe at its core it consists of loosely coupled business services that become the building blocks for higher level business processes. How Business Services are defined, and how an organization can combine them into composite applications is where the challenge exists. If you look at any SOA product vendors web site, according to them all you have to do is just use their products, drag and drop some icons, click deploy, and wow....a business process has been implemented. If you download any vendors product suite and go through their tutorials, their story seems to hold water. Of course, how many organizations need a Hello World SOA application? Seriously, the tutorials are nice, but they barely scratch the surface of the challenges that you can encounter when trying to migrate towards an SOA.

So, being pragmatists, we don't believe that it is so "easy" to create a Service Oriented Architecture within in an existing enterprise. We do know the benefits of moving towards an SOA, but it is hard to find the facts around the issues encountered along the way. So what we have decided to do is take an existing enterprise, with systems that cannot be modified in any significant way, and combine them to implement a business process, documenting our issues along the way. The assumption here is that the business has already committed to moving to an SOA and a top down approach to defining business processes has already occurred. We are just going to implement one of these business processes (and service enable the applications to support it along the way).

I have chosen to use Cape Clear's ESB product to implement our business process. In subsequent posts, I will not only discuss the challenges encountered using this product, but also in migrating our stove piped enterprise towards an SOA.

Steven Smith
Enterprise Architect
Chariot Solutions