Custom XSLT Vs In-Built Functoids in BizTalk Mapper - Best Practice

Hi There,

After 3 years of extensive experience developing integration applications using BizTalk, I have now come to the point realizing the best practices which should be followed while developing integration solutions. I say this because if you are not following or giving attention to the practices being followed then for sure you are making the code as non-readable or which cannot be easily understood by a new developer - in other words, you are pushing the whole solution towards an utter mess.

Integration solutions are by default complex in nature due to the difference in environment and operations being followed in different systems. To make two system talk to each other, it requires tedious amount of work of transforming/translating the data which the other system can understand.

In these three years of my 'Integration Journey', I have had the opportunity to solve various complex integration problems under the guidance of very smart and experienced professionals - fortunately :) . From day one of starting my career as a BizTalk developer, I have always been taught to follow best practices while developing applications be it a POC,  a new project or an enhancement to an already built solution. 

In this blog I'm going to share my experience of when to go with XSLT or functoids or both.

Custom XSLT and Functoids - A Tale Of Two Sides Of The Same Coin

They both are two sides of the same coin because they both are used to get the same output, it is up to to us which side we should get inclined to. 

To Go With Functoids - My Take :- 

Functoids are inbuilt tools in biztalk mapper which have most of the functions embedded in them just like a xslt function and are used in a graphical manner using links from source file and providing output to the target file.

1. Functoids are easy to use,understand and require no coding,  a new biztalk developer is initially habituated to use functoids.

2. Fuctoids should be used where the mapping scenario is simple and doesn't require the use of too many functoids.

3. It is faster to analyse with functoids of what the map is actually doing, while analyzing an XSLT code is lot more difficult as one has to parse through complete XSLT to understand what it is trying to do.

4.You can get to understand the functionality of a functoid by name while with XSLT function you will have to keep some reference in order to know the name of the funcion to use , for example if you want to concatenate two strings , you will get that by string concatenate functoid while with XSLT you will have to search for the function name which concatenates two strings .. i.e string-concate().

5. In other words , use of functoids comes into picture when you are not proficient with XSLT or when the mapping is not very complex.

To Go With XSLT - My Take :- 

1. XSLT should be used where throughput and performance are the concerns, that again depends on the complexity of the map. A custom XSLT is indeed the right choice for getting high performance.

2. It is better to go with XSLT when there are lot of links or connections required for getting a result because having too many connections between the elements and functoids makes the map complex and hard to read.

3. For case where complex looping is required where we have to loop the target record node with no element to loop with in source file then we should go with XSLT instead of table looping functoid. I found table looping functoid to be complex when we had to map data to more than 5-6 elements because it required maintaining lots of columns/rows and lots of links in order to fetch the data.   

THE FINAL CALL -

Microsoft BizTalk team created functoids to make transformation simple and I bet you will not find such an enhanced and simple to use transformation tool in any other integration platforms, so my take would be to use functoids where there is not much complex set of rules involved in transforming the data but I prefer using both of them i.e. use of functoids on nodes where logic is simple and inline XSLT/XSLT Template in scripting functoid on nodes where mapping the data is complex so ultimately I end up using functoids but some of you may agree that it is actually using both.  :) 

Whatever you use should be clean and readable , this is what I have learnt so far.

These were just my thoughts on best practices to go with while choosing between functoids and XSLT.

HAPPY HOOKING STUFFS !!

---------------------------

Saurav Suman

Senior Analyst @ Accenture

Resolving SQL notification error in BizTalk

Hi there,

I was recently working with notification feature in BizTalk and was getting an error from the BTS engine whenever I switched on the receive location.

"The messaging engine failed to add a receive location...."

After going through several articles on MSDN and other forums I realised that the possible cause to the above problem could be due to the following :-

1. SQL broker is not enabled
2. Notification statement is not correctly written.

I checked my database to confirm that the SQL broker is enabled which I found it to be true in my case.

After confirming the above I went through my receive location binding properties to check the notification statement.

Nothing looked wrong at the above setting unless I went through this MSDN site and came across a NOTE which stated that - "You must specifically specify the column names in the statement as shown in this SELECT statement. Also, you must always specify the table name along with the schema name. For example, dbo.Employee."

I went back to the receive location binding property and edited my statement as 'Select ID from dbo.Employee.' . Now when I switched on the receive location , the error was gone and I was now getting the notification message whenever a change was applied on the table.

The lesson learnt from the above issue was that we should always specify the table name along with schema name in a statement.

Regards,
Saurav
BizTalk Developer @ JKT





   

Sequential And Parallel Convoys In BizTalk

Hi All,

One of the powerful feature in BizTalk is correlating a message to the correct instance of an orchestration. This is done through two ways : Sequential and Parallel convoy correlation.

Sequential convoy means correlating the incoming messages from different or same source to an instance of orchestration serially i.e. in a sequential manner but one major difference from parallel is that the process will finish execution after an interval regardless of arrival/non arrival of other messages in a convoy.

Below sample sequential convoy process is an example of how sequential convoy works.








In the above example I have taken two schemas - customer order and customer bill, I will explain the scenario for both sequential and parallel convoys using the order-bill relationship. The above scenario is for sequential convoy where a customer places an order and then pays for the order he/she has placed. Here the relation between order and bill is an unique number which is an element present in both the schemas and both the messages will be correlated based on value present in that field.

Here, first the order is placed and is subscribed by the orchestration which picks the order xml and initializes the correlation , it is then the orchestration which waits for 10 seconds for bill to be received. The bill xml if dropped within time interval of 10 seconds is picked by the orch and then rest of the process happens and if it is not dropped within 10 seconds after the order is dropped then the instance is execution ends. Sequential convoy gives birth to 'zombies' which means lost message, in the above scenario if bill is dropped after an interval of 10 seconds when order is dropped then it will be suspended with no subscriber found error message because the instance to which the bill message was assigned has ended. Although there is no proper way to avoid this problem but it can be reduced to some extent by increasing the timespan in delay shape. So from the above example we see that sequential convoy follows "OR" logic and can be implemented in scenarios where dependency on other message is a primary constraint.Here both bills and orders consume same instance of orchestration in which the correlation set is initialized by order receive shape and bill receive shape follows the correlation set initialized by order shape.So in this scenario order receive shape will be ACTIVE.

Correlation set initialized by ORDER receive shape.

Correlation set initialized by BILL receive shape.

Below process design is an example of parallel correlation which is following the same order-bill relationship scenario.



In the above example the scenario is same but it follows "AND" logic where both the receive shape are ACTIVE and both of them have same 'INITIALIZING' correlation set which means that the process will begin execution unless both order and bill messages have been received with same unique number.One important thing is to note that though the sequential convoy is using the correlation on unique number but we cannot use the same correlation set for parallel convoy, we will have to create a new correlation set.

Hope you guys enjoyed reading the article.

Regards,
Saurav
Software Engineer @ Agility

Compensation Through Orchestration In BizTalk

Hi there,

Have you ever thought of rollback of a process through orchestration in BizTalk? Well, compensation is the feature which does that. Infact,compensation is the most underrated and underused feature in biztalk. A few days ago I was working on a R&D for a project to find out the features provided by compensation and to my surprise it stood to my expectations. But in this post, it would be difficult for me to explain the concept of compensation but rather I would be explaining the process which I incoporated in my sample project. I would advice you all to go through this fantastic blog by
Charles Young which clears the concept of compensation with transactions.Also, a very important point to keep in mind is that compensation works with those transaction which commits.

My Process

Explanation Of The Process:

The above process is a simple example of compensation on a customer order. In this sample i took one schema with two root nodes - CustomerDetails and OrderDetails with a one promoted common field (OrderID) for correlation.

In my process i've used L-R transaction for each scope (Scope_Customers and Scope_Orders).Scope_Main is also L-R and kept my whole orchestration as L-R.

Each scope has a compensation block with its respective logic for rollback - i.e. delete operation.

The whole process is a simple insertion process in which first the customer detail gets inserted followed by the order details. And the logic of the process is based on the fact that if any of the two individual L-R transaction scope fails then rollback happens i.e. compensation block gets fired if an exception is caught by the main L-R scope. 

Scenario:

The process is fired and insertion of customer details are done. Now insertion of order details starts, but what if  due to some issue an exception is caught and insertion of order detail fails? In this case compensation acts as a handy feature which roll backs the insertion done in customer table because it got committed. When the exception is caught by any of the exception handlers which resides in the main L-R scope then the respective comensate shape will get fired and will refer to the defined scope (In this case, Compensate_Customer will get fired  on Transaction_Customer).

Also, I've kept a tracking variable- Status (of boolean type) for the decide shape which fires an Email to the developer that an exception has been caught and rollback got initiated.Status = 0 is for no exception and should be kept under expression shape of CustomerInsert and OrderInsert.While Status = 1 should be kept under the expression shape of  each of the exception handlers.

Expression Shape Contents In Customer_Insert Scope

Expression Shape Contents In one of the exception handlers

Also, keep the synchronized property of each of the individual transaction scope(Customer and Orders) to True if you want exception to be caught and keep the RetryCount and RetryInterval on each of the send ports to 0.

Compensation can be used with atomic transactions also but i didn't use atomic transaction in my process as i was consuming services from a request-response port (WCF-CUSTOM) and atomic transaction doesn't allows that.

Hope this post comes useful to those who were confused with compensation like me :)

Thank You.

Regards,

Saurav Suman

Software Engineer @ Agility

MQ Correlation Through Orchestration In Biztalk(Without Using MQSeries Adapter)

Hi there ,

In this post I'm going to explain in detail about how to push and retrieve message from IBM websphere MQ through orchestration in biztalk. To begin with, ensure that you have IBM websphere mq installed and configured in your system properly.

The Process:

The below screenshot depicts the overall process

1. Create any schema you like. I have created the following schema as depicted in the figure below.

2 . Add orchestration to your project. Create one receive shape of message type = your schema and bind it to logical receive port.

3. Drop one scope shape and make transaction type property of that scope to atomic.Inside the scope, place one assignments shape.

4. Add amqmdnet.dll file as a reference to your project. The file can be found at default installation directory : C:\Program Files (x86)\IBM\WebSphere MQ\bin

5. Create following variables under the above created scope.

a. quemgr - type: IBM.WMQ.MQQueueManager 

b. ReqMsg - type: IBM.WMQ.MQMessage

c. RespMsg - type: IBM.WMQ.MQMessage

d. rqstqueue - type: IBM.WMQ.MQQueue   

e. respqueue - type: IBM.WMQ.MQQueue

f. gmo-            type: IBM.WMQ.MQGetMessageOptions

6. I have written the following lines of code to push and retrieve the msg with same           correlationID from MQ.

7. I'll be explaining this code block by block from request message creation to response message    creation . Our primary aim is to push and retrieve the message from queue based on correlationID.

Create one variable of type System.xml.xmldocument. This variable will hold the values arriving from receive shape.

8. Now we will be generating guid for our request message.correlationID . I tried creating the guid in the expression shape but since XLang doesn't support byte [] array , i was compelled to make one class library for creating guid and using it to assign it to request message correlationID.

9. Add one C# Class library project in the same solution. Name it : GUIDCLASSLib and the class as ClassGuid.

10. Add the following lines of code as shown below:

11. Build the project and add the class library reference to your project. Also register the dll file in    gac through gacutil /i [dll name] command.

12. Now switch back to message assignment shape and assign the values retrieved from receive message shape to the xml variable : Variable_XML= Message_Rcv.

13. Copy the following code with values that of your environment.

IBM.WMQ.MQEnvironment.Hostname = "your host name";
IBM.WMQ.MQEnvironment.Port = your port [int]; //ex: 7714
IBM.WMQ.MQEnvironment.Channel = "your channel name"; //ex: APP.ALL

quemgr = new IBM.WMQ.MQQueueManager("Queue Manager Name"); //ex:SDCQMGR.T1

rqstqueue = quemgr.AccessQueue("Your_ request queue name",IBM.WMQ.MQC.MQOO_OUTPUT + IBM.WMQ.MQC.MQOO_FAIL_IF_QUIESCING);

respqueue = quemgr.AccessQueue("Your_ response queue naem",IBM.WMQ.MQC.MQOO_INPUT_AS_Q_DEF + IBM.WMQ.MQC.MQOO_FAIL_IF_QUIESCING);


The MQEnvironment sets the environment variables to the given values. MQQueueManager manages all the queues and channels which reside under it . rqstqueue variable is the instance of IBM.WMQ.MQQueue type which will act as a Request queue. Similarly, respqueue is of type MQQueue and will act as response queue.

14.Now we will be creating instance of MQMessage, One request message and response message respectively. First we will be instantiating ReqMsg variable.

ReqMsg = new IBM.WMQ.MQMessage();

15. Copy the following code for setting the property of request message. This request message carries the content of xml file which we are receiving at receive shape.

ReqMsg.WriteString(Variable_XML.OuterXml); \\the original message.
ReqMsg.Format = IBM.WMQ.MQC.MQFMT_STRING;
ReqMsg.MessageType =IBM.WMQ.MQC.MQMT_REQUEST;
ReqMsg.Report = IBM.WMQ.MQC.MQMO_MATCH_CORREL_ID;
ReqMsg.ReplyToQueueName = "FOCiS.TO.EIL";
ReqMsg.ReplyToQueueManagerName = "SDCQMGR.T1";
Variable_obj = new GUIDCLASSLib.ClassGuid(); \\ create variable Variable_obj of type GUIDCLASSLib.ClassGuid()..

ReqMsg.CorrelationId = Variable_obj.returnguid(); \\assigning the guid to correlationId

16. Now put ReqMsg into rqstqueue :

              RequestQueue.Put(ReqMsg);

17. Log the request message.

        System.Diagnostics.EventLog.WriteEntry("RequestQueueMsg",ReqMsg.ToString());
18. Similarly instantiate the response message variable RespMsg .

RespMsg = new IBM.WMQ.MQMessage();
RespMsg.MessageType = IBM.WMQ.MQC.MQMT_REPLY;
RespMsg.CorrelationId = ReqMsg.CorrelationId;

19. Also add the following lines of codes for GetMessageOptions.

gmo = new IBM.WMQ.MQGetMessageOptions();
gmo.Options = IBM.WMQ.MQC.MQGMO_WAIT;
gmo.WaitInterval = 5000;
gmo.MatchOptions = IBM.WMQ.MQC.MQMO_MATCH_CORREL_ID;

20. Now get the response message by passing RespMsg and gmo as parameters to ResponseQueue.Get() method.

ResponseQueue.Get(RespMsg, gmo);

21. Log the response message:
 
     System.Diagnostics.EventLog.WriteEntry("ResponseMessage",RespMsg.ReadString(RespMsg.MessageLength));

22. Sign and build the project and deploy it to Othe BTS. You can also create one send port to send the response message to the disk. After deployment configure the application with one receive port and location . Drop the generated instance into the the receive location and check your event log.
That's all, your job is done.

I tried explaining the subject in the best possible manner I could. Your doubts and suggestions would be whemingly responded.

Hope you all had a good read. Happy blogging and reading :) .

Thank You.

Saurav Suman
Software Engineer @ Agility
Hyderabad,India.

Parallel Convoy Correlation In BTS

Hi there,


In this blog , i'll show you how parallel convoy correlation works in biztalk by showing a pracical example.A parallel convoy enables multiple single messages to join together to achieve a required result.The set of related messages can arrive in any order, but BizTalk Server must receive all of them before starting the process.

Scenario:

Suppose you go to a shopping site and you decide to buy an item, your item will not be booked unless and until the system receives the payment confirmation as well as the stock availability.
We will be creating two schemas here, one for the payment and other for the stock availability with one common field element Status so that we are able to correlate these two schemas.

Steps

1.Create one schema for payment information

2.Create one schema for stock availability

3.Promote the Status element of both the schemas. (Do Quick Promotion)

4.Add an orchestration process to your project.

5.Perform the following steps in orchestration.

 a)Drag and drop a parallel action shape.

 

 b)Take two receive shape

 c)Now go to the orchestration view and expand types. Right click correlation types and add new            correlation type. Update the correlation property as shown in the figure below as well as the
   description,identifier.

 d)Similary add a new correlation set as shown in the figure.

 e)Rename one receive shape to Receive_Payment and set the following properties:

    Activate = true; Initializing Correlation Set=Name of Correlation Set; Message and operation.

 f)Rename the other receive shape to Receive_StockInfo and set the following properties:

    Activate = true; Initializing Correlation Set=Name of Correlation Set; Message and operation.

 g)Make a final destination schema as shown below

 h)Drop a transform shape in your orchestration process and update it as shown below,make         destination your final send shape.

 i)Link the source schema and destination schema as shown below

 
 j)Finally drop a send shape with message type = your destination schema

 f) Your whole business process should look like this:

6.Sign your project and deploy it to biztalk admin console

7.Configure your send, receive port and orchestration accordingly in administrative console.


Do leave your comments if you face any problem.

Happy blogging!!

Saurav Suman
Software Engineer @  Agility

Renaming Your Biztalk Machine Without Loosing Data

Hi there,

I recently came across a situation where i had to change my machine name.You all may be wondering about the big stuff in that but let me tell you that it turned out to be a big headache for me when i had to reconfigure my biztalk again as i had to preserve my old data.I followed the following steps which helped me to get my biztalk running back....

Note: Do this on your own risk, it worked for me but i'm sure if you follow the steps carefully it would work for you too.


Steps to configure your biztalk incase of machine name change

1. Add your current domain as member to each biztalk group by going to the computer management.

   Administrators,BizTalk Application Users,BizTalk Isolated Host Users,BizTalk Server  Administrators,BizTalk Server B2B Operators,BizTalk Server Operators,SSO Administrators(Very  Important).

2. Navigate to  your installation directory and search for SampleUpdateInfo.xml file.
   You can find SampleUpdateInfo file in Microsoft  Biztalk Server 2013\Bins32\Schema\Restore           folder.

3. Edit SampleUpdateInfo file like this:-
 
   Find and Replace all the “SourceServer” value with your original Server name.
   Find and Replace all the “DestinationServer” value with your new Server name.
   Stuff related to Analysis, BAM, RuleEngine, HWS, and EDI are commented by default, if you are      using them in your environment un-comment the required ones.

   save and close the file after editing.

4. Open Command prompt and change the directory to following location :
    D:\Installation\Microsoft  Biztalk Server 2013\Bins32\Schema\Restore

  (You would have your biztalk installed in different drive but the script files reside under                     \bins32\schema\Restore folder)

5.After changing the directory, run the following command to update the biztalk database:-

   cscript UpdateDatabase.vbs SampleUpdateInfo.xml

 //Make sure that the management database,messagebox db and rule engine db gets updated.

 Again run the following command for updating the registry:-

   cscript UpdateRegistry.vbs SampleUpdateInfo.xml

 //This script will update the local registry with the correct server name. You need to run this script on each BizTalk server you have    in the group.

6.Restart the WMI service by going to services.msc
 
  //This step is required because most of the administration tasks you perform from the admin console depends on WMI.

7.Promote the new server as master secret server:
  Follow these steps :-

A) On the Start menu, click All Programs, click Microsoft Enterprise Single Sign-On, and then click SSO Administration.

B) In the scope pane, right click System, and then click Properties. The Master secret server is displayed on the General tab of the System    Properties dialog box.

C) Click Change to select a new Master secret server.

D) Logon to the new Master secret server to restore the Master secret to the registry of the new Master secret server.

E) On the Start menu, click Run, and then type cmd.

F) At the command line prompt, go to the Enterprise Single Sign-On installation directory. The default installation directory is
   :\Program Files\Common Files\Enterprise Single Sign-On.

G) Restart the new Master Secret Server.

H) Type ssoconfig –restoreSecret , where is the path and name of the file where the master secret is stored.
   The master secret is stored in the registry at the following location:
   HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ENTSSO\SSOSS

8.Restart your machine.

9.Open Sql Server Management Studio and logon using windows authentication

  Check the following table and column name under BizTalkMgmtDb for new server name, if the column still has the old server name then update that with the new server name:-

   adm_group table, SSOServerName
   adm_server table, Name column
   adm_MessageBox, DBServerName

10. Restart all the BizTalk/SSO services.

11.Open the BizTalk administration console, click on the existing node (the one pointing to original server), right-click and remove.
 Right click on the “BizTalk Server 2006 Administration” node and select “Connect to Existing      Group…”. Provide the new Server Name and select the BizTalkMgmtDb database.

Click Ok.You will get all your application. Incase the host instance is unable to start, try reconfiguring it with your btsadmin credentials.

12. Restart your system again and try running any working application.If you are able to run your application then your setup was a success otherwise you need to look into the the error and try resolving those errors by yourself.