Configure Auto-Recovery in Oracle SOA Suite

Oracle SOA Suite 11g has some great features to recover faulted instances automatically. When a BPEL process flow errors out, it is retried with all its invocations. This is undesirable in some cases. For example if re-calling a composite results in duplicated data, data have been changed before the recovery is planned to be executed or you do not want to create too many composite instances in order to save the space in your SOAINFRA-schema. See below the different places where automatic recoveries are configured / disabled.

1) Change RecurringScheduleConfig (see also the screenshot below) 

• Right-click soa-infra (SOA_cluster_name) 
• Choose SOA Administration > BPEL Properties 
• Click "More BPEL Configuration Properties" 
• Click "Recovery Config" 
• Change values for RecurringScheduleConfig 
• maxMessageRaiseSize = 0 
• startWindowTime = 00:00 
• stopWindowTime = 00:00 
• Click Apply 

2) Change StartupScheduleConfig (see also the screenshot below)

• Right-click soa-infra (SOA_cluster_name)
• Choose SOA Administration > BPEL Properties
• Click "More BPEL Configuration Properties"
• Click "Recovery Config"
• Change values forStartupScheduleConfig
• maxMessageRaiseSize = 0
• startupRecoveryDuration = 0
• subsequentTriggerDelay = 0
• Click Apply

3) Change GlobalTxMaxRetry
The property GlobalTxMaxRetry specifies how many retries are performed if an error is identified as a retriable one. For example, after several web service invocations, if dehydration fails due to a data source error, then this is identified as a retriable error and all activities from the prior dehydration state are retried. If the activities being retried are not idempotent (that is, their state can change with each retry and is not guaranteed to give the same behavior), then multiple retries can be problematic.

You can set GlobalTxMaxRetry to 0 in the Systems MBean Browser.

• Right-click soa-infra (SOA_cluster_name)
• Choose SOA Administration > Common Properties
• Click "More SOA Infra Advanced Configuration Properties"
• Click "GlobalTxMaxRetry"
• In the Value field, enter an appropriate value
• Click Apply

Additional Information

• Oracle Fusion Middleware Administrator's Guide: Runtime Diagnostics Troubleshooting

Undeploy multiple SOA composites with WLST or ANT

As part of our current project the Build Management team asked for a solution to undeploy multiple composites at one time. Of course you have the “Undeploy All From This Partition” menu option in Enterprise Manager but since we have a lot of deployments every day the guys wanted to have a script solution. It is even more important for the nightly deployments on our continuous integration environment – strange, we couldn’t find anybody who wants to do the undeployment via Enterprise Manager manually every night ;-)

However with WLST or ANT the SOA Suite comes with two options to undeploy composites via script. In this article I’d like to explain you both ways.

Undeployment with WLST

You can test the steps below on Oracle's Pre-built Virtual Machine for SOA Suite and BPM Suite 11g.

1) Change to the WLST directory under MIDDLEWARE_HOME/Oracle_SOA1/common/bin.

cd /oracle/fmwhome/Oracle_SOA1/common/bin/

2) Open WLST

./wlst.sh

3)  Connect to the SOA server

wls:/offline> connect('weblogic','welcome1','t3://soabpm-vm:7001')
Connecting to t3://soabpm-vm:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'dev_bpm'.

wls:/dev_bpm/serverConfig>

4) Run the delete command for the appropriate partition

wls:/dev_bpm/serverConfig> sca_deletePartition('test')

partitionName = test

Partition was successfully deleted.

wls:/dev_bpm/serverConfig>

Please take into account that the command deletes all composites as well as the partition itself. If you need the partition for future deployments just recreate it with the sca_createPartition WLST-command. Also check the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference for a complete list of SOA Suite Custom WLST Commands.

Undeployment with ANT

Another option is to use ANT for the undeployment of multiple composites. The key here is to reference the file ant-sca-mgmt.xml within your custom ANT-target. The file comes with SOA Suite as well as JDeveloper. See my quick example below:

<target name="sca_deletePartition">

  <echo>Undeploy Composites</echo>      

  <ant antfile="/oracle/fmwhome/Oracle_SOA1/bin/ant-sca-mgmt.xml"

     inheritall="false" target="deletePartition">         

      <property name="host" value="soabpm-vm"/>         

      <property name="port" value="7001"/>         

      <property name="user" value="weblogic"/>         

      <property name="password" value="welcome1"/>         

      <property name="partition" value="testPartition"/>      

  </ant>   

</target>

Again this command deletes all composites of the given partition as well as the partition itself. See the Developer's Guide - Managing SOA Composite Applications with Script for more details.

Update: After reading this article Christoph Burmeister from our Build Management team created a pom-snippet for this to make the maven fans happy as well :-) Many thanks Christoph!!

Undeployment of composites when soa-infra is down

Did you already have the problem that you couldn’t restart your server because of a certain (maybe broken) composite? Did you ever wish to perform an undeployment of your composites while soa-infra is down? In some of our projects we already had this requirement. Below you’ll find an explanation with a step-by-step example of how-to get rid of your composites – even when the soa-infra is not reachable.

Enterprise Manager with a list of the deployed composites (relevant for this post: HelloWorld 1.0.0)

Problem

A restart of the Managed Server failed because one of the composites referred to a file that could not be found. Therefore we received the following error message:  

weblogic.work.ExecuteThread.run(ExecuteThread.java:176)

Caused By: java.io.IOException: oracle.mds.exception.MDSException: MDS-00054: The file to be loaded oramds:/deployed-composites/default/HelloWorld_rev1.0.0/… does not exist.

        at oracle.mds.internal.net.AbstractOraMDSURLConnection.getPDocStream(AbstractOraMDSURLConnection.java:388)

        at oracle.mds.internal.net.AbstractOraMDSURLConnection.getInputStream(AbstractOraMDSURLConnection.java:176)

        at oracle.mds.internal.net.OraMDSURLConnection.getInputStream(OraMDSURLConnection.java:67)

        at java.net.URL.openStream(URL.java:1010)

        at oracle.fabric.common.metadata.MetadataManagerImpl.getInputStreamFromAbsoluteURL(MetadataManagerImpl.java:276)

        at oracle.integration.platform.common.MDSMetadataManagerImpl.getInputStreamFromAbsoluteURL(MDSMetadataManagerImpl.java:555)

        at oracle.integration.platform.common.MDSMetadataManagerImpl.transferFile(MDSMetadataManagerImpl.java:852)

        at oracle.integration.platform.common.MDSMetadataManagerImpl.transferFiles(MDSMetadataManagerImpl.java:839)

        at oracle.integration.platform.common.MDSMetadataManagerImpl.copyTree(MDSMetadataManagerImpl.java:820)

        at

How-to fix it?

Since redeployment is not possible because of a non-reachable soa-infra, we needed to look for the available options to delete the current deployment in an offline mode. This should help us to restart the server again. For this we did some research in the web but unfortunately most of the suggested solutions are only applicable when soa-infra is online. One of these examples is the export / import functionality for the MDS via Enterprise Manager - see the article Undeploy composites manually for more details. The functionality is very useful but it requires a running soa application. After that I tried to export the deployed-composites.xml as described in the above blog using WLST – unfortunately the result wasn’t satisfying:

Command:

wls:/domain/serverConfig> exportMetadata(application='soa-infra',server='<MY_MANAGED_ SERVER_SOA>',toLocation='<MY_TO_LOCATION>',docs='/deployed-composites/deployed-composites.xml')

 

Result:

Location changed to domainRuntime tree. This is a read-only tree with DomainMBean as the root.
For more help, use help(domainRuntime)
Executing operation: exportMetadata.

Traceback (innermost last):
  File "<console>", line 1, in ?
  File "/opt/oracle/Middleware/oracle_common/common/wlst/mdsWLSTCommands.py", line 237, in exportMetadata
  File "/opt/oracle/Middleware/oracle_common/common/wlst/mdsWLSTCommands.py", line 727, in executeAppRuntimeMBeanOperation
  File "/opt/oracle/Middleware/oracle_common/common/wlst/mdsWLSTCommands.py", line 697, in getMDSAppRuntimeMBean
UserWarning: MDS-91002: MDS Application runtime MBean for "soa-infra" is not available. "exportMetadata" operation failure.

Solution

Besides other options like direct access to MDS via Java (Full access to MDS Repository) or SQL (MDS_PATHS Table) we discovered a new and up-to-date Metalink-Note ([ID 1380835.1]) which described basically what we wanted. Thanks a lot to Markus Lohn (Oracle) for his support! We successfully followed the steps within the note and after that the server came up. Below I describe the approach with the above Hello World example. 

1. Download and copy the ShareSoaInfraPartition.ear from Oracle to access the MDS offline (Download it from the Metalink note). 

2. Connect to WLST and deploy the above application on the AdminServer. Use the WLST under $MIDDLEWARE_HOME/ oracle_common/common/bin.

3. Run the below command to export the metadata from the AdminServer. Change the values appropriate to your environment.

Command:

wls:/domain/serverConfig> 

exportMetadata(application='ShareSoaInfraPartition',server='<MY_ADMIN_ SERVER>',toLocation='<MY_TO_LOCATION>',docs='/deployed-composites/deployed-composites.xml')

Result:

Executing operation: exportMetadata.
Operation "exportMetadata" completed. Summary of "exportMetadata" operation is:
List of documents successfully transferred:
/deployed-composites/deployed-composites.xml
1 documents successfully transferred.

4. Open the deployed-composites.xml within deployed-composites folder under <MY_TO_LOCATION>.

5. Delete the composite series or revision which is causing the problem and save the file.

<composite-series name="default/HelloWorld" default="default/HelloWorld!1.0.0">
  <composite-revision dn="default/HelloWorld!1.0.0" state="on" mode="active" location="dc/soa_09001e40-be6f-4bc4-8a96-201f3848c38d">
    <composite dn="default/HelloWorld!1.0.0*soa_09001e40-be6f-4bc4-8a96-201f3848c38d" deployedTime="2012-03-09T13:15:14.139+01:00"/>
  </composite-revision>
</composite-series>

6. Import the updated file under <MY_FROM_LOCATION> into MDS.

Command:

wls:/domain/serverConfig> importMetadata(application='ShareSoaInfraPartition',server='AdminServer',fromLocation='<MY_FROM_LOCATION>',docs='/deployed-composites/deployed-composites.xml')

Result:

Executing operation: importMetadata.

Operation "importMetadata" completed. Summary of "importMetadata" operation is:
List of documents successfully transferred:

/deployed-composites/deployed-composites.xml
1 documents successfully transferred.

7. Restart your SOA server or soa-infra.

Enterprise Manager with a list of the deployed composites (note that HelloWorld 1.0.0 is not available)