Oracle SOA / BPM 12c - Useful Upgrade Content

With this post I'd like to provide a list of useful material regarding Oracle SOA 12c Upgrade.

• Oracle® Fusion Middleware Upgrading to the Oracle Fusion Middleware Infrastructure
  https://docs.oracle.com/middleware/1221/core/INFUP/toc.htm 
• Oracle® Fusion Middleware Upgrading SOA Suite and Business Process Management
  https://docs.oracle.com/middleware/1221/core/FUPSS/toc.htm
• Best practices upgrade blogs from Oracle:
  https://blogs.oracle.com/SOA/tags/upgrade    
• Slideshare (UKOUG 2015): What's Stopping You from Upgrading to Oracle SOA Suite 12c?
  http://www.slideshare.net/DaniloSchmiedel1/whats-stopping-you-from-upgrading-to-oracle-soa-suite-12c
• Slideshare: Road to SOA Suite 12c
  http://de.slideshare.net/opitzconsulting/doag-2014-roadtosoasuite12cschmiedel 
• Upgrade videos:
  http://docs.oracle.com/middleware/1221/cross/upgrade_videos.htm
• 2 Minute Tech Tip
  https://blogs.oracle.com/archbeat/entry/2mtt 
• Troubleshooting the Upgrade
  http://docs.oracle.com/middleware/1213/core/FUPSS/know_base.htm#FUPSS339 

In addition I'd like to mention the SOA Expert Series. In this webinar, David Shaffer (Middleworks),  Deepak Arora (Oracle A-Team), Antony Reynolds & Kathryn Lustenberger (Oracle Prod Mgmt) and myself shared tips, tricks and best practices for upgrading to SOA Suite 12c. You can download the slides as well as the recording from middleworks.com.
 

Upgrading Oracle SOA Suite from 11g to 12c

Today I'd like to share my presentation on upgrading Oracle SOA Suite from 11g to 12c. It contains information on how the product evolved the last couple of years. It also explains different upgrade strategies, the difference between in-flight upgrade and migration, important pre-upgrade tasks, the upgrade steps itself and post-upgrade steps.   

Additionally it lists the experiences and upgrade results that we've achieved in two different scenarios: in-flight upgrade as well as migration. 

Road to SOA Suite 12c

Moving from a File-Based MDS Repository to a Database-Based MDS in Oracle BPM / SOA 12c

You might have noticed that Business Rules editing during runtime is not available in SOA / BPM 12c domains that have been installed from the Quickstart distribution. This is because for those domain configs the underlying MDS is configured as a file-based repository. For the development in our projects we very often use a SOA / BPM Compact Domain. The change of Business Rules without re-deployment is quite important for us because we work a lot with Adaptive Case Management – and there Business Rules and live changes are a key benefit (which we are also demonstrating in demos / POCs / etc.).

Rules Editing in SOA Composer (with DB-based MDS)

You can move from a file-based repository to a database-based repository. In our CattleCrew-Blogpost Re-configure a compact domain to use a DB-based MDS instead of a File-based MDS my colleague Sven Bernhardt and me describe how to do that. Please note that you cannot move from a database-based repository to a file-based repository. Please also note that this approach is not documented in the Oracle documentation, so it is not officially supported

After going through the steps below it should be possible to edit rules and DVMs during runtime.

Oracle BPM 12c - Quick Start Installation (uncensored)

Getting started in 15 minutes!

One of the challenges with previous releases was, that SOA & BPM composites couldn't be deployed and tested on the JDeveloper integrated Weblogic server. Therefore a separate installation of SOA/BPM Suite or a virtual image was necessary to start developing. Now with the new release Oracle introduced a single-click installer for SOA & BPM Suite. Among other new features (like debugging & testing capabilities, templating, optimized foodprint, etc.) this really helps to increase developer productivity.

The video below demonstrates that with Oracle SOA & BPM 12c it just takes 15 minutes to get started - install JDeveloper, start the Weblogic server, develop a simple Hello World, deploy the process and test it from Enterprise Manager.


Do you feel inspired? Just download the software from OTN and try it yourself (SOA-Download; BPM-Download). Have fun!

Configure Auto-Recovery in Oracle SOA Suite

Oracle SOA Suite 11g has some great features to recover faulted instances automatically. When a BPEL process flow errors out, it is retried with all its invocations. This is undesirable in some cases. For example if re-calling a composite results in duplicated data, data have been changed before the recovery is planned to be executed or you do not want to create too many composite instances in order to save the space in your SOAINFRA-schema. See below the different places where automatic recoveries are configured / disabled.

1) Change RecurringScheduleConfig (see also the screenshot below) 

• Right-click soa-infra (SOA_cluster_name) 
• Choose SOA Administration > BPEL Properties 
• Click "More BPEL Configuration Properties" 
• Click "Recovery Config" 
• Change values for RecurringScheduleConfig 
• maxMessageRaiseSize = 0 
• startWindowTime = 00:00 
• stopWindowTime = 00:00 
• Click Apply 

2) Change StartupScheduleConfig (see also the screenshot below)

• Right-click soa-infra (SOA_cluster_name)
• Choose SOA Administration > BPEL Properties
• Click "More BPEL Configuration Properties"
• Click "Recovery Config"
• Change values forStartupScheduleConfig
• maxMessageRaiseSize = 0
• startupRecoveryDuration = 0
• subsequentTriggerDelay = 0
• Click Apply

3) Change GlobalTxMaxRetry
The property GlobalTxMaxRetry specifies how many retries are performed if an error is identified as a retriable one. For example, after several web service invocations, if dehydration fails due to a data source error, then this is identified as a retriable error and all activities from the prior dehydration state are retried. If the activities being retried are not idempotent (that is, their state can change with each retry and is not guaranteed to give the same behavior), then multiple retries can be problematic.

You can set GlobalTxMaxRetry to 0 in the Systems MBean Browser.

• Right-click soa-infra (SOA_cluster_name)
• Choose SOA Administration > Common Properties
• Click "More SOA Infra Advanced Configuration Properties"
• Click "GlobalTxMaxRetry"
• In the Value field, enter an appropriate value
• Click Apply

Additional Information

• Oracle Fusion Middleware Administrator's Guide: Runtime Diagnostics Troubleshooting

Remove data in Oracle MDS

The two main database schemas in the Oracle SOA Suite database repository are: <PREFIX>_SOAINFRA and <PREFIX>_MDS. Composite instance and runtime information are stored in the SOAINFRA schema. Commonly used metadata like WSDLs, XSDs, rules, fault policies, etc. as well as composite deployments are stored within the MDS schema. 

With every deployment / import of the metadata artifacts a new document version will be created in the MDS. This means that re-importing an updated WSDL-file into the MDS does not delete the previous version of the document. Furthermore we sometimes need to remove unnecessary and unwanted files from the repository. If this is not considered you might end in problems like below:

ORA-01654: unable to extend index DEV_MDS.MDS_PATHS_U2 by 128 in tablespace DEV_MDS
ORA-06512: at "DEV_MDS.MDS_INTERNAL_COMMON", line 865
ORA-06512: at "DEV_MDS.MDS_INTERNAL_COMMON", line 1021
ORA-06512: at "DEV_MDS.MDS_INTERNAL_COMMON", line 1121
ORA-06512: at "DEV_MDS.MDS_INTERNAL_COMMON", line 1216
ORA-06512: at "DEV_MDS.MDS_INTERNAL_COMMON", line 1872
ORA-06512: at line 1
 

In order to avoid production problems because of a full MDS tablespace you should clean up the schema from time to time. This post explains the options that Oracle SOA Suite provides to remove contents from MDS.

OPTION 1: Remove directories and files from MDS using WLST

1) Start WLST from SOA_HOME/common/bin/wlst.sh.

Example:
 
[oracle@soabpm-vm ~]$ cd /oracle/fmwhome/Oracle_SOA1/common/bin 
[oracle@soabpm-vm bin]$ ./wlst.sh 
wls:/offline>
 

2) Execute the following command: sca_removeSharedData('http://<soahost>:<soaport>', 'directory', 'user', 'password')

Example (delete folder 'interfaces' and all its subdirectories and files):

wls:/offline> sca_removeSharedData(‘http://localhost:8001’, ‘interfaces’, ‘weblogic’, ‘welcome1’)
 

Note: With the command above you can just remove directories and files which are stored under "apps".

OPTION 2: Remove directories and files from MDS using ANT

Oracle provides some ANT scripts which can be used to integrate the "remove" command into your central build & deploy proccess. Just search for the "removeSharedData" command in ant-sca-deploy.xml. On the server this file is located under SOA_HOME/bin. You can also find the file in your JDeveloper install folder under MIDDLEWARE_HOME/jdeveloper/bin.

1) Execute the command: ant -f ant-sca-deploy.xml removeSharedData -DserverURL=server.url -DfolderName=folder.name -Drealm=realm -Duser=user -Dpassword=password -DfailOnError=true/false

Example (delete folder 'interfaces' and all its subdirectories and files):
 
ant -f ant-sca-deploy.xml removeSharedData -DserverURL=http://localhost:8001 -DfolderName=interfaces"
 

Note: With the command above you can just remove directories and files which are stored under "apps". See the Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite for a detailed description of the parameters.

OPTION 3: Remove files from MDS using WLST

1) Start WLST from SOA_HOME/common/bin/wlst.sh

Example:

[oracle@soabpm-vm ~]$ cd /oracle/fmwhome/Oracle_SOA1/common/bin
[oracle@soabpm-vm bin]$ ./wlst.sh
wls:/offline>
 

2) Connect to the SOA server: connect('user', 'password', 't3://<soahost>:<soaport>')

Example: 

wls:/offline> connect('weblogic', 'welcome1', 't3://localhost:8001') 
 

3) Execute the command: deleteMetadata(application=’application-name', server=’soaserver-name’, docs=’absolutePath’)

Example (delete all files in '/apps/interfaces' as well as the files in its subdirectories): 
 
wls:/...> deleteMetadata(application='soa-infra',server='soa_server1',docs='/apps/interfaces/**') 
 

Note: With the command above all files in directory 'apps/interfaces' will be deleted. Please note the double asterisk (**) at the end of the this parameter. The asterisk (*) represents all documents under the current directory. The double asterisk (**) represents all documents under the current directory and also recursively includes all documents in subdirectories. Furthermore see the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference for a detailed description of the valid arguments.

OPTION 4: Remove files from MDS using MBean Browser

You can call the command "deleteMetadata" also from the Enterprise Manager / MBean Browser.

1) Login to EM (http://host:port/em)
2) Expand SOA
3) Right-click on soa-infra
4) Select Administration -> MDS Configuration
5) Click Runtime MBean Browser
6) Click Operations tab
7) Click deleteMetadata operation
8) Provide parameters:
      docs - list of entries to remove (fully qualified path, eg: /apps/interfaces/**)
      restrictCustTo - default
      excludeAllCust - false
      excludeBaseDocs - false
      excludeExtendedMetadata - false
      cancelOnException - true
9) Click Invoke

Note: See the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference for a detailed description of the parameters.

OPTION 5: Purge Metadata Version History using Fusion Middleware Control

For database-based MDS, you can purge the metadata version history using the Fusion Middleware Control (Enterprise Manager). This operation purges the version history of unlabeled documents from the application's repository partition. The tip version (the latest version) is not purged, even if it is unlabeled.

1) Login to EM (http://host:port/em)
2) Expand SOA
3) Right-click on soa-infra
4) Select Administration -> MDS Configuration
5) Scroll to the “Purge” section: enter a value in the Purge all unlabeled past versions older than field and click on the Purge button
6) In the Confirmation dialog box, click Close

Note: For more details see the Oracle Fusion Middleware Administrators Guide. Section 14.3.11.2 also explains how to purge metadata version history using WLST.

How to view the MDS content

After you removed files and/or directories from the MDS you might want to have a look at the actual content. The easiest way to do this is to create a MDS connection in JDeveloper. For more details see the Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite section "43.4.2.1 Create a SOA-MDS Connection".

Some additional information about the MDS

• Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite - Deploying and Using Shared Metadata Across SOA Composite Applications
• Oracle Fusion Middleware Administrator's Guide - Managing  the Oracle Metadata Repository
• Storing SCA Metadata in the Oracle Metadata Services Repository
• Full access to MDS Repository

Connect to multiple SOA / BPM Virtual Machines from different computers in your network

Today I’d like to explain how to share instances of Oracle's Pre-built Virtual Machine for SOA Suite and BPM Suite across different computers in your network. Three requirements I had to address:

1. Call of external services like GeoNames (http://www.geonames.org/postal-codes/) from within the BPM Suite VM
2. Connection to multiple BPM Suite VMs from Host (e.g. JDeveloper and Browser)
3. Connection to multiple BPM Suite VMs from different developer computers in the network 
4. Share data between VMs and developer computers via FTP

The default options already support connections between host and client. However the target here was to address multiple independent VMs from different computers in the same network. The following settings worked in my environment:

 

1) Start the Oracle VM VirtualBox Manager and open network settings of the VM.

 

2) Select Bridged Adapter in “Attached to”. Bridged Networking is used when you want your vm to be a full network citizen, i.e. to be an equal to your host machine on the network (see Networking in VirtualBox for more details).

 

3) Select the name of your network adapter. Take into account that when you change your connection method (WIFI / Cable) a different network adapter is required. Get the appropriate name from the network settings of the host.

 

4) Specify the Adapter Type.

 

5) Start the VM.

 

6) Per default the VM receives the IP via DHCP method. Open a terminal window in the client and send the following command as root user to get the current IP address information of your VM. Instead of DHCP you could also change the connection method of your VM to a static address.

ifconfig <CONNECTION-NAME>  (e.g. ifconfig eth1) 

7) Another way to get the IP information is the network icon at the bottom of the Oracle VM VirtualBox window.

 

8) Connect or ping this IP address (here: 192.168.2.131) from the host or from a different computer in your network. I suggest adding the IP address to your etc/hosts-file in order to address the VM via names instead of a static IP addresses. If successful start the SOA / BPM Server and enjoy the remote development. If not please check your firewall and router settings. Good luck!

 

Undeploy multiple SOA composites with WLST or ANT

As part of our current project the Build Management team asked for a solution to undeploy multiple composites at one time. Of course you have the “Undeploy All From This Partition” menu option in Enterprise Manager but since we have a lot of deployments every day the guys wanted to have a script solution. It is even more important for the nightly deployments on our continuous integration environment – strange, we couldn’t find anybody who wants to do the undeployment via Enterprise Manager manually every night ;-)

However with WLST or ANT the SOA Suite comes with two options to undeploy composites via script. In this article I’d like to explain you both ways.

Undeployment with WLST

You can test the steps below on Oracle's Pre-built Virtual Machine for SOA Suite and BPM Suite 11g.

1) Change to the WLST directory under MIDDLEWARE_HOME/Oracle_SOA1/common/bin.

cd /oracle/fmwhome/Oracle_SOA1/common/bin/

2) Open WLST

./wlst.sh

3)  Connect to the SOA server

wls:/offline> connect('weblogic','welcome1','t3://soabpm-vm:7001')
Connecting to t3://soabpm-vm:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'dev_bpm'.

wls:/dev_bpm/serverConfig>

4) Run the delete command for the appropriate partition

wls:/dev_bpm/serverConfig> sca_deletePartition('test')

partitionName = test

Partition was successfully deleted.

wls:/dev_bpm/serverConfig>

Please take into account that the command deletes all composites as well as the partition itself. If you need the partition for future deployments just recreate it with the sca_createPartition WLST-command. Also check the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference for a complete list of SOA Suite Custom WLST Commands.

Undeployment with ANT

Another option is to use ANT for the undeployment of multiple composites. The key here is to reference the file ant-sca-mgmt.xml within your custom ANT-target. The file comes with SOA Suite as well as JDeveloper. See my quick example below:

<target name="sca_deletePartition">

  <echo>Undeploy Composites</echo>      

  <ant antfile="/oracle/fmwhome/Oracle_SOA1/bin/ant-sca-mgmt.xml"

     inheritall="false" target="deletePartition">         

      <property name="host" value="soabpm-vm"/>         

      <property name="port" value="7001"/>         

      <property name="user" value="weblogic"/>         

      <property name="password" value="welcome1"/>         

      <property name="partition" value="testPartition"/>      

  </ant>   

</target>

Again this command deletes all composites of the given partition as well as the partition itself. See the Developer's Guide - Managing SOA Composite Applications with Script for more details.

Update: After reading this article Christoph Burmeister from our Build Management team created a pom-snippet for this to make the maven fans happy as well :-) Many thanks Christoph!!

Undeployment of composites when soa-infra is down

Did you already have the problem that you couldn’t restart your server because of a certain (maybe broken) composite? Did you ever wish to perform an undeployment of your composites while soa-infra is down? In some of our projects we already had this requirement. Below you’ll find an explanation with a step-by-step example of how-to get rid of your composites – even when the soa-infra is not reachable.

Enterprise Manager with a list of the deployed composites (relevant for this post: HelloWorld 1.0.0)

Problem

A restart of the Managed Server failed because one of the composites referred to a file that could not be found. Therefore we received the following error message:  

weblogic.work.ExecuteThread.run(ExecuteThread.java:176)

Caused By: java.io.IOException: oracle.mds.exception.MDSException: MDS-00054: The file to be loaded oramds:/deployed-composites/default/HelloWorld_rev1.0.0/… does not exist.

        at oracle.mds.internal.net.AbstractOraMDSURLConnection.getPDocStream(AbstractOraMDSURLConnection.java:388)

        at oracle.mds.internal.net.AbstractOraMDSURLConnection.getInputStream(AbstractOraMDSURLConnection.java:176)

        at oracle.mds.internal.net.OraMDSURLConnection.getInputStream(OraMDSURLConnection.java:67)

        at java.net.URL.openStream(URL.java:1010)

        at oracle.fabric.common.metadata.MetadataManagerImpl.getInputStreamFromAbsoluteURL(MetadataManagerImpl.java:276)

        at oracle.integration.platform.common.MDSMetadataManagerImpl.getInputStreamFromAbsoluteURL(MDSMetadataManagerImpl.java:555)

        at oracle.integration.platform.common.MDSMetadataManagerImpl.transferFile(MDSMetadataManagerImpl.java:852)

        at oracle.integration.platform.common.MDSMetadataManagerImpl.transferFiles(MDSMetadataManagerImpl.java:839)

        at oracle.integration.platform.common.MDSMetadataManagerImpl.copyTree(MDSMetadataManagerImpl.java:820)

        at

How-to fix it?

Since redeployment is not possible because of a non-reachable soa-infra, we needed to look for the available options to delete the current deployment in an offline mode. This should help us to restart the server again. For this we did some research in the web but unfortunately most of the suggested solutions are only applicable when soa-infra is online. One of these examples is the export / import functionality for the MDS via Enterprise Manager - see the article Undeploy composites manually for more details. The functionality is very useful but it requires a running soa application. After that I tried to export the deployed-composites.xml as described in the above blog using WLST – unfortunately the result wasn’t satisfying:

Command:

wls:/domain/serverConfig> exportMetadata(application='soa-infra',server='<MY_MANAGED_ SERVER_SOA>',toLocation='<MY_TO_LOCATION>',docs='/deployed-composites/deployed-composites.xml')

 

Result:

Location changed to domainRuntime tree. This is a read-only tree with DomainMBean as the root.
For more help, use help(domainRuntime)
Executing operation: exportMetadata.

Traceback (innermost last):
  File "<console>", line 1, in ?
  File "/opt/oracle/Middleware/oracle_common/common/wlst/mdsWLSTCommands.py", line 237, in exportMetadata
  File "/opt/oracle/Middleware/oracle_common/common/wlst/mdsWLSTCommands.py", line 727, in executeAppRuntimeMBeanOperation
  File "/opt/oracle/Middleware/oracle_common/common/wlst/mdsWLSTCommands.py", line 697, in getMDSAppRuntimeMBean
UserWarning: MDS-91002: MDS Application runtime MBean for "soa-infra" is not available. "exportMetadata" operation failure.

Solution

Besides other options like direct access to MDS via Java (Full access to MDS Repository) or SQL (MDS_PATHS Table) we discovered a new and up-to-date Metalink-Note ([ID 1380835.1]) which described basically what we wanted. Thanks a lot to Markus Lohn (Oracle) for his support! We successfully followed the steps within the note and after that the server came up. Below I describe the approach with the above Hello World example. 

1. Download and copy the ShareSoaInfraPartition.ear from Oracle to access the MDS offline (Download it from the Metalink note). 

2. Connect to WLST and deploy the above application on the AdminServer. Use the WLST under $MIDDLEWARE_HOME/ oracle_common/common/bin.

3. Run the below command to export the metadata from the AdminServer. Change the values appropriate to your environment.

Command:

wls:/domain/serverConfig> 

exportMetadata(application='ShareSoaInfraPartition',server='<MY_ADMIN_ SERVER>',toLocation='<MY_TO_LOCATION>',docs='/deployed-composites/deployed-composites.xml')

Result:

Executing operation: exportMetadata.
Operation "exportMetadata" completed. Summary of "exportMetadata" operation is:
List of documents successfully transferred:
/deployed-composites/deployed-composites.xml
1 documents successfully transferred.

4. Open the deployed-composites.xml within deployed-composites folder under <MY_TO_LOCATION>.

5. Delete the composite series or revision which is causing the problem and save the file.

<composite-series name="default/HelloWorld" default="default/HelloWorld!1.0.0">
  <composite-revision dn="default/HelloWorld!1.0.0" state="on" mode="active" location="dc/soa_09001e40-be6f-4bc4-8a96-201f3848c38d">
    <composite dn="default/HelloWorld!1.0.0*soa_09001e40-be6f-4bc4-8a96-201f3848c38d" deployedTime="2012-03-09T13:15:14.139+01:00"/>
  </composite-revision>
</composite-series>

6. Import the updated file under <MY_FROM_LOCATION> into MDS.

Command:

wls:/domain/serverConfig> importMetadata(application='ShareSoaInfraPartition',server='AdminServer',fromLocation='<MY_FROM_LOCATION>',docs='/deployed-composites/deployed-composites.xml')

Result:

Executing operation: importMetadata.

Operation "importMetadata" completed. Summary of "importMetadata" operation is:
List of documents successfully transferred:

/deployed-composites/deployed-composites.xml
1 documents successfully transferred.

7. Restart your SOA server or soa-infra.

Enterprise Manager with a list of the deployed composites (note that HelloWorld 1.0.0 is not available)