Purging data from Oracle SOA Suite 11g

Part1: How can I purge data from Oracle SOA Suite 11g (PS6 11.1.1.7) using the purge script provided by Oracle?

Introduction

This blog will explain how to purge (remove unwanted data) data within Oracle SOA Suite 11g (PS6 11.1.1.7).

The series of blogs will cover the following:

• Part 1: How can I purge data from Oracle SOA Suite 11g (PS6 11.1.1.7) using the purge script provided by Oracle? 
• How does Oracle SOA Suite 11g (PS6 11.1.1.7) store data? 
• What data does Oracle SOA Suite 11g (PS6 11.1.1.7) store? 
• Why do you need to purge Oracle SOA Suite 11g (PS6 11.1.1.7) data? 
• What are the purging options available for Oracle SOA Suite 11g (PS6 11.1.1.7)?
• Which data will be purged by the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script? 
• List of composite instance states that will be considered for purging by the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script 
• How to install the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script? 
• How to execute the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script? 
• What is Looped purging (Oracle SOA Suite 11g (PS6 11.1.1.7) purge script)? 
• What is Parallel purging (Oracle SOA Suite 11g (PS6 11.1.1.7) purge script)? 
• Description of parameters used by the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script 
• Example 1: Executing the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script for all composites 
• Example 2: Executing the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script for a specific composite

Oracle SOA Suite 11g (PS6 11.1.1.7) data

How does Oracle SOA Suite 11g (PS6 11.1.1.7) store data?
SOA Suite uses a database schema called SOAINFRA (collection of database objects such as tables, views, procedures, functions etc.) to store data required for the running of SOA Suite applications. The SOAINFRA (SOA Infrastructure) schema is also referred to as the ‘dehydration store’ acting as the persistence layer for capturing SOA Suite data.

What data does Oracle SOA Suite 11g (PS6 11.1.1.7) store?

Composite instances utilising the SOA Suite Service Engines (BPEL, mediator, human task, rules, BPM, OSB, EDN etc.) will write data to tables residing within the SOAINFRA schema. Each of the engines will either write data to specific engine tables (e.g. the CUBE_INSTANCE table is used solely by the BPEL engine) or common tables that are shared by the SOA Suite engines such as the AUDIT_TRAIL table.
Few examples (below) of the type of data that is stored within the SOAINFRA schema:

• Message payload (e.g. input, output) 
• Scope (e.g. variables)
• Auditing (e.g. data flow timestamps) 
• Faults 
• Deferred (messages that can be recovered) 
• Metrics 

Why do you need to purge Oracle SOA Suite 11g (PS6 11.1.1.7) data?

Data within the Oracle SOA Suite database can grow to substantial levels in a short space of time. Payload sizes and volume of data will have an impact on available disk space which in turn will affect the performance of SOA Suite. For example, EM console can often become slow to navigate, increasing number of messages becoming stuck or requiring recovery, JTA transaction problems etc.

Purging itself, can become challenging if the data has not been maintained due to the large number of composite instances. Therefore, establishing a purge strategy and implementing it on a regular basis will help maintain the health of SOA Suite keeping the environment running efficiently.


What are the purging options available for Oracle SOA Suite 11g (PS6 11.1.1.7)?

Oracle provides three options for purging Oracle SOA Suite 11g data:

• EM Console: Within the Enterprise Manager console the ‘Delete with Options’ can be used to manually delete many instances at once however, this may lead to transaction timeouts and is not recommended for large volumes. 
• Purge Script: This is the process of deleting instances that are no longer required using stored procedures that are provided with Oracle SOA Suite 11g out of the box. 
• Partitioning: Instances are segregated based on user defined criteria within the database, when a partition is not required it will be dropped freeing the disk space.

Oracle SOA Suite 11g (PS6 11.1.1.7) purge script

Which data will be purged by the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script?

The purge script will delete composite instances that are in the following states:

Completed
The purge script will NOT delete composite instances that are in the following states:

Running (in-flight)
Suspended
List of composite instance states that will be considered for purging by the
Oracle SOA Suite 11g (PS6 11.1.1.7) purge script:
How to install the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script?
The following details will be required:

• Database host details:
  - hostname (IP address)
  - username
  - password
  
• SOA Database schema details:
  - prefix
  - password
  
• Full path of the SOA Suite home folder
• Full path of the directory where the Oracle purge script will write log information to (a folder on the database host) 

‘DEV'was the soainfra schema prefix used for the examples below.

a. Log into the Database host server.

sqlplus / as sysdba

c. Grant privileges to the soainfra (database) user that will be executing the scripts:

GRANT EXECUTE ON DBMS_LOCK TO _SOAINFRA;
GRANT CREATE JOB TO _SOAINFRA;
GRANT CREATE EXTERNAL JOB TO _SOAINFRA;

d. Exit SQL*Plus and go to the location of the Oracle purge script:

exit
$cd /rcu/integration/soainfra/sql/soa_purge/

e. Connect to the database as the soainfra user using SQL*Plus:

sqlplus _SOAINFRA/

@soa_purge_scripts.sql

Procedure created.
Function created.
Type created.
Type body created.
PL/SQL procedure successfully completed.
Package created.
Package body created.

f. Exit SQL*Plus and create a directory where the log files (generated by the Oracle purge script) should be written to:

exit
$mkdir -p /PurgeLogs

g. Connect to the database with SQL*Plus as SYSDBA and declare the directory:

sqlplus / as sysdba

CREATE OR REPLACE DIRECTORY SOA_PURGE_DIR AS '/PurgeLogs';

GRANT READ, WRITE ON DIRECTORY SOA_PURGE_DIR TO _SOAINFRA;

All the database objects required for purging data using the Oracle purge script are now loaded into the SOAINFRA schema ready for use.
How to execute the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script?

There are two options for running the purge script:

• Looped
• Parallel 

What is Looped purging (Oracle SOA Suite 11g (PS6 11.1.1.7) purge script)?

Looped purge is a single threaded PL/SQL script that will iterate through the SOAINFRA tables and delete instances matching the parameters specified.
What is Parallel purging (Oracle SOA Suite 11g (PS6 11.1.1.7) purge script)?

Parallel purge is essentially the same as the looped purge. It is meant to be more efficient as it uses the dbms_scheduler package to spawn multiple purge jobs all working on a distinct subset of data. There are 2 more parameters that can be specified in addition to the ones used by the looped purge. This is designed to purge large data volumes hosted on high-end database nodes with multiple CPUs and a good IO sub-system. A maintenance window should be used as it requires a lot of resources.

Example 1: Executing the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script to purge data for all composites

DECLARE

max_creation_date timestamp;
min_creation_date timestamp;
batch_size integer;
max_runtime integer;
retention_period timestamp;

BEGIN
min_creation_date := to_timestamp('2010-06-01','YYYY-MM-DD');
max_creation_date := to_timestamp('2010-06-30','YYYY-MM-DD');
max_runtime := 60;
retention_period := to_timestamp('2010-07-01','YYYY-MM-DD');
batch_size := 10000;

soa.delete_instances(
min_creation_date => min_creation_date,
max_creation_date => max_creation_date,
batch_size => batch_size,
max_runtime => max_runtime,
retention_period => retention_period);

END;
/

b. Parallel

DECLARE

max_creation_date timestamp;
min_creation_date timestamp;
batch_size integer;
max_runtime integer;
retention_period timestamp;
DOP integer;
max_count integer;
purge_partitioned_component boolean;

BEGIN
min_creation_date := to_timestamp('2010-06-01','YYYY-MM-DD');
max_creation_date := to_timestamp('2010-06-30','YYYY-MM-DD');
max_runtime := 60;
retention_period := to_timestamp('2010-07-01','YYYY-MM-DD');
batch_size := 10000;
DOP := 3;
max_count := 1000000;
purge_partitioned_component := false);


soa.delete_instances_in_parallel (
min_creation_date => min_creation_date,
max_creation_date => max_creation_date,
batch_size => batch_size,
max_runtime => max_runtime,
retention_period => retention_period,
DOP => DOP,
max_count => max_count,
purge_partitioned_component => purge_partitioned_component);

END;
/

Example 2: Executing the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script to purge data from a specific composite

Same as Example 1 Scenario but with an additional requirement of only purging data from the composite named OrderBookingComposite . No other composite data should be purged.

Composite details can be gathered by querying the COMPOSITE_INSTANCE table within the SOAINFRA schema. The column named COMPOSITE_DN (distinguished name) holds the details required by the purge script:
a. Looped

DECLARE
min_creation_date timestamp;
max_creation_date timestamp;
batch_size number;
max_runtime number;
retention_period timestamp;
purge_partitioned_component boolean;
composite_name varchar2(200);
composite_revision varchar2(200);
soa_partition_name varchar2(200);

BEGIN
min_creation_date := to_timestamp('2010-06-01','YYYY-MM-DD');
max_creation_date := to_timestamp('2010-06-30','YYYY-MM-DD');
max_runtime := 60;
retention_period := to_timestamp('2010-07-01','YYYY-MM-DD');
batch_size := 10000;
purge_partitioned_component := true;
composite_name := 'OrderBookingComposite';
composite_revision := '1.0';
soa_partition_name := 'default';


soa.delete_instances(
min_creation_date => min_creation_date,
max_creation_date => max_creation_date,
batch_size => batch_size,
max_runtime => max_runtime,
retention_period => retention_period,
purge_partitioned_component => purge_partitioned_component,
composite_name => composite_name,
composite_revision => composite_revision,
soa_partition_name => soa_partition_name);

END;
/

b. Parallel

DECLARE
min_creation_date timestamp;
max_creation_date timestamp;
batch_size number;
max_runtime number;
retention_period timestamp;
DOP integer;
max_count integer;
purge_partitioned_component boolean;
composite_name varchar2(200);
composite_revision varchar2(200);
soa_partition_name varchar2(200);

BEGIN
min_creation_date := to_timestamp('2010-06-01','YYYY-MM-DD');
max_creation_date := to_timestamp('2010-06-30','YYYY-MM-DD');
max_runtime := 60;
retention_period := to_timestamp('2010-07-01','YYYY-MM-DD');
batch_size := 10000;
DOP := 3
max_count := 1000000;
purge_partitioned_component := true;
composite_name := 'OrderBookingComposite';
composite_revision := '1.0';
soa_partition_name := 'default';


soa.delete_instances(
min_creation_date => min_creation_date,
max_creation_date => max_creation_date,
batch_size => batch_size,
max_runtime => max_runtime,
retention_period => retention_period,
DOP => DOP,
max_count => max_count,
purge_partitioned_component => purge_partitioned_component,
composite_name => composite_name,
soa_partition_name => soa_partition_name);

END;
/

Conclusion

This blog has provided a basic understanding of the purge script contained within Oracle SOA Suite 11g (PS6 11.1.1.7).

A long term purging strategy needs to be implemented and in order to do so, a good understanding of the workings of the purge script is required along with an awareness of the issues related to the script.

Therefore, leading on from part 1 there will be few more blogs that will cover other the following: