NOTE:
Please note that the instructions given in this blog post are applicable ONLY for ESB-4.0.X and G-reg-4.1.X, G-reg-4.5.0 versions. The deployment approach of the latest ESB versions has been changed and the recommended deployment synchronization approach has been switched to SVN based synchronizer. Because of that the following set of instructions cannot be applied to the latest WSO2 ESB and G-reg product versions.
Managing artifacts of a service oriented solution is one of the most important features expected from any SOA middleware platform. In a typical service oriented project, SOA artifacts are subjected to move through multiple phases. Usually, separate environments are maintained for the activities associated with those phases.
e.g:-
Service development and system integration tasks are carried out in a separate physical environment. Depending on the requirements, there can be multiple SOA middleware solutions used in development environment to facilitate service development, integration and deployment processes.Once the development tasks are completed, SOA artifacts are transferred into QA environment for QA verification.
Solution testing is done in an independent environment which is usually identical to the production settings. Various functional and non-functional tests are performed in QA environment. Upon successful QA verification, the artifacts are moved to production or staging environment
When the service oriented solution becomes complex and there are large number of SOA artifacts, automated artifact governance mechanism is required to ensure smooth transition between various environments. In this post, I will take you through a simplified SOA artifact governance process using WSO2 SOA middleware stack. We will explore a use case similar to the following.
Separate development and QA environments are used to maintain the SOA artifacts. For example, solution developers implement various configuration artifacts in ESB development node. After completing those development tasks, solution developers move those artifacts in to QA environment which allows the testers to carryout QA activities in an independent environment. The moving of artifacts will be done automatically using the features provided by central SOA governance solution.
In this example, we will use
WSO2 ESB as the enterprise service bus middleware,
WSO2 Governance Registry as the SOA governance solution.
Lets go through each of the steps in detail. We need to have 2 WSO2 ESB instances and 1 WSO2 G-reg instance which runs on mySQL. All these instances will be deployed in single host.
Setting up G-reg and ESB productsStep 1Download the latest versions of WSO2 ESB and WSO2 G-reg from
here. First we will configure WSO2 G-reg instance. Extract wso2greg-4.x.x.zip into your file system. We will refer to the extracted location as GREG_HOME.
By default, WSO2 products run on the file based embedded H2 database. Since we need two ESB instances connect to the database used by G-reg instance, we will configure G-reg to run on a mySQL database. Enter the following commands to create a database and grant permission to a user.
mysql> create database greg_db;
Query OK, 1 row affected (0.00 sec)
mysql> use greg_db;
Database changed
mysql> grant all on greg_db.* to regadmin@localhost identified by "regadmin";
Query OK, 0 rows affected (0.05 sec)
Step 2Now, we need to change the default database configuration in G-reg instance through registry.xml configuration file. Open GREG_HOME/repository/conf/registry.xml and update dbConfig element as shown below.
<dbConfig name="wso2registry">
<url>jdbc:mysql://localhost:3306/greg_db</url>
<userName>regadmin</userName>
<password>regadmin</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<minIdle>5</minIdle>
</dbConfig>
Copy
mysql jdbc driver to GREG_HOME/repository/components/lib and start G-reg server by running wso2server.sh
Step 3We have completed the configurations of WSO2 Governance Registry instance. Lets move forward with setting up ESB instance in development environment.
Extract the downloaded wso2esb-4.x.x.zip into a directory in your file system. We will refer to this directory as ESBDEV_HOME. Since we are running all product instances in a single machine, we need to start them up in different ports. Therefore, change port offset parameter in ESBDEV_HOME/repository/conf/carbon.xml as follows.
<Offset>1</Offset>
Step 4As shown in the above diagram, we will mount /_system/config collection, where the ESB artifacts are stored in to /_system/dev collection of G-reg instance. ESBDEV_HOME/repository/conf/registry.xml is used to configure that.
<dbConfig name="configgovregistry">
<url>jdbc:mysql://localhost:3306/greg_db</url>
<userName>regadmin</userName>
<password>regadmin</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<minIdle>5</minIdle>
</dbConfig>
<remoteInstance url="https://localhost:9443/registry">
<id>configgov</id>
<dbConfig>configgovregistry</dbConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/config" overwrite="true">
<instanceId>configgov</instanceId>
<targetPath>/_system/dev</targetPath>
</mount>
<mount path="/_system/governance" overwrite="true">
<instanceId>configgov</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>
Start ESB dev instance. If you log in to management console of the ESB instance (https://localhost:9444/carbon) and access registry browser, you will notice the config and governance collections are mounted (See the arrow icons) to the remote G-reg database.
Step 5Repeat step 3 and 4 with another copy of wso2esb-4.x.x.zip. Lets call it ESBQA instance. (The root directory of ESBQA instance will be referred to as ESBQA_HOME). Make sure to specify a different offset value in carbon.xml. In this instance, we will mount /_system/config collection to /_system/qa collection of G-reg instance. Update registry.xml of ESBQA node as follows.
<dbConfig name="configgovregistry">
<url>jdbc:mysql://localhost:3306/greg_db</url>
<userName>regadmin</userName>
<password>regadmin</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<minIdle>5</minIdle>
</dbConfig>
<remoteInstance url="https://localhost:9443/registry">
<id>configgov</id>
<dbConfig>configgovregistry</dbConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/config" overwrite="true">
<instanceId>configgov</instanceId>
<targetPath>/_system/qa</targetPath>
</mount>
<mount path="/_system/governance" overwrite="true">
<instanceId>configgov</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>
Start the server. We are done with the product configurations. However, we have not done any artifact governance configuration yet. In the next steps, we will look into moving artifacts between the two ESB instances.
Using WSO2 Governance Registry to move artifacts between environmentsAs I explained at the beginning, we use central governance registry instance to manage the artifacts produced by each of DEV and QA environments. The ESB artifacts produced by development environment are stored under /_system/dev collection of governance registry where as the artifacts used in QA environment are stored in /_system/qa collection. When the development tasks are compelte and ready for QA, we can manually copy the artifacts into the relevant locations and configure the QA environment. Obviously it will be a painful task to copy large number of artifacts into various locations of ESB QA environment by hand. Therefore, we need some kind of automated artifact copying mechanism.
WSO2 Governance Registry provides us with registry extension features to extend the core functionality of G-reg to use in these types of situations. The
lifecycle management feature is such a useful extension provided by WSO2 Governance Registry which can be used to manage life cycle of a resource stored in registry.
The standard WSO2 G-reg distribution is shipped with a default ServiceLifeCycle which can be used to move service artifacts among different environments. We will modify the default ServiceLifeCycle to copy artifacts stored under /_system/dev collection to /_system/qa when promoting from development stage to testing.
Log in to G-reg management console and navigate to
Extensions --> Configure --> Lifecycles. Click on Add New LifeCycle and add the following life cycle.
<aspect name="ESBLifeCycle" class="org.wso2.carbon.governance.registry.extensions.aspects.DefaultLifeCycle">
<configuration type="literal">
<lifecycle>
<scxml xmlns="http://www.w3.org/2005/07/scxml"
version="1.0"
initialstate="Development">
<state id="Development">
<datamodel>
<data name="checkItems">
<item name="Configurations Completed" forEvent="">
</item>
<item name="Transform Rules Done" forEvent="">
</item>
<item name="Routing Rules Completed" forEvent="">
</item>
</data>
<data name="transitionExecution">
<execution forEvent="Promote" class="org.wso2.carbon.governance.registry.extensions.executors.CopyExecutor">
<parameter name="currentEnvironment" value="/_system/dev"/>
<parameter name="targetEnvironment" value="/_system/qa"/>
</execution>
</data>
<data name="transitionUI">
<ui forEvent="Promote" href="../lifecycles/pre_invoke_aspect_ajaxprocessor.jsp?currentEnvironment=/_system/dev/"/>
</data>
</datamodel>
<transition event="Promote" target="Testing"/>
</state>
<state id="Testing">
<transition event="Demote" target="Development"/>
</state>
</scxml>
</lifecycle>
</configuration>
</aspect>
You can see that we have used
org.wso2.carbon.governance.registry.extensions.executors.CopyExecutor class to copy one or more resources from one environment to another. Executors are custom extensions to G-reg which used to trigger a custom execution logic at the time of state transition (e.g:- Dev to QA, QA to Production). When artifacts are promoted to Testing state from Development, the CopyExecutor is triggered and the events defined under transitionExecution element are carried out. In this lifecycle, CopyExecutor copies any resource which is at
currentEnvironment to
targetEnvironment
Note that, CopyExecutor has not been shipped by default in WSO2 G-reg-4.1.X versions. Hence, you need to implement org.wso2.carbon.governance.registry.extensions.interfaces.Execution interface. However, in current trunk based versions (which will be released within few months), this executor is included by default and you can use it out-of-the-box.
Once the life cycle is created and saved, we can apply it to the relevant collection in registry. In this example, we need to move the artifacts stored at /_system/dev to /_system/qa. Hence, we must apply
ESBLifecycle to /_system/dev collection.
Navigate to /_system/dev collection in registry browser. Click on
Lifecycle in the left pane. Select Add Lifecycle and choose ESBLifeCycle from the dropdown list. Click on Add to add ESBLifeCycle to /_system/dev collection.
We do not have any artifacts in /_system/dev collection yet. Therefore, lets add some ESB artifacts through ESB management console. Log in to the management console of ESB DEV instance and add a sequence. (say it is "devsequence). This will create a new sequence in ESB and it will be stored under /_system/dev in G-reg
Now, go back to G-reg management console and navigate to /_system/dev collection. Select the check-list items such as Configurations Completed, Transform Rules Defined etc.. and click on Promote.
After promotion, click on Search --> metadata in left menu in G-reg management console and search for the resource which has just been added ("devsequence"). The resource will be copied to /_system/qa collection.
We have just completed automatic promotion of resources which have been created from ESB DEV environment to ESB QA environment. However, you will notice that the ESB sequence artifact which has just been added, is not "deployed" in ESB QA instance. Because of that, you will not see "devsequence" in sequence list of ESB QA instance. The artifact is added to the relevant registry location but in order for deployment of the resource, ESB needs to check-out the resource from
/_system/qa/repository/deployment/server/synapse-configs/default/sequences location in registry to the corresponding deployment directory of file system of ESB QA server.
DeploymentSynchronizer comes into action in this situation. As I have explained in a previous
blog post, DeploymentSynchronizer can be used to synchronize deployment artifacts among cluster nodes.
Lets enable registry based deployment synchronizer for ESB DEV and ESB QA instances.
- Shutdown both ESB DEV and ESB QA servers
- From WSO2 Carbon-4.0.0 release onwards, we need to enable Tribes based clustering in order for deployment synchronizer to communicate with cluster nodes. Open ESBDEV_HOME/repository/conf/axis2/axis2.xml and enable clustering and specify a unique domain name.
<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
<parameter name="domain">wso2.charitha.domain</parameter>
Repeat the same with ESBQA_HOME/repository/conf/axis2/axis2.xml as well.
- Enable deployment synchronizer configuration in ESBDEV_HOME/repository/conf/carbon.xml
<DeploymentSynchronizer>
<Enabled>true</Enabled>
<AutoCommit>true</AutoCommit>
<AutoCheckout>true</AutoCheckout>
</DeploymentSynchronizer>
- Similarly, Enable deployment synchronizer configuration in ESBQA_HOME/repository/conf/carbon.xml. ESB QA instance automatically checks out the resources from /_system/qa collection of G-reg. Therefore, we just enable AutoCheckout in that node.
<DeploymentSynchronizer>
<Enabled>true</Enabled>
<AutoCommit>false</AutoCommit>
<AutoCheckout>true</AutoCheckout>
</DeploymentSynchronizer>
We have completed enabling deployment synchronizer for both ESB development and QA servers. Now, try out adding another sequence from ESB DEV instance (say it is "devsequence2"). Once the sequence is added, access G-reg management console and browse /_system/dev collection. Promote the lifecycle again. This time, you will notice that devsequence2 is deployed in ESB QA instance upon successful promotion
In this post, I took you through the steps of moving artifacts between different environments using WSO2 ESB and WSO2 G-reg. We created artifacts in ESB using management console. However, the recommended best practice is to build SOA artifacts using a tool like WSO2 Carbon Studio and upload the CAR artifacts to ESB. We will look into CAR based SOA governance mechanism in a future post.
