IT-eye

Business Intelligence is all about data and the meaning of that data. Terms like metadata, master data, reference data and transaction data are very common in a Business Intelligence project and data warehouse projects and referred as data categories. 

Before we can identify the meaning of our data in our data warehouse / business intelligence project, we need to understand the meaning all the data categories we have mentioned above.
 

Meta data
Meta data is data about the data meaning. It can contain details about the structure of database tables and objects, but also information on how data is extracted, transformed and loaded from source to target. It can also contain information of the origin of the data.
 

Master data versus Reference data
Master data and reference data are two major categories of data that are often thought of as the same, but in reality they are quite different. Even though they have strong dependencies on each other.

Master data is the data without which you cannot do any transactions and is mandatory for every organization. It describes the things that interact when a transaction occurs. For instance, master data that represents product and customer must be present before the transaction is fired to sell a product to a customer.

Reference data is any kind of data that is used solely to categorize other data found in a database, or solely for relating data in a database to information beyond the boundaries of the enterprise. Volumes of reference data are much lower than what is involved in master data and reference data changes more slowly than master data. An example of reference data is the use of code tables, which consists of codes and/or acronyms, a description and little else. The code tables hold information about product line, gender, country or customer type ect.

Transaction data
These are the business documents that you create using the master data like purchase orders, sales orders etc. Transactional Data can change very often and are not constant. And this data is created/modified out of an application transaction.

Well I hope that clarifies it.

“A small impression”
This November 22 and November 23 the Dutch Architectural Congress was held. A couple of colleagues of mine, including me represented IT-eye. This edition of the congress was special for us, because we were nominated for the Dutch Championship for ICT Architecture with our SOA case. Our IT architect Mike van Alst is responsible for SOA solution IT-eye implemented successfully for our customer. Mike wrote the architectural roadmap to SOA and presented it to the Dutch ICT Architecture board as well.

Together with nine other contributions IT-eye WON the Dutch Championship for ICT Architecture yesterday. It is a great honour for us to win this price and it is a confirmation as well that we are doing the right thing with ICT Architecture in combination with SOA.
In addition to the Dutch Championship for ICT Architecture the congress was very interesting. In two days I have heard a lot of information about ICT architectures, Enterprise Architectures, SOA, business rules etc.
The congress started with a lecture by Rob Cremer about trends in ICT and the speed of changes. It’s amazing to hear what today’s technology already can and will do in the (near) future. A lot of presentations hooked up on these topics and threw more light on different aspects.

For example Nokia gave a presentation about their architecture of the current and future cell-phone and The Open Group discussed TOGAF. A nice lecture was given by Prof. Frank van Harmelen, (VU Amsterdam) about the Semantic Web.

The second day I joined different SOA lectures. Some gave an impression of an SOA case which has been implemented for a customer, but, in my opinion, the most interesting lecture was about Value patterns and anti-patterns for SOA. It explained some patterns how to substantiate an SOA in a business case from a financial point of view. Furthermore which roles an IT Architect must fulfill and how to act within an organization to achieve or maintain the Enterprise Architecture. A couple of tips and tricks were given for this particular case.
Summarizing I can say that the two days congress brought me a lot of information, new topics on IT architecture and useful feedback which I can use in my projects directly. And of course, the cream on the pie, IT-eye won the Dutch Championship for ICT Architecture!

I’ve just been browsing through some Oracle Openworld presentations on SOA and BPEL, and found some interesting info in a presentation about Oracle’s SOA Suite 11G. One of the big improvements in 11g will be on monitoring, aka Business Activity Monitoring (BAM). BAM will enable you to monitor events on all your business processes.

This means that the system has to be able to handle and query data from a large number of events. You’ll have a continuous stream of event data. This is where Continuous Query Language comes in. CQL is like a SQL query with additional expressions describing how to filter the data before it’s stored.

Here’s an example:
select w.location
, avg(w.water_level)
from water_level_stream w
[range by '10 minutes' slide by '10 minutes']
where w.state = 'zeeland'
group by w.location


From a continuous stream of water level data this query would calculate a water level average for every 10 minutes.

The main reason for implementing a Service Oriented Architecture and BPEL is to enable organizations to adapt to their changing environment. Today more and more organizations experience that their current monolithic architecture restricts them in developing new business opportunities. Therefore, the impact of IT on their business is big. 

In a current SOA implementation project one of the objectives was to simplify and optimize processes. In the design phase we used different techniques to gather requirements. Our main focus was to get input for developing BPEL processes and identifying services. 

During the project we found that the mindset of the stakeholders was case oriented instead of process oriented.  In case orientation, participants only think about the uniqueness of the case and all exceptions. In the process orientation the participants focus on the main process and try to map every case. Decisions have to be made for cases that not fit in the main process. Process orientation is needed to simplify and optimize processes. 

In the next phase (new processes) of the project we organized 4 workshops and used brown paper technique to design the process.  In a brown paper session a big brown paper is stuck on the wall. Participants use a rectangle piece of paper to represent a process step and paper in the shape of diamond is used to represent a decision. Participants themselves actively create the process by discussing and change the order of symbols. Our role was to facilitate the process and check if all participants agree on the design.

During the workshops the case orientation surfaced and blocked the workshop process.  We decided to use intervention to get commitment to the process orientation. After this intervention the workshop accelerated. Nine proposals for simplifying process were produced. More and more participants adapted the process orientation.  In meetings we started to use metaphors tree (case orientation) and forest (process orientation), to show the participants mindset.

The use of interventions, metaphors, brown paper sessions are not in the standard repertoire of requirements engineering. You will find them in the repertoire of organization change management. This is why we are now training our consultants change management skills. With this business centric approach, instead of information centric or technology centric approach, our SOA implementation will add value to customers business.

One of the most annoying problems with building dynamic websites is how to control page cache.

The most used solution is to use a <meta content=”no-cache” http-equiv=”Pragma”/> tag inside the <HEAD> tag. Voilà, problem solved. Wrong! For two reasons:
1. It’s HTML, (most) proxy servers will still cache your pages since they don’t read HTML
2. The HTTP specification does not set any guidelines for Pragma response headers; instead, Pragma request headers are discussed.
Although a few caches may honor this header, the majority won’t, and it won’t have any effect.

However, true HTTP headers are almost always obeyed by any cache. With ADF Faces / JSF it is quite easy to gain control over your caching strategies with a phaselistener:


package nl.iteye.utils;
import javax.faces.context.FacesContext;
import javax.faces.event.PhaseEvent;
import javax.faces.event.PhaseId;
import javax.faces.event.PhaseListener;
import javax.servlet.http.HttpServletResponse;

public class NoCachePhaseListener implements PhaseListener {

    public PhaseId getPhaseId() {
        return PhaseId.RENDER_RESPONSE;
    }

    public void afterPhase(PhaseEvent phaseEvent) {
    }

    public void beforePhase(PhaseEvent phaseEvent) {
        FacesContext facesContext = phaseEvent.getFacesContext();
        HttpServletResponse response = (HttpServletResponse) facesContext.getExternalContext().getResponse();
        response.addHeader(“Pragma”, “no-cache”);
        response.addHeader(“Cache-Control”, “no-cache”);
        response.addHeader(“Cache-Control”, “no-store”);
        response.addHeader(“Cache-Control”, “must-revalidate”);
        response.addHeader(“Expires”, “Mon, 1 Jan 2006 05:00:00 GMT”);//in the past
    }
}

And don’t forget to add the viewhandler to the faces-config.xml within the <lifecycle> tag : <phase-listener>nl.iteye.utils.NoCachePhaseListener</phase-listener> to make this magic work!

I’ve mentioned this before in this blog, but i’ll say it again. If you do web development there are two firefox plugins you cannot live without: webdeveloper and firebug. I just read a post on Ajaxian about firebug 1.0: GetFirebug: 1.0 is getting close. Firebug 1.0 is going to be good. Some of it’s features are mentioned on the getfirebug website. Have a look a the pages for monitor network activity, css layout, javascript profiling. Very nice.

When running serveral mappings we get the following warning: ORA-12855: cannot run parallel or insert direct load in a loopback. This error only occurs when running in Set_based mode.
When switch back to Row_based mode the mappings run fine.

It took me a while to find out what the meaning of this error is and how to prevent this error to occur. Now I found the solution I wanted to share it with you all.

Situation: The mappings that got this warning all had an database link to the source tables and were running in parallel dml mode. Using UPDATE/INSERT target loading type. The database link is pointing to a different schema then the target but is located within the same database instance.

Solution: Apparently when running in PDML mode it is not allowed to use database links to the same database instance.
The solution lies in the configuration settings where you need to remove the database link name and place the schema name in the objects that are refering to them.

Thought: Why using database links within the same database instance?
Well, Oracle Warehouse Builder generates them for each source module in your project by default. That’s why.
But by my experience never use database links which are referring to different schema’s in the same database instance. Even though it seems to save you time in configering all your mappings, you’ll better use schema names in the configuration settings.

Does it take a lot of time to change? … Not if you are using a script like below.

puts -nonewline "CONNECT TO orclserver AS 'mngr_owb' USE REPOSITORY 'MNGR_OWB' .. "
OMBDISC
OMBCONNECT mngr_owb/owb@orclserver:1521:orcl USE REPOSITORY 'MNGR_OWB'
puts "DONE"
# Vraag de datum op; Deze wordt verwerkt in het bestandsnaam.
set v_datum [clock format [clock seconds] -format "%Y%m%d_%H%M%S"]
set fname [ open "C:\Temp\databaselink1_$v_datum.log" w ]
set project "DATAWAREHOUSE"
set moduleName "01_DATAWAREHOUSE"
puts -nonewline "GO TO $project / $moduleName .. "
OMBCC '$project'
OMBCC '$moduleName'
puts "DONE"
puts -nonewline "GET MAPPING LIST OF $project / $moduleName .. "
set mappingList [ OMBLIST MAPPINGS 'T_IBIS.*']
puts "DONE"
foreach mappingName $mappingList {
set tabelList [ OMBRETRIEVE MAPPING '/$project/$moduleName/$mappingName' GET TABLE OPERATORS ]
foreach tableName $tabelList {
set tableDBL [ OMBRETRIEVE MAPPING '/$project/$moduleName/$mappingName' OPERATOR '$tableName' GET PROPERTIES(DATABASE_LINK) ]
puts "$mappingName - $tableName - $tableDBL"
puts $fname "$mappingName - $tableName - $tableDBLr"
if { $tableDBL != "{}" } {
puts -nonewline " DELETE DATABASELINK AND ADD SCHEMA NAME OF $tableName .. "
OMBALTER MAPPING '$mappingName'
MODIFY OPERATOR '$tableName' SET PROPERTIES (DATABASE_LINK, SCHEMA) VALUES ('', 'MNGR_SA')
puts "DONE"
puts $fname " DELETE DATABASELINK AND ADD SCHEMA NAME OF $tableName .. DONEr"
}
}
}
puts -nonewline "COMMITTING .. "
OMBCOMMIT
puts "DONE"
OMBDISCONNECT

Good luck.

This monday, the new stable re-designed and re-written Apache SOAP stack is released, Axis 2.0 version 1.1. Axis2.0 is a big improvement in respect to its predecessor Axis1.0.
The modular-architecture of Axis2.0 contains build-in support for various WS-* standards and the Stax-based AXIOM object model for processing XML documents improves the performance a lot.

The Axis2.0 framework als contains a renewed client API. Where Axis1.0 only support blocking invocation models (synchronious calls), Axis2.0 now supports both blocking and nonblocking invocation models (asynchronous). The latter is handy when you have to make service calls that take a considerable amount of time to reponse, so that you don’t want to wait on it or prevent time-out errors.

As a developer you also have the abillity to select a specific transport channel for your purpose:

• Send a message using HTTP and receive the response using the same HTTP channel (use of one dual-channel transport)
• Send a message using HTTP and receive the response using a different HTTP channel (two dual-channel transports)
• Send a message using SMTP and receive the response using SMTP (two different transports)

It even contains Spring support enabling you to expose your beans directly as Axis2.0 services.

Despite all the benefits, it still has some limitations. Axis2.0 does not has support yet for rpc/encoded style WSDL documents (in stub generation). This limitation restricts me to use Axis1.0 as the WS client API on my current SOA project in which we are stucked to rpc/encoded services.

Knowing that Axis2.0 has great benefits in its use, I still miss in my baggage a well-defined overview and comparison of the available webservice frameworks out there these days . I have to do more on-line research to gain it, I guess

Communication nation has a funny sketch which exactly describes a situation i ran into last week. For the last year or so I’ve been trying to improve my presentations by getting rid of most bulletpoints. Many people have argued that long lists of bulletpoints don’t work on slides. Your audience will read you slides and be too distracted to listen to your talk. Triggered by Tufte’s rant on powerpoint The cognitive style of powerpoint, i’ve read beyond bullets, Presentation Zen, Really BAD PowerPoint, and others. The lesson is, less text is better, usually only a title and an image is enough. Similar to a documentary on television, use the visuals to add to your talk, not to display your words.

Anyway, last week after a 2 hours talk i got a comment from someone in the audience that i should put more words on my slides. Because my slides had so little text, he had been continuously busy taking notes. Guess I’ll have to include some notes on my handouts next time. Should i distribute the handouts before or after the talk? If i give them before the talk, and they’ll start reading the slides during the talk, i might as well put the words on the slides. But if i pass out the handouts after the talk, they won’t be able to add their own notes to the handouts.

Update:
There’s a lot of info from beyond bullets on microsoft’s office website: making presentations. I like this one: 10 tips to enhance your presentations. The first tip is not to include your logo on every slide, it just distracts. Instead you can add your logo on the handouts, in case you’re afraid people will not remember you or your company name.

If you use Correlation, the easiest way to do so, is to open the invoke or receive you want to create the correlation on. Go to the Correlation tab, create a new correlation, create the corresponding correlationSet and the appropriate property-aliases. That should show a screen like:

Another way is to create the aliases by hand. Go to your structure pane in JDeveloper, select the Properties and click ‘Add’. It will show:

If you do it the ‘regular’ way …… it will NOT work. Why? Because when the JDeveloper wizard creates your aliases, it misses one important part, being the ‘part’ of the message where the correlation field(s) are located. Below you’ll see a screenshot of two aliases. The first one I made using the wizard, the second I made by hand.

It’s easy to see what’s missing. There’s no reference to the messagepart. The BPEL will compile and deploy normally. However, when you try to execute it, the correlation will fail, due to the fact that the correlation-alias references a not existing field. You’ll see the following message:

[javax.servlet.ServletException]
Failed to evaluate correlation query.
Failed to evaluate the correlationAlias query "/client:correlationRequest/client:input"
on the element "null".
Please check your BPEL/WSDL source to make sure that property alias is defined correctly.So my advice to you all is to define the property-aliases by hand!