What is new for DataStage 8 on the Information Server

 

DataStage 8 on the Information Server looks the same as previousreleases but has some major changes under the hood and a few extrabells and whistles.

This post looks at what is new or changed in DataStage jobs. Thereare a lot of new functions for managing, running and reporting on jobsbut I will talk about that in another post or you can look back at my(much) earlier DataStage Hawk preview post.

Goodbye DataStage 7

It's time to bid goodbye to tired old DataStage 7.

You did a good job, you struggled on for as long as you could, butlike all DataStage versions through the annuls of history you didn'thave the right metadata repository and you didn't play well with yourbrothers and sisters in your suite.

DataStage 8 on the other hand is much shinier and comes with abetter metadata story as you get the new Metadata Server and the commonconnectors:

Release Date

The Windows version of the Information Server and DataStage 8 are out now. No sign yet of the version for other platforms.

DataStage Versions

DataStage 8 can only upgrade a DataStage 7 server, it cannot upgradeprevious versions of servers though it can co-exist with previousversions. DataStage 8 can however import and upgrade export files fromearlier versions of DataStage. I don't know how far back this supportgoes.

All the DataStage 7.x versions are available in version 8:

• DataStage Enterprise Edition: Parallel, Server and Sequence Jobs
• DataStage Server Edition: Server and Sequence Jobs
• DataStage MVS: Mainframe Jobs
• DataStage Enterprise for z/OS: runs on Unix System Services

DataStage for PeopleSoft: 2 CPU limit with Server and Sequence jobs.

I don't know whether you will ever seethis version of DataStage in the PeopleSoft EPM bundle, however you maybe able to upgrade existing PeopleSoft implementations to this version.Drop me a message if you try.

DataStage Addons

The DataStage Enterprise Packs and Change Data Capture componentsare available in version 8 as shown in the version 8 architectureoverview:

Enterprise PACKs

• SAP BW Pack
  • BAPI: (Staging Business API) loads from any source to BW.
  • OpenHub: extract data from BW.
• SAP R/3 Pack
  • ABAP: (Advanced Business Application Processing) auto generateABAP, Extraction Object Builder, SQL Builder, Load and execute ABAPfrom DataStage, CPI-C Data Transfer, FTP Data Transfer, ABAP syntaxcheck, background execution of ABAP.
  • IDoc: create source system, IDoc listener for extract, receive IDocs, send IDocs.
  • BAPI: BAPI explorer, import export Tables Parameters Activation, call and commit BAPI.
• Siebel Pack
  • EIM: (data integration manager) interface tables
  • Business Component: access business views via Siebel Java Data Bean
  • Direct Access: use a metadata browser to select data to extract
  • Hierarchy: for extracts from Siebel to SAP BW.
• Oracle Applications Pack
  • Oracle flex fields: extract using enhanced processing techniques.
  • Oracle reference data structures: simplified access using the Hierarchy Access component.
  • Metadata browser and importer
• DataStage Pack for PeopleSoft Enterprise
  • Import business metadata via a metadata browser.
  • Extract data from PeopleSoft tables and trees.
• JD Edwards Pack
  • Standard ODBC calls
  • Pre-joined database tables via business views

Change Data Capture

These are add on products (at an additional fee) that attachthemselves to source databases and perform change data capture. Mostsource system database owners I've come across don't like you playingwith their production transactional database and will not let you nearit with a ten foot poll, but I guess there are exceptions:

• Oracle
• Microsoft SQL Server
• DB2 for z/OS
• IMS

There are three ways to get incremental feeds on the Information Server: the CDC products for DataStage, the Replication Server (renamed Information Integrator: Replication Edition, does DB2 replication very well) and the change data capture functions within DataStage jobs such as the parallel CDC stage.

Removed Functions

These are the functions that are not in DataStage 8, please imaging the last waltz playing in your head as you peruse this list:

• dssearch command line function
• dsjob "-import"
• Version Control tool
• Released jobs
• Oracle 8i native database stages
• ClickPack

The loss of the Version Control tool isnot a big deal as the import/export functions have been improved.Building a release file as an export in version 8 is easier thanbuilding it in the Version Control tool in version 7.

Database Connectivity

The common connection objects functionality means the very widerange of DataStage database connections are now available acrossInformation Server products.

Latest supported databases for version 8:

• DB2 8.1, 8.2 and 9.1
• Oracle 9i, 10i, 10gR2 not Oracle 8
• SQL Server 2005 plus stored procedures.
• Teradata v2r5.1, v2r6.0, v2r6.1 (DB server) / 8.1 (TTU) plus Teradata Parallel Transport (TPT) and stored procedures and macro support, reject links for bulk loads, restart capability for parallel bulk loads.
• Sybase ASE 15, Sybase IQ 11.5, 12.5, 12.7
• Informix 10 (IDS)
• SAS 612, 8.1, 9.1 and 9.1.3
• IBM WS MQ 6.1, WS MB 5.1
• Netezza v3.1
• ODBC 3.5 standard and level 3 compliant
• UniData 6 and UniVerse ?
• Red Brick ?

This is not the complete list. Somedatabase versions are missing, more databases can be accessed throughthe ODBC stage and there may be some databases missing.

New Database Connector Functions

This is a big area of improvement.

• LOB/BLOC/CLOB Data:pictures, documents etc of any size can now be moved between databases.After choosing the LOB data type you can choose to pass the data inlineor as a link reference.
• Reject Links: optionally append error codes and messages,conditionally filter types of rejection, fail a job based on apercentage threshold of failures.
• Schema Reconciliation:where the hell has this function been all my life? Automaticallycompare your DataStage schema to the database schema and perform minordata type conversions.
• Improved SQL Builder that supports more database types, although ifyou didn't like the version 7 one you wont like the 8 one either. (KimDuke, I'm looking at you).
• Test button on connectors. Test! You don't have to view data or run a job to find out if the stupid thing works.
• Drag and drop your configured database connections onto jobs.
• Before and after SQL defined per job or per node with a failure handling option. Neater than previous versions.

DataStage 8 gives you access to thelatest versions of databases that DataStage 7 may never get. Extrafunctions on all connectors includes improved reject handling, LOBsupport and easier stage configuration.

Code Packs

These packs can be used by server and/or parallel jobs to interactwith other coding languages. This lets you access programming modulesor functions within a job:

• Java Pack: produce or consume rows for DataStage Parallel or Server jobs. Use a java transformer.
• Web Service Pack: access web services operations in a Server job transformer or Server routine.
• XML Pack: read, write or transform XML files in parallel or server jobs.

The DataStage stages, custom stages,transformer functions and routines will usually be faster attransforming data than these packs however they are useful for re-usingexisting code.

New Stages

A new stage from the IBM software family, new stages from newpartners and the convergence of QualityStage functions into Datastage.Apart from the SCD stage these all come at an additional cost.

• WebSphere Federation and Classic Federation
• Netezza Enterprise Stage
• SFTP Enterprise Stage
• iWay Enterprise Stage
• Slowly Changing Dimension: for type 1 and type 2 SCDs.
• Six QualityStage stages

There are four questions thathave been asked since the dawn of time. What is the meaning of life?What's this rash that comes and goes? If you leave me can I come too?How do a populate a slowly changing dimension using DataStage? Theanswers being 42, visit a clinic, piss off and use the new SCD stage.

New Functions Existing Stages

• Complex Flat File Stage: Multi Format File (MFF) in addition to existing cobol file support.
• Surrogate Key Generator: now maintains the key source via integrated state file or DBMS sequence.
• Lookup Stage: range lookups by defining checking high and low range fields on the input or reference data table. Updatable in memory lookups.
• Transformer Stage: new surrogate key functions Initialize() and GetNextKey().
• Enterprise FTP Stage: now choose between ftp and sftp transfer.

You can achieve most of these functionsin the current version with extra coding except for in-memory lookups.This is a killer function in DataStage 8.

Platforms

These are the platforms for the released Windows version and the yetto be released Linux/Unix version along with the C++ compiler that youonly need for parallel jobs that will use transformers. You do not needthis compiler for Server Edition.

-Windows 2003 SP1
•Visual Studio .NET 2003 C++, Visual Studio .NET 2005 C++ or Visual Studio .NET 2005 Express Edition C++
-AIX 5.2 & 5.3
•XL C/C++ Enterprise Edition 7.0, 8.0 compiler
-HP-UX 11i v1 & v2
•aC++ A.03.63 compiler
-Red Hat ASE 4.0
•gcc3.23 compiler
-SuSEES, 9.0
•gcc3.3.3 compiler
-Solaris 2.9 & 2.10
•Sun Studio 9, 10 , 11 compiler

Database Repository

Note the database compatibility for the Metadata Server repositoryis the latest versions of the three DBMS engines. DB2 is an optionalextra in the bundle if you don't want to use an existing database.

• IBM UDB DB2 ESE 9
  -IBM Information Server does not support the Database Partitioning Feature (DPF) for use in the repository layer
  -DB2 Restricted Enterprise Edition 9 is included with IBM InformationServer and is an optional part of the installation however its use isrestricted to hosting the IBM Information Server repository layer andcannot be used for other applications
• Oracle 10g
• SQL Server 2005

If you are a cheapskate and you reallydon't like DB2, in fact you would cross the street if you saw it comingin the other direction, you might be able to load the repository into afree (express) version of SQL Server or Oracle, however you might hit aproblem with the DBMS license CPU restriction. If you get this workingdrop me a comment.

Languages

Foreign language support for the graphical tools and product messages:

Chinese (Simplified and Traditional), Czech, Danish, Finnish,French, German, Italian, Japanese, Korean, Norwegian, Polish,Portuguese, Russian, Spanish and Swedish.

Commiserations to the Welsh. For theTrekkies out there keep the writin campaign going, it is only a matterof time before they add Klingon. It is on the product path right afterHigh Elf.