Essbase Connector Error – Client Commands are Currently Not Being Accepted

Advertisements
DataStage Essbase Connector

While investigating a recent Infosphere Information Server (IIS), Datastage, Essbase Connect error I found the explanations of the probable causes of the error not to be terribly meaningful.  So, now that I have run our error to ground, I thought it might be nice to jot down a quick note of the potential cause of the ‘Client Commands are Currently Not Being Accepted’ error, which I gleaned from the process.

Error Message Id

  • IIS-CONN-ESSBASE-01010

Error Message

An error occurred while processing the request on the server. The error information is 1051544 (message on contacting or from application:[<<DateTimeStamp>>]Local////3544/Error(1013204) Client Commands are Currently Not Being Accepted.

Possible Causes of The Error

This Error is a problem with access to the Essbase object or accessing the security within the Essbase Object.  This can be a result of multiple issues, such as:

  • Object doesn’t exist – The Essbase object didn’t exist in the location specified,
  • Communications – the location is unavailable or cannot be reached,
  • Path Security – Security gets in the way to access the Essbase object location
  • Essbase Security – Security within the Essbase object does not support the user or filter being submitted. Also, the Essbase object security may be corrupted or incomplete.
  • Essbase Object Structure –  the Essbase object was not properly structured to support the filter or the Essbase filter is malformed for the current structure.

Related References

IBM Knowledge Center, InfoSphere Information Server 11.7.0, Connecting to data sources, Enterprise applications, IBM InfoSphere Information Server Pack for Hyperion Essbase

Printable PDF Version of This Article

 

Parallel jobs on Windows fail with APT_IOPort::readBlkVirt;error

Advertisements
APT_IOPort::readBlkVirt Error Screenshot

This a known error for windows systems and applies to DataStage and DataQuality jobs using the any or all the three join type stages (Join, Merge, and Lookup).

Error Message

  • <<Link name>>,0: APT_IOPort::readBlkVirt: read for block header, partition 0, [fd 4], returned -1 with errno 10,054 (Unknown error)

Message ID

  • IIS-DSEE-TFIO-00223

Applies To

  • Windows systems only
  • Parallel Engine Jobs the three join type stages (Join, Merge, and Lookup). It does not apply to Server Engine jobs.
  • Infosphere Information Server (IIS), Datastage and DataQuality 9.1 and higher

The Fix

  • Add the APT_NO_IOCOMM_OPTIMIZATION in project administrator and set to blank or 0. I left it blank so it would not impact other jobs
  • Add the environment variable to the job producing the error and set to 1

What it APT_NO_IOCOMM_OPTIMIZATION Does

  • Sets the use of shared memory as the transport type, rather than using the default sockets transport type.
  • Note that in most cases sockets transport type is faster, so, you likely will not to set this across the project as the default for all job. It is best to apply it as necessary for problematic jobs.

Related References

InfoSphere DataStage and QualityStage, Version 9.1 Job Compatibility

IBM Support, JR54078: PARALLEL JOBS ON WINDOWS FAIL WITH APT_IOPORT::READBLKVIRT; ERROR

IBM Support, Information Server DataStage job fails with unknown error 10,054.

 

DataStage – How to Pass the Invocation ID from one Sequence to another

Advertisements
DataStage Invocation ID Passing Pattern Overview

When you are controlling a chain of sequences in the job stream and taking advantage of reusable (multiple instances) jobs it is useful to be able to pass the Invocation ID from the master controlling sequence and have it passed down and assigned to the job run.  This can easily be done with needing to manual enter the values in each of the sequences, by leveraging the DSJobInvocationId variable.  For this to work:

  • The job must have ‘Allow Multiple Instance’ enabled
  • The Invocation Id must be provided in the Parent sequence must have the Invocation Name entered
  • The receiving child sequence will have the invocation variable entered
  • At runtime, a DataStage invocation id instance of the multi-instance job will generate with its own logs.

Variable Name

  • DSJobInvocationId

Note

This approach allows for the reuse of job and the assignment of meaningful instance extension names, which are managed for a single point of entry in the object tree.

Related References: 

IBM Knowledge Center > InfoSphere Information Server 11.5.0

InfoSphere DataStage and QualityStage > Designing DataStage and QualityStage jobs > Building sequence jobs > Sequence job activities > Job Activity properties

OLTP vs Data Warehousing

Advertisements

OLTP Versus Data Warehousing

I’ve tried to explain the difference between OLTP systems and a Data Warehouse to my managers many times, as I’ve worked at a hospital as a Data Warehouse Manager/data analyst for many years. Why was the list that came from the operational applications different than the one that came from the Data Warehouse? Why couldn’t I just get a list of patients that were laying in the hospital right now from the Data Warehouse? So I explained, and explained again, and explained to another manager, and another. You get the picture.
In this article I will explain this very same thing to you. So you know  how to explain this to your manager. Or, if you are a manager, you might understand what your data analyst can and cannot give you.

OLTP

OLTP stands for OLine Transactional Processing. With other words: getting your data directly from the operational systems to make reports. An operational system is a system that is used for the day to day processes.
For example: When a patient checks in, his or her information gets entered into a Patient Information System. The doctor put scheduled tests, a diagnoses and a treatment plan in there as well. Doctors, nurses and other people working with patients use this system on a daily basis to enter and get detailed information on their patients.
The way the data is stored within operational systems is so the data can be used efficiently by the people working directly on the product, or with the patient in this case.

Data Warehousing

A Data Warehouse is a big database that fills itself with data from operational systems. It is used solely for reporting and analytical purposes. No one uses this data for day to day operations. The beauty of a Data Warehouse is, among others, that you can combine the data from the different operational systems. You can actually combine the number of patients in a department with the number of nurses for example. You can see how far a doctor is behind schedule and find the cause of that by looking at the patients. Does he run late with elderly patients? Is there a particular diagnoses that takes more time? Or does he just oversleep a lot? You can use this information to look at the past, see trends, so you can plan for the future.

The difference between OLTP and Data Warehousing

This is how a Data Warehouse works:

The data gets entered into the operational systems. Then the ETL processes Extract this data from these systems, Transforms the data so it will fit neatly into the Data Warehouse, and then Loads it into the Data Warehouse. After that reports are formed with a reporting tool, from the data that lies in the Data Warehouse.

This is how OLTP works:

Reports are directly made from the data inside the database of the operational systems. Some operational systems come with their own reporting tool, but you can always use a standalone reporting tool to make reports form the operational databases.

Pro’s and Con’s

Data Warehousing

Pro’s:

  • There is no strain on the operational systems during business hours
    • As you can schedule the ETL processes to run during the hours the least amount of people are using the operational system, you won’t disturb the operational processes. And when you need to run a large query, the operational systems won’t be affected, as you are working directly on the Data Warehouse database.
  • Data from different systems can be combined
    • It is possible to combine finance and productivity data for example. As the ETL process transforms the data so it can be combined.
  • Data is optimized for making queries and reports
    • You use different data in reports than you use on a day to day base. A Data Warehouse is built for this. For instance: most Data Warehouses have a separate date table where the weekday, day, month and year is saved. You can make a query to derive the weekday from a date, but that takes processing time. By using a separate table like this you’ll save time and decrease the strain on the database.
  • Data is saved longer than in the source systems
    • The source systems need to have their old records deleted when they are no longer used in the day to day operations. So they get deleted to gain performance.

Con’s:

  • You always look at the past
    • A Data Warehouse is updated once a night, or even just once a week. That means that you never have the latest data. Staying with the hospital example: you never knew how many patients are in the hospital are right now. Or what surgeon didn’t show up on time this morning.
  • You don’t have all the data
    • A Data Warehouse is built for discovering trends, showing the big picture. The little details, the ones not used in trends, get discarded during the ETL process.
  • Data isn’t the same as the data in the source systems
    • Because the data is older than those of the source systems it will always be a little different. But also because of the Transformation step in the ETL process, data will be a little different. It doesn’t mean one or the other is wrong. It’s just a different way of looking at the data. For example: the Data Warehouse at the hospital excluded all transactions that were marked as cancelled. If you try to get the same reports from both systems, and don’t exclude the cancelled transactions in the source system, you’ll get different results.

online transactional processing (OLTP)

Pro’s

  • You get real time data
    • If someone is entering a new record now, you’ll see it right away in your report. No delays.
  • You’ve got all the details
    • You have access to all the details that the employees have entered into the system. No grouping, no skipping records, just all the raw data that’s available.

Con’s

  • You are putting strain on an application during business hours.
    • When you are making a large query, you can take processing space that would otherwise be available to the people that need to work with this system for their day to day operations. And if you make an error, by for instance forgetting to put a date filter on your query, you could even bring the system down so no one can use it anymore.
  • You can’t compare the data with data from other sources.
    • Even when the systems are similar. Like an HR system and a payroll system that use each other to work. Data is always going to be different because it is granulated on a different level, or not all data is relevant for both systems.
  • You don’t have access to old data
    • To keep the applications at peak performance, old data, that’s irrelevant to day to day operations is deleted.
  • Data is optimized to suit day to day operations
    • And not for report making. This means you’ll have to get creative with your queries to get the data you need.

So what method should you use?

That all depends on what you need at that moment. If you need detailed information about things that are happening now, you should use OLTP.
If you are looking for trends, or insights on a higher level, you should use a Data Warehouse.

 Related References

Oracle – How to get a list of user permission grants

Advertisements

Since the Infosphere, information server, repository, has to be installed manually with the scripts provided in the IBM software, sometimes you run into difficulties. So, here’s a quick script, which I have found useful in the past to identify user permissions for the IAUSER on Oracle database’s to help rundown discrepancies in user permissions.

SELECT *

FROM ALL_TAB_PRIVS

WHERE  GRANTEE = ‘iauser’

If we cannot run against the ALL_TAB_PRIVS view, then we can try the ALL_TAB_PRIVS view:

SELECT *

FROM USER_TAB_PRIVS

WHERE  GRANTEE = ‘iauser’

Related References

oracle help Center > Database Reference > ALL_TAB_PRIVS view

What are the Core Capability of Infosphere Information Server?

Advertisements

Three Core Capabilities of Information Server

InfoSphere Information Server (IIS) has three core capabilities:

  • Information Governance
  • Data Integration
  • Data Quality

What the Core Capabilities Provide

The three-core capability translate in to the high-level business processes:

Information Governance – Understand and collaborate

Provides a centrally managed repository and approach, which provides:

  • Information blueprints
  • Relationship discovery across data sources
  • Information technology (IT)-to-business mapping

Data Integration – Transform and deliver

A data integration capability, which provides:

  • Transformation
    • Massive scalability
    • Power for any complexity
    • Total traceability
  • Delivery
    • Data capture at any time
    • Delivery anywhere
    • Big data readiness

Data Quality – Cleanse and monitor

To turn data assets into trusted information:

  • Analysis & validation
  • Data cleansing
  • Data quality rules & management

Related References

IBM Knowledge Center, InfoSphere Information Server Version 11.5.0

Overview of IBM InfoSphere Information Server, Introduction to InfoSphere Information Server

SFDC – Using a timestamp literal in a where clause

Advertisements

Salesforce Connector

Working with timestamp literals in the Infosphere SFDC Connector soql is much like working date literals.  So, here a quick example which may save you some time.

SOQL Timestamp String Literals Where Clause Rules

Basically, the timestamp pattern is straight forward and like the process for dates, but there are some differences. The basic rules are for a soql where clause:

  • No quotes
  • No functions
  • No Casting function, or casting for the where soql where clause to read it
  • It only applies to datetime fields
  • A Timestamp identifier ‘T’
  • And the ISO 1806 time notations

Example SOQL Timestamp String Literals

So, here are a couple of timestamp string literal examples in SQL:

  • 1901-01-01T00:00:00-00:00
  • 2016-01-31T00:00:00-00:00
  • 9999-10-31T00:00:00-00:00

Example SQL with Timestamp String Literal Where Clause

 

Select e.Id,

e.AccountId,

e.StartDateTime

From Event e

WHERE e.StartDateTime > 2014-10-31T00:00:00-00:00

 

Related References

Salesforce Developer Documentation

Home, Developer Documentation, Force.com SOQL and SOSL Reference

https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_soql_select_dateformats.htm

Salesforce Workbench

Home, Technical Library, Workbench

W3C

Date Time Formats

 

SFDC Salesforce Connector – Column Returns Null values, when SOQL Returns Data in Workbench

Advertisements

Salesforce Connector

Recently, encountered a scenario, which is a little out of the norm while using the SFDC Connector.  Once the issue is understood, it is easily remedied.

The problem / Error

  • SOQL run in Salesforce workbench and column returns data
  • The DataStage job/ETL runs without errors or warnings
  • The target column output only returns null values

The Cause

In short the cause is a misalignment between the SOQL field name and the column name in the columns tab of the connector.

The Solution

The fix is simply to convert the dots in the field name to underscores.   Basically, a field name on SOQL of Account.RecordType.Name becomes Account_RecordType_Name.

Example Field / Column Name  Fix

Example SQL

Select c.Id,

c.AccountId,

c.CV_Account_Number__c,

c.Name,

c.Role__c,

c.Status__c,

c.Account.RecordType.Name

From Contact c

Columns Tab With Correct Naming Alignment

Please note that the qualifying dots have been converted to underscores.

SFDC Connector Columns Tab

Related References

 

SFDC – Using a date literal in a where clause

Advertisements

I found working with date literal, when working with the Infosphere SFDC Connector soql, to be counterintuitive for me.  At least as I, normally, as I use SQL.  I spent a little time running trials in Workbench, before I finally locked on to the ‘where clause’ criteria data pattern.  So, here a quick example.

SOQL DATE String Literals Where Clause Rules

Basically, the date pattern is straight forward. The basic rules are for a soql where clause:

  • No quotes
  • No functions
  • No Casting function, or casting for the where soql where clause to read.

Example SOQL DATE String Literals

So, here are a couple of date string literal examples in SQL:

  • 1901-01-01
  • 2016-01-31
  • 9999-10-31

Example SQL with Date String Literal Where Clause

 

Select

t.id,

t.Name,

t.Target_Date__c,

t.User_Active__c

From Target_and_Segmentation__c t

where t.Target_Date__c > 2014-10-31

 

Related References

Salesforce Developer Documentation

Home, Developer Documentation, Force.com SOQL and SOSL Reference

https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_soql_select_dateformats.htm

Salesforce Workbench

Home, Technical Library, Workbench

 

Data Modeling – Fact Table Effective Practices

Advertisements

Here are a few guidelines for modeling and designing fact tables.

Fact Table Effective Practices

  • The table naming convention should identify it as a fact table. For example:
    • Suffix Pattern:
      • <<TableName>>_Fact
      • <<TableName>>_F
    • Prefix Pattern:
      • FACT_<TableName>>
      • F_<TableName>>
    • Must contain a temporal dimension surrogate key (e.g. date dimension)
    • Measures should be nullable – this has an impact on aggregate functions (SUM, COUNT, MIN, MAX, and AVG, etc.)
    • Dimension Surrogate keys (srky) should have a foreign key (FK) constraint
    • Do not place the dimension processing in the fact jobs

Infosphere DataStage – Boolean Handling for Netezza

Advertisements

Beware when you see this message when working with Boolean in DataStage, the message displays as informational (at list it did for me) not as a warning or an error.  Even though it seems innocuous, what it meant for my job, was the Boolean (‘true’ / ‘false’) was not being interpreted and everything posted to ‘false’.

In DataStage the Netezza ‘Boolean’ field/Data SQL type maps to the ‘Bit’ SQL type, which expects a numeric input of Zero (0) or one (1).  So, my solution (once I detected the problem during unit testing) was to put Transformer Stage logic in place to convert the Boolean input to the expected number value.

Netezza to Datastage Data Type Mapping

Netezza data types

InfoSphere DataStage

data types (SQL types)

Expected Input value

BOOLEANBit0 or 1 (1 = true, 0 = false)

Transformer Stage logic Boolean Handling Logic

A Netezza Boolean field can store: true values, false values, and null. So, some thought should be given to you desired data outcome for nulls

This first example sets a that the nulls are set to a specific value, which can support a specific business rule for null handling and, also, provide null handling for non-nullable fields.  Here we are setting nulls to the numeric value for ‘true’ and all other non-true inputs to ‘false’.

If isnull(Lnk_Src_In.USER_ACTIVE) then 1 Else if Lnk_Src_In.USER_ACTIVE = ‘true’ Then 1 Else 0

These second examples sets a that the nulls are set by the Else value, if your logic direction is correct value and still provides null handling for non-nullable fields.

  • If  Lnk_Src_In.USER_ACTIVE = ‘true’ Then 1 Else 0

  • If  Lnk_Src_In.USER_ACTIVE = ‘False’ Then 0 Else 1

Director Log Message

Message ID

  • IIS-DSEE-TBLD-00008

Message Text

  • <<Link Name Where Message Occurred>>: Numeric string expected. Use default value.

Or something like this:

  • <<Link Name Where Message Occurred>>: Numeric string expected for input column ‘<<Field Name Here>>‘. Use default value.

Related References

Boolean

PureData System for Analytics, PureData System for Analytics 7.2.1, IBM Netezza user-defined functions, UDX data types reference information, Supported data types, Boolean

https://www.ibm.com/support/knowledgecenter/en/SSULQD_7.2.1/com.ibm.nz.udf.doc/r_udf_boolean_datatype.html

Data types and aliases

PureData System for Analytics, PureData System for Analytics 7.2.1, IBM Netezza stored procedures, NZPLSQL statements and grammar, Variables and constants, Data types and aliases

https://www.ibm.com/support/knowledgecenter/en/SSULQD_7.2.1/com.ibm.nz.sproc.doc/c_sproc_data_types_aliases.html

Logical data types

PureData System for Analytics, PureData System for Analytics 7.2.1, IBM Netezza database user documentation, Netezza SQL basics, Data types, Logical data types

https://www.ibm.com/support/knowledgecenter/en/SSULQD_7.2.1/com.ibm.nz.dbu.doc/r_dbuser_data_types_logical.html

Data type conversions from Netezza to DataStage

InfoSphere Information Server, InfoSphere Information Server 11.5.0, Connecting to data sources, Databases, Netezza Performance Server, Netezza connector, Designing jobs by using the Netezza connector, Defining a Netezza connector job, Data type conversions, Data type conversions from Netezza to DataStage

https://www.ibm.com/support/knowledgecenter/en/SSZJPZ_11.5.0/com.ibm.swg.im.iis.conn.netezza.use.doc/topics/nzcc_mappingdatatypes.html

InfoSphere DataStage – Operations Manual Template

Advertisements

When projects transition to production, the receiving team need some transition documentation to help with knowledge transfer and to guide them while they get accustomed to operating the application and can form their own documentation.  This temple is a shell, which usually provides enough detail, with some updating, to provide a quick reference regarding the application.  Also, having a temple can be a real time-saver, as opposed to writing it from scratch.

Related References

InfoSphere DataStage – DataStage Parallel Job Peer Code Review Checklist Template

Advertisements

Peer code review happens during the development phase and focuses on the overall quality and compliance to standards of code and configuration artifacts. However, the hard part of performing a Peer code review isn’t, performing the review, but rather to achieving consistency and thoroughness in the review.   This is where a checklist can contribute significantly, providing a list of things to check and providing a relative weight for the findings.  I hope this template assists with your DataStage job review process.

Netezza / PureData – How to add comments on a field

Advertisements

The ‘Comment on Column’ provides the same self-documentation capability as ‘Comment On table’, but drives the capability to the column field level.  This provides an opportunity to describe the purpose, business meaning, and/or source of a field to other developers and users.  The comment code is part of the DDL and can be migrated with the table structure DDL.  The statement can be run independently or working with Aginity for PureData System for Analytics, they can be run as a group, with the table DDL, using the ‘Execute as a Single Batch (Ctrl+F5) command.

Basic ‘COMMENT ON field’ Syntax

  • The basic syntax to add a comment to a column is:

COMMENT ON COLUMN <<Schema.TableName.ColumnName>> IS ‘<<Descriptive Comment>>’;

Example ‘COMMENT ON Field’ Syntax

  • This is example syntax, which would need to be changed and applied to each column field:

COMMENT ON COLUMN time_dim.time_srky IS ‘time_srky is the primary key and is a surrogate key derived from the date business/natural key’;

Netezza / PureData – Table Documentation Practices

Advertisements

Applying a few table effective practices can:

  • Significantly aide others in understanding the purpose and use of tables and column fields
  • Provide more migratable meta data to inform and enable the capabilities of other tools
  • And, from a self-preservation point of view, reduce the number of meetings, emails, and phone required to explain and clarify what the table and fields business meaning and roles

Among the Effective practices when modeling and design tables are:

  • Add Primary Key
  • Add table comments
  • Add column field comments
  • Add ‘Organize by’ fields to provide essential optimization on frequently used and/or key performance fields

What is the URL for InfoSphere Information Analyzer

Advertisements

This is one of those things, which usually comes up during new installs.   To access the Infosphere information analyzer thin client there are two approaches. First, is via the Infosphere launchpad page and, the second is to go directly to the information analyzer page. Both URLs are provided in below.

InfoSphere LaunchPad URL

https:// <<Host_Server>> : 9443/ibm/iis/launchpad/

Information Analyzer URL 

https:// <<Host_Server>> : 9443/ibm/iis/dq/da/login.jsp

Related References

Netezza Connector Stage, Table name required warning for User-defined SQL Write mode

Advertisements

Recently, while working at a customer site and I encountered an anomaly in the Netezza Connector stage, when choosing ‘User-defined SQL’ write mode, the ‘Table name’ displays a caution/warning even though a table name should not be required.  If you are using a user-defined SQL statement and/or have parametrized your SQL scripts to make the job reusable, each SQL and/or SQL script would have its own schema and table name being passed in.  After some investigation, a workaround was found, which both allows you to populate table name and leverage with different schema and table names within your SQL statement and/or.

Table Name, User-defined SQL, Warning

You will notice, in a screenshot below the ‘User-defined SQL’, ‘write mode’, the property has been chosen, a parameter has been placed in the ‘User-defined SQL’ property, and ‘Read user-defined SQL from a file’ property has been set to ‘Yes’.  However, the yellow triangle displays on the ‘Table name’ property marking it as a required item.  This, also, occurs when placing SQL statements in the User-defined SQL property, whether reading from a file or not.

Table Name, User-defined SQL, Warning Workaround

After some experimentation, the workaround is straight forward enough.  Basically, give the ‘table name’ property something to read successfully, so it can move on to the user-defined SQL and/or user-defined SQL file script, which the process actually needs to execute. In the screenshot below, the SYSTEM.DEFINITION_SCHEMA._V_DUAL view was used, so, it could be found, then the script file passed by the parameter runs fine.  Another view or table, which the DataStage user has access to, should just as well.

Related References

DB2 JDBC ISJDBC.CONFIG Configuration

Advertisements

Here are a few pointers for building an IBM InfoSphere Information Server (IIS) isjdbc.config file for an IBM DB2 Universal Driver, Type 4.

Where to place JAR files

For Infosphere Information Server installs, as a standard practice, create a custom jdbc file in the install path.  And install any download Jar file not already installed by other applications in the jdbc folder. Usually, jdbc folder path looks something like this:

  • /opt/IBM/InformationServer/jdbc

CLASSPATH

  • db2jcc.jar
  • Classpath must have a complete path and jar name

CLASS_NAMES

  • ibm.db2.jcc.DB2Driver

JAR Source URL

DB2 DEFAULT PORT

  • 5000

JDBC URL FORMAT

  • jdbc:db2://[:]/

JDBC URL EXAMPLE

jdbc:db2://127.0.0.1:50000/IADB

isjdbc.config EXAMPLE

CLASSPATH=/opt/IBM/InformationServer/ASBNode/lib/java/db2jcc.jar;

CLASS_NAMES=com.ibm.db2.jcc.DB2Driver;

Isjdbc.config FILE PLACEMENT

  • /opt/IBM/InformationServer/Server/DSEngine

Related References

DB2 to Netezza data type conversions

Advertisements

This is an extract table I created from the IBM source, a while back when investing what format to convert data fields into for IBM Infosphere Datastage.  I have had it floating around in my notes, but lately, I have found myself referencing it to help other team members, so, it seems useful to include it here.   The notes column is just a few snippets of information, which I have found useful to reference when planning data field conversions.

Transformer data typeDB2 data typeNetezza data typeNotes
BIGINTBIGINTBIGINT 
BINARYCHARACTERNot supported 
BITSMALLINTBOOLEAN 
CHARCHARACTERCHAR 
DATEDATEDATE 
DECIMALDECIMALNUMERIC or DECIMAL 
DOUBLEDOUBLEDOUBLE PRECISION or FLOAT(15) 
FLOATDOUBLEFLOAT 
INTEGERINTEGERINTEGER 
LONGVARBINARYBLOBNot supported 
LONGVARCHARCLOBVARCHARThe maximum character string size is 64,000.
NUMERICDECIMALNUMERIC 
REALREALREAL or FLOAT(6) 
SMALLINTSMALLINTSMALLINT 
TIMETIMETIMEDB2 9.1 TIME data type does not support fractional digits or microseconds.
TIMESTAMPTIMESTAMPTIMESTAMPDB2 9.1 TIME data type does not support fractional digits or microseconds.
TINYINTSMALLINTBYTEINT 
VARBINARYVARCHARNot supported 
VARCHARVARCHARVARCHARThe maximum character string size is 64,000.
WCHARGRAPHICNot supported 
WLONGVARCHARLONG VARGRAPHICNot supported 
WVARCHARVARGRAPHICNot supported 

Related References

InfoSphere Information Server InfoSphere Information Server 11.5.0 – Datatypes

IBM PureData System for Analytics, Version 7.1 – Data types and aliases

Netezza – JDBC ISJDBC.CONFIG Configuration

Advertisements

This jdbc information is based on Netezza (7.2.0) JDBC for InfoSphere Information Server11.5.  so, here are a few pointers for building an IBM InfoSphere Information Server (IIS) isjdbc.config file.

Where to place JAR files

For Infosphere Information Server installs, as a standard practice, create a custom jdbc file in the install path.  And install any download Jar file not already installed by other applications in the jdbc folder. Usually, jdbc folder path looks something like this:

  • /opt/IBM/InformationServer/jdbc

CLASSPATH

  • nzjdbc3.jar
  • Classpath must have complete path and jar name

CLASS_NAMES

  • netezza.Driver

JAR Source URL

IBM Netezza Client Components V7.2 for Linux

File name

  • nz-linuxclient-v7.2.0.0.tar.gz

Unpack tar.gz

  • tar -zxvf nz-linuxclient-v7.2.0.0.tar.gz -C /opt/IBM/InformationServer/jdbc

DB2 DEFAULT PORT

  • 1521

JDBC URL FORMAT

  • jdbc:netezza://:/

JDBC URL EXAMPLE

  • jdbc:netezza://10.999.0.99:5480/dashboard

isjdbc.config EXAMPLE

CLASSPATH=usr/jdbc/nzjdbc3.jar;/usr/jdbc/nzjdbc.jar;/usr/local/nz/lib/nzjdbc3.jar;

CLASS_NAMES= org.netezza.Driver;

Isjdbc.config FILE PLACEMENT

  • /opt/IBM/InformationServer/Server/DSEngine

Related References

Netezza JDBC Error – Unterminated quoted string

Advertisements

The ‘Unterminated quoted string’ error occurs from time to time when working with the InfoSphere DataStage Netezza JDBC Connector stage and is nebulas, at best.  However, the solution is, normally, straight forward enough once you understand it.  Usually, this error is the result of target table fields or field being shorter than the input data.  The fix is, normally, to compare you input field lengths (or composite field length, if consolidation fields into one field) and adjusting the field length higher.  In some cases, if business rules allow you may be able to substring or truncate the input data length (not a recommended approach), but information can be lost with this approach.

Error

org.netezza.error.NzSQLException: ERROR:  Unterminated quoted string

Example Error Message

Tgt_IIS_Job_Dim,0: The connector encountered a Java exception:  org.netezza.error.NzSQLException: ERROR:  Unterminated quoted string    at org.netezza.internal.QueryExecutor.getNextResult(QueryExecutor.java:287)    at org.netezza.internal.QueryExecutor.execute(QueryExecutor.java:76)  at org.netezza.sql.NzConnection.execute(NzConnection.java:2904)       at org.netezza.sql.NzStatement._execute(NzStatement.java:885)           at org.netezza.sql.NzPreparedStatament.executeUpdate(NzPreparedStatament.java:229)   at com.ibm.is.cc.jdbc.CC_JDBCRecordDataSetConsumer.executeStatements(CC_JDBCRecordDataSetConsumer.java:2846)               at com.ibm.is.cc.jdbc.CC_JDBCBigBufferRecordDataSetConsumer.consumeBigBuffer(CC_JDBCBigBufferRecordDataSetConsumer.java:712)

What is a DataStage Sequence

Advertisements

A sequence is an orchestration mechanism within IBM Infosphere Information server DataStage used to organize ETL’s and sequence  batch event streams, which are normally called from an Enterprise Scheduling tool or other processes.  Sequences commonly establish down stream dependencies with DataStage and/or actions (e.g. notification) based on process status reporting (e.g. Success, Failure, etc.).

IBM InfoSphere Business Glossary Anywhere Client Is Gone

Advertisements

I thought I would post this, to point out what appears to be a case of incorrect  documentation, with regard to InfoSphere Information Server (IIS) 11.5 and 11.3.

According to IBM’s own documentation ‘InfoSphere Business Glossary, InfoSphere Business Glossary Anywhere, and InfoSphere Metadata Workbench are consolidated into the new product InfoSphere Information Governance Catalog.’  So, beware of these obsolete links, provided below.

Source link

Obsolete IBM Documentation Links for InfoSphere Business Glossary Anywhere InfoSphere

11.5

11.3

 

SQL Server JDBC ISJDBC.CONFIG Configuration

Advertisements

This jdbc information is based on Oracle Database 11g Release 2, (11.2.0.4) JDBC for InfoSphere Information Server11.5, and ReedHat Linux 6.  so, here are a few pointers for building an IBM InfoSphere Information Server (IIS) isjdbc.config file.

Where to place JAR files

For Infosphere Information Server installs, as a standard practice, create a custom jdbc file in the install path.  And install any download Jar file not already installed by other applications in the jdbc folder. Usually, jdbc folder path looks something like this:

  • /opt/IBM/InformationServer/jdbc

CLASSPATH

  •  sqljdbc.jar
  •  sqljdbc4.jar
  •  sqljdbc41.jar
  •  sqljdbc42.jar
  • Classpath must have complete path and jar name

CLASS_NAMES

  • microsoft.jdbc.sqlserver.SQLServerDriver;com.microsoft.sqlserver.jdbc

JAR Source URL

DEFAULT PORT

  • 1433

JDBC URL FORMAT

  •  jdbc:microsoft:sqlserver://HOST:1433;DatabaseName=DATABASE

JDBC URL EXAMPLE

  • jdbc:sqlserver://RNO-SQLDEV-SVR1DEV01:55198;databaseName=APP1;

isjdbc.config EXAMPLE

CLASSPATH=/opt/IBM/InformationServer/jdbc/sqljdbc_3.0/enu/sqljdbc4.jar;/opt/IBM/InformationServer/jdbc/sqljdbc_3.0/enu/sqljdbc.jar;/opt/IBM/InformationServer/jdbc/sqljdbc_3.0/enu/sqljdbc41.jar;/opt/IBM/InformationServer/jdbc/sqljdbc_3.0/enu/sqljdbc42.jar;

CLASS_NAMES=com.microsoft.jdbc.sqlserver.SQLServerDriver;com.microsoft.sqlserver.jdbc

Isjdbc.config FILE PLACEMENT

  • /opt/IBM/InformationServer/Server/DSEngine

Related References

Vendor Reference Link:

Oracle JDBC ISJDBC.CONFIG Configuration

Advertisements

This jdbc information is based on Oracle Database 11g Release 2 (11.2.0.4), on a RAC (Oracle Real Application Clusters), JDBC for InfoSphere Information Server11.5 on Red Hat Linux.  so, here are a few pointers for building an IBM InfoSphere Information Server (IIS) isjdbc.config file.

Where to place JAR files

For Infosphere Information Server installs, as a standard practice, create a custom jdbc folder in the install path and copy the jar file into the folder (not install activity required).   Usually, jdbc folder path looks something like this:

  • /opt/IBM/InformationServer/jdbc

JAR Source URL

  • In this example, we used the jar files from the client install, but if you want to skip the client install you can download the drivers here: Oracle JDBC Drivers

Oracle DEFAULT PORT

  • 1521

JDBC URL FORMAT

  • jdbc:oracle:thin:@//:/ServiceName

or

  • jdbc:oracle:thin:@<host>:<port>:<SID>

JDBC URL EXAMPLE

  • jdbc:oracle:thin:@//RAC01-scan:1521/DW

Create And Place A jdbc configuration file

The Isjdbc.config file needs to be placed in the DSEngine directory:

Isjdbc.config File Path

  • /opt/IBM/InformationServer/Server/DSEngine

isjdbc.config Example

CLASSPATH=/opt/app/oracle/product/11.2.0/client_1/jdbc/lib/ojdbc6.jar;

CLASS_NAMES=oracle.jdbc.OracleDriver

isjdbc.config Properties Notes

CLASSPATH

  • jar
  • Classpath must have complete path and jar name

CLASS_NAMES

  • oracle.jdbc.OracleDriver

Related References

What are the Tiers of InfoSphere Information Server (IIS)?

Advertisements

Tiers of InfoSphere Information Server (ISS)

Infosphere Information Server (IIS) is organized into these four tiers:

Client Tier

The client tier is a logical tier which consists of the client programs and consoles that are used for development, administration, and other tasks. The Infosphere information server consoles included in the client tier are, actually, web-based components, while other components, such as Datastage Designer, are windows client-based applications.

Services Tier

The services tier consists of the application server (IBM WebSphere), common services, and product services for the suite and product modules. The services tier provides common services, such as security, and services which are specific to certain product modules. On the services tier, IBM WebSphere® Application Server hosts the services and the InfoSphere Information Server applications which are Web-based.

Engine Tier

The engine tier is the logical group of engine components, communication agents, and so on. The engine runs jobs and other tasks for product components and modules, such as, DataStage and Information Analyzer.

Repository Tier

The repository tier consists of the IBM InfoSphere Information Server metadata repository and, if installed, other data stores to support other product modules. The metadata repository contains the shared metadata, data, and configuration information for InfoSphere® Information Server product components and modules. The other data stores store extended data for use by the product modules they support, such as the operations database, which is a data store that is used by the engine operations console.

Related References

Infosphere Data Architect Install Error CRIMC1029E or CRIMC1085E

Advertisements

The CRIMC1029E / CRIMC1085E errors may be caused by running the incorrect Infosphere Data Architect installer.  If you run the admin installer (launchpad.exe) on 64bit windows with insufficient privileges, the process will throw a CRIMC1029E / CRIMC1085E error.

What the Error Looks Like

Installation failed.

CRIMC1029E: Adding plug-in com.ibm.etools.cobol.win32_7.0.921.v20140409_0421 to repository E:Program FilesIBMSDPShared failed.

CRIMC1085E: Resumable download failed for: file:/E:/InstallIBM/INFO_DATA_ARCH_V9.1.2_WIN/disk2/ad/plugins/file000086.

‘plug-in com.ibm.etools.cobol.win32_7.0.921.v20140409_0421’ does not exist (at file:/E:/InstallIBM/INFO_DATA_ARCH_V9.1.2_WIN/disk2/ad/plugins/file000086).

Elapsed time 00:00.00.

Solution

Run the launchpad_win_nonadmin64.exe file, instead of the launchpad.exe file.

Reference Link:

Installing IBM InfoSphere Data Architect with the Installation Manager Install wizard

Infosphere Datastage – How to Generate Datastage Technical Design Job Reports

Advertisements

Advantages of Self-Documentation

The ability to self-document is one of Infosphere Datastage often overlooked and underappreciated capabilities.  However, the ability to generate a very detailed technical design specification can provide several advantages:

  • Time savings: The report only takes a minute, or so, for each job to generate, which is considerably faster than even a high-speed skill designer could create the document with the same level of detail.
  • Accuracy: The report, at the time of generation, is rendered from the metadata and, there, contains no human errors such as typos, human assumptions, and/or omissions.
  • Distributed User Accessibility: In addition to being able to save the reports off for other purposes, the report can be retained in the report repository and made available for viewing by teams and stakeholders, if so enable.

Report Generation Steps

Step 1: Log in to DataStage from your client.

Step 2: Navigate to the Job you wish to document and open/edit job.

Step 3: Go to ‘File’ > and Click on ‘Generate Report’.

Step 4:  When the report Details dialog box appears:

  • Choose Style Sheet
  • Set ‘Automatically Purge After:’
  • Edit Report Name and Description, if desired, and click ‘OK’.

Step 5 To view the report in DataStage, Click on the hyperlink in the DataStage box shown below.

Step 6:  The rendered Datastage job report will appear in the web browser.

Step 7: to save the report for use outside of Infosphere, within the browser, navigate to ‘File’ > ‘Save As’.

Step 4: Navigate to the desired path and choose “Web Archive, single file (*.mht)” and Click Save.  I have found the “*.mht” to be the best format for exporting DataStage reports, but you may choose any format which meets your needs.

Oracle SQL – Removing Non-Numeric Characters

Advertisements

Sometimes, when converting fields from text to numeric fields in SQL, there is a need to remove one or more non-numeric values.  These approaches have been useful for handling multiple non-numeric values.

Removing non-numeric values – Approach 1:

I have found this approach useful when working with character (Char, Varchar, NVARCHAR) fields being converted to non-integers outputs (e.g. Decimal).

CAST(( CASE WHEN TRIM(<<SourceFieldName>>) > ‘9999999999’ or  length(REGEXP_SUBSTR(<<SourceFieldName>>, ‘[A-z]’, 1, 1))>0 or trim(<<SourceFieldName>>) in (‘-‘,’+’,’?’,’/’,’*’) THEN NULL ELSE trim(<<SourceFieldName>>) END) AS    DECIMAL(10,2)) AS <<TargetFieldName>>,

Removing non numeric values – Approach 2:

If you need to know if a content of a field is numeric, I have most this approach useful when working with character (Char, Varchar, NVARCHAR) fields being converted to integers:

CAST(( CASE WHEN LENGTH(TRANSLATE(<<SourceFieldName>>,’0123456789′,”))=0 THEN (<<SourceFieldName>> ELSE NULL END) as INTEGER) as <<TargetFieldName>>,

Related Posts