Print

SSIS Skip Blank Rows in Flat File Source

What?
This is a quick article to remind me on how to skip blank rows when using a Flat file as a data source. I would receive another Microsoft error as clear as mud:
copyraw 
Error: The conditional operation failed.

Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.  The "component 
"MyDerivedColumns" (4228)" failed because error code 0xC0049063 occurred, and the error
row disposition on "output column "DC_MyDate" (7349)" specifies failure on error. An 
error occurred on the specified object of the specified component.  There may be error 
messages posted before this with more information about the failure.
  1.  Error: The conditional operation failed. 
  2.   
  3.  Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.  The "component 
  4.  "MyDerivedColumns" (4228)" failed because error code 0xC0049063 occurred, and the error 
  5.  row disposition on "output column "DC_MyDate" (7349)" specifies failure on error. An 
  6.  error occurred on the specified object of the specified component.  There may be error 
  7.  messages posted before this with more information about the failure. 

Why?
So the solution must be obvious to you by now. At least that's what Microsoft people think to themselves every time they see the error they programmed in.

Apparently this problem also happens when you have a Data File of varying column numbers. My solution below will also fix this.

How?
My data file had some empty rows amongst the data. The easy fix is by dealing with any empty rows. The best way I've seen that has worked in nearly all cases, is to use the "conditional split" task and test one of the columns for a length of greater than 1:


Almost!
A problem not covered by many sites was happening to me when I had blank rows in a flat file, where it appeared to add the special carriage return or line feed characters to the next viable row, see my original data file here (sample data):
And this is what a Data Viewer returned when executed:
My columns were being pushed out of alignment (something about {CR}{LF} being displayed for the blank rows and inserted in the next line) and creating discrepancies that would error the flow.

Solution Found!!!

Taking the idea that the Carriage Return (CR) or Line Feeds (LF) are being inserted, simply extract the entire row as a column and trim it. I separate out the first column as my index/key field so I can check if this is a valid data row (the blank ones may return special chars rather than Null).

My example here is that the first column is our employee number and the second column will be the remaining data on that row.

So I setup my connection manager as follows (note the "Ragged Right" and delimiter as "{CR}{LF}" - my rows to skip is because of the state of my data):
Crucial fix here, add 2 to the OutputColumnWidth value. It seems you need to set the data column to accommodate for the special characters the blank rows somehow feed in. In this case, the Ragged Right should go to the next {CR}{LF}:
The Data Flow from file to valid columns:
  1. Flat File Source: Ignore failure on Truncate, Output "KeyColumn" and "DataColumn".
  2. Derived Column Task: Trim and Replace special characters from "DataColumn" and "KeyColumn".
  3. Conditional Split Task: Check keycolumn is not null and data column is not blank.
  4. Derived Column Task: Format keycolumn as per your requirements and use substring on trimmed DataColumn to retrieve each data column.


Additional
The TRIM in SSIS does not appear to get rid of new line and carriage return symbols!!! Use the Replace(String, "\r", "") and Replace(String, "\n", "").

Feedback
This solution took me two days to solve despite following an innumerable number of sites advising but none of them having my issue of rows after blank ones being out of alignment. TWO DAYS!!! I hope this helps you!

If you know of other ways to resolve this, I would be the first to be interested in how...
Category: SQL Server Integration Services :: Article: 505