SSIS Skip Blank Rows in Flat File Source

What?
This is a quick article to remind me on how to skip blank rows when using a Flat file as a data source. I would receive another Microsoft error as clear as mud:
copyraw
Error: The conditional operation failed.

Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.  The "component 
"MyDerivedColumns" (4228)" failed because error code 0xC0049063 occurred, and the error
row disposition on "output column "DC_MyDate" (7349)" specifies failure on error. An 
error occurred on the specified object of the specified component.  There may be error 
messages posted before this with more information about the failure.
  1.  Error: The conditional operation failed. 
  2.   
  3.  Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.  The "component 
  4.  "MyDerivedColumns" (4228)" failed because error code 0xC0049063 occurred, and the error 
  5.  row disposition on "output column "DC_MyDate(7349)" specifies failure on error. An 
  6.  error occurred on the specified object of the specified component.  There may be error 
  7.  messages posted before this with more information about the failure. 

Why?
So the solution must be obvious to you by now. At least that's what Microsoft people think to themselves every time they see the error they programmed in.

Apparently this problem also happens when you have a Data File of varying column numbers. My solution below will also fix this.

How?
My data file had some empty rows amongst the data. The easy fix is by dealing with any empty rows. The best way I've seen that has worked in nearly all cases, is to use the "conditional split" task and test one of the columns for a length of greater than 1:


Almost!
A problem not covered by many sites was happening to me when I had blank rows in a flat file, where it appeared to add the special carriage return or line feed characters to the next viable row, see my original data file here (sample data):
And this is what a Data Viewer returned when executed:
My columns were being pushed out of alignment (something about {CR}{LF} being displayed for the blank rows and inserted in the next line) and creating discrepancies that would error the flow.

Solution Found!!!

Taking the idea that the Carriage Return (CR) or Line Feeds (LF) are being inserted, simply extract the entire row as a column and trim it. I separate out the first column as my index/key field so I can check if this is a valid data row (the blank ones may return special chars rather than Null).

My example here is that the first column is our employee number and the second column will be the remaining data on that row.

So I setup my connection manager as follows (note the "Ragged Right" and delimiter as "{CR}{LF}" - my rows to skip is because of the state of my data):
Crucial fix here, add 2 to the OutputColumnWidth value. It seems you need to set the data column to accommodate for the special characters the blank rows somehow feed in. In this case, the Ragged Right should go to the next {CR}{LF}:
The Data Flow from file to valid columns:
  1. Flat File Source: Ignore failure on Truncate, Output "KeyColumn" and "DataColumn".
  2. Derived Column Task: Trim and Replace special characters from "DataColumn" and "KeyColumn".
  3. Conditional Split Task: Check keycolumn is not null and data column is not blank.
  4. Derived Column Task: Format keycolumn as per your requirements and use substring on trimmed DataColumn to retrieve each data column.


Additional
The TRIM in SSIS does not appear to get rid of new line and carriage return symbols!!! Use the Replace(String, "\r", "") and Replace(String, "\n", "").

Feedback
This solution took me two days to solve despite following an innumerable number of sites advising but none of them having my issue of rows after blank ones being out of alignment. TWO DAYS!!! I hope this helps you!

If you know of other ways to resolve this, I would be the first to be interested in how...
Category: SQL Server Integration Services :: Article: 505

Credit where Credit is Due:


Feel free to copy, redistribute and share this information. All that we ask is that you attribute credit and possibly even a link back to this website as it really helps in our search engine rankings.

Disclaimer: Please note that the information provided on this website is intended for informational purposes only and does not represent a warranty. The opinions expressed are those of the author only. We recommend testing any solutions in a development environment before implementing them in production. The articles are based on our good faith efforts and were current at the time of writing, reflecting our practical experience in a commercial setting.

Thank you for visiting and, as always, we hope this website was of some use to you!

Kind Regards,

Joel Lipman
www.joellipman.com

Related Articles

Joes Revolver Map

Joes Word Cloud

Accreditation

Badge - Certified Zoho Creator Associate
Badge - Certified Zoho Creator Associate

Donate & Support

If you like my content, and would like to support this sharing site, feel free to donate using a method below:

Paypal:
Donate to Joel Lipman via PayPal

Bitcoin:
Donate to Joel Lipman with Bitcoin bc1qf6elrdxc968h0k673l2djc9wrpazhqtxw8qqp4

Ethereum:
Donate to Joel Lipman with Ethereum 0xb038962F3809b425D661EF5D22294Cf45E02FebF
© 2024 Joel Lipman .com. All Rights Reserved.