different formats of the flat file for the same target

[Former Member]

In our deployment, we use plugin code to extract the csv files in the required format. The customers are on same version of datamart, but they are on different versions of source database - from 3.x to 4.5 depending on which version of application they are using. In 4.0, we introduced a new column email in the user table in the source database. Accordingly, plugin will add the field in the csv file. But not all the customers will get the upgraded version of plugin at the same time. So ETL code needs to decide which data flow to process depending on the format of the csv file to load data to the same target table. I made the email field in the target table nullable but it still expects the same format of the csv file with delimiter for null value.

Need help to achieve this. Can I read the structure of the flat file in DS or get the count of delimiters so that I can use a conditional to use different data flow based on the format of the flat files.

Can I make the email column in the flat file optional?

Thanks much in advance.

[former_member589506]

You can add an email column that maps to null in a query transform for the source that does not contain this column.

Or else you can define two different file formats that map to the same file. One with the column and one without

[Former Member]

You might also want to look at the "Adaptable Schema" option for file format objects. If this option is set to yes then the number of columns that are contained in your CSV file can vary with no ill effect other than null or ignored columns.