Auto Import CSV to Data warehouse cloud table

[theanmozhi]

Hi Experts,

we have a batch of CSV files in local system and would need to auto load the csv data to DWC system table.

can we do this via custom sql query and load into DWC ? or please let me know the ideal approach on how to automate the process of importing csv to datawarehouse cloud.

[liskb]

I am looking for a similar solution. We have custom tables we have created from CSV files. We can manually reload the data from CSV in the Data builder, but is there a way to automate loading from CSV that exists on our local network or from a cloud location where we may keep these files? Can a Data flow do this?

[atharvapangerkar]

HI theankathi ,

Did you find if there is any way to automate the process of importing CSV files?

[XaviPolo]

Currently you can use SFTP connection and Data Flows to import CSVs data to DWC tables.

[XaviPolo · Accepted solution]

At DWC Roadmap a FTP based connection is planned for Q3 2021, and this option will allow to import CSV data with Data Flows.

[XaviPolo]

At the moment it is not possible to do this automatically. There is no "CSV" connection type that allows remote access or use it in Data Flow.

I have not tested it, but I think that an option, although complex, would be to use in DWC a JDBC connection, and to configure a JDBC driver to access CSV (there are paid ones and some open source). This would allow viewing the files as remote tables.

Other options are possible, but they involve using third party systems (HANA, Azure, etc) and access through these systems.