Question Re: Auto Import CSV to Data warehouse cloud table in Technology Q&A

Auto Import CSV to Data warehouse cloud table

theanmozhi — Thu, 21 Jan 2021 16:50:52 GMT

Hi Experts,

we have a batch of CSV files in local system and would need to auto load the csv data to DWC system table.

can we do this via custom sql query and load into DWC ? or please let me know the ideal approach on how to automate the process of importing csv to datawarehouse cloud.

Re: Auto Import CSV to Data warehouse cloud table

liskb — Thu, 24 Jun 2021 16:44:57 GMT

I am looking for a similar solution. We have custom tables we have created from CSV files. We can manually reload the data from CSV in the Data builder, but is there a way to automate loading from CSV that exists on our local network or from a cloud location where we may keep these files? Can a Data flow do this?

Re: Auto Import CSV to Data warehouse cloud table

XaviPolo — Fri, 25 Jun 2021 07:29:06 GMT

At the moment it is not possible to do this automatically. There is no "CSV" connection type that allows remote access or use it in Data Flow.

I have not tested it, but I think that an option, although complex, would be to use in DWC a JDBC connection, and to configure a JDBC driver to access CSV (there are paid ones and some open source). This would allow viewing the files as remote tables.

Other options are possible, but they involve using third party systems (HANA, Azure, etc) and access through these systems.

Re: Auto Import CSV to Data warehouse cloud table

XaviPolo — Fri, 25 Jun 2021 10:47:52 GMT

At DWC Roadmap a FTP based connection is planned for Q3 2021, and this option will allow to import CSV data with Data Flows.

Re: Auto Import CSV to Data warehouse cloud table

atharvapangerkar — Wed, 22 Feb 2023 05:43:59 GMT

HI theankathi ,

Did you find if there is any way to automate the process of importing CSV files?

Re: Auto Import CSV to Data warehouse cloud table

XaviPolo — Wed, 22 Feb 2023 15:01:33 GMT

Currently you can use SFTP connection and Data Flows to import CSVs data to DWC tables.