Data Services and Amazon S3 - Integration?

[#1]

A client just threw a Data Services question at me that I can't answer - how do you get Data Services to integrate with Amazon S3?  They want to read files from S3 into Data Services for analysis and transformation, and then write their output back to S3 (ideally).

I did some googling and surprisingly, I can't find any discussions of anyone having done this before.

Could Data Services could call FTP to move files in and out of S3?

Is there some other interface or API that could be called from Data Services using the SQL transform (via a custom stored procedure)?

Has anyone already solved this?

I would be delighted to hear your ideas or war stories.

Cheers!

Michael

[Former Member]

Is this topic really a Bump?

Nobody on SCN has any experience, or even ideas on how Data Services integrates with S3?

[Former Member]

Have not needed to do this yet, but this did come up as a dicussion recently with a customer. Pretty sure you'll need to use a web service data store and use REST API (Data Services 14.2.2 would be required I believe for REST) as according to S3 current documentation SOAP support over HTTP has be depricated. Looks like SOAP still works over HTTPS but will not be supported with new functionality moving forward.

Looks like you create buckets on S3 and configure who and how you can access, but comunication is via web services.

Here is information for writing to S3 using buckets:

Uploading files to Amazon S3 - Amazon Redshift

Also, there are a ton of SDK's available for data exchange. Here are the docs for Python that could be used from a user defined transform in Data Services:

boto: A Python interface to Amazon Web Services — boto v2.32.0

To start I would think you would just need to create a web service data store and be familar with calling functions from DS look like.

Hope this helps.

[Former Member]

Hi Michael,

Amazon Redshift uses industry standard SQL and is accessed using standard JDBC, or ODBC, drivers.

I have never worked with redhsift driver but you can use the below thread to read data from AWS. Hope it is useful

Data Services with Amazon S3/Redshift as target | SCN

Additional Link

Configure an ODBC Connection - Amazon Redshift

Regards

Arun Sasi