Python Meeting Data Lake Part 1 - Connect and Query

[shivamshukla12]

Hi All ,

I am sharing my new knowledge on Data Lake - exploring all the way how we can interact with data lake directly.

Pre-requisites: https://developers.sap.com/tutorials/hana-cloud-dl-clients-overview.html Create your own data lake instance under managed or stand alone both are fine.

Here is my DL Instance up and running.

 

Install SAP IQ Client - Make sure you pick the latest version from SAP Software downloads.

Also you can follow the above tutorials for understanding data lake & SAP HANA Cloud instance. Data lake is one of the best way to store different kind of data from different source at one place and importantly at a very low-cost.

there are already few posts which can let you start easily on this topic , I am sharing my own learning So here we go .

• Go to BTP Cockpit & create your free trial account. https://account.hana.ondemand.com/

• Create your development space and under the SAP HANA Cloud create data lake instance , Choose IP whitelisting according to the requirement.

 

• SAP IQ Installation - Download SAP IQ drivers from SAP Software downloads.

• Open ODBC in administrator mode (I am explaining this in my windows system)  - if you see SAP IQ in below screen it means drivers installation is fine in your system

• 

• Now go to your HANA Data Lake instance and right click on the top right and copy SQL Endpoints & fill it in below driver details.

 

• Test Connection.

• If this is done - now you are good to go for ODBC Connection to your data lake - Programming environment is your choice now.

• I am using Jupyter notebook and Python (PYODBC) for interacting with data lake & have created few tables and inserted some data as well which bring back in my python client.

• https://pypi.org/project/pyodbc/

Install PYODBC

• 

• Open Jupyter notebook and try to connect with Data Lake Instance .

 

Import Packages and provide connection details.

import pyodbc

cnxn = pyodbc.connect('DSN=HDLSA;UID=HDLADMIN;PWD=abc1234@123A')

 

Open cursor and execute some select statements.

cur = cnxn.cursor()

cur.execute('SELECT * FROM HOTEL.HOTEL')

Hotel Table under HOTEL Schema is already created , follow data lake tutorials.

Fetch Data

rows = cur.fetchall()

rows

Print Records.

So here comes to and end of connecting data lake from Python , In next part , we will be uploading the data from CSV to Data Lake.

Keep learning Keep querying 🙂

 

Thanks.