This post describes setting up an eclipse based development environment for HANA. While I'm focusing on python, much of the post will apply to other development languages as well.
In the context of SAP's framework for microservice based
Multi-Target Application(MTA) development, your python code may not be front and center as the main component of your application, but instead can make a lot of sense as a worker component that can be sized and scaled as work loads require. Python is a great choice for scientific analysis and machine learning tasks and the community has contributed many libraries that lend themselves to this use. As such, I'll be focusing on setting up a development environment for python as a single component with the understanding that it will likely be part of a larger application. Full MTA development is supported by the
WebIDE but as of this writing doesn't support python as a module language.
If you need a HANA setup to work with, I encourage you to try
HANA Express(HXE). If you have a laptop that can support it, you may want to try my blog post
Setting up a HANA Express Python Machine Learning API Demo VM, which goes through all the steps of getting python built an installed in HXE.
While you can manage your project and all the tasks needed to deploy it with a good text editor and the command line tools, most developers prefer a more functional development environment. We'll start with eclipse since it's freely available and provides a good basis for the common tasks we'll want to perform.
Installing Eclipse:
The first question is which version of eclipse?
While it's tempting to grab the latest version (4.9 as of this writing), I strongly suggest that you use the Photon release(4.8). Pick the packaging that most closely fits your need from this download page.
https://www.eclipse.org/downloads/packages/release/photon/r
Since I'll also be doing some NodeJS development I picked
for my platform (MacOS). This version is set up for HTML and NodeJS and has git plugins installed already.
Installing the HANA plugin:
The reason for picking an older version of eclipse is that the plugin for HANA was last updated for the photon release of eclipse. Visit the
tools page for the latest and decide if you can use a more recent version.
To install SAP HANA Tools, proceed as follows:
- In Eclipse, choose in the menu bar Help > Install New Software...
- For Eclipse Photon (4.8), add the URL https://tools.hana.ondemand.com/photon
- Press Enter to display the available features.
- Select the desired features and choose Next.
- On the next wizard page, you get an overview of the features to be installed. Choose Next.
- Confirm the license agreements and choose Finish to start the installation.
After restarting eclipse, open the SAP HANA Administration Console perspective.
Connecting to an on-prem HANA:
When working with HANA on-premise systems whether they be on a server, running on AWS, or in a VM on your own machine, the process is the same. Open the SAP HANA Administration Console perspective. Click on the system icon and Add System...
Enter the Host Name of the server and add :Port of the first tenant (not the System database).
Enter the SYSTEM user and password.
If the server is reachable from your workstation on 30015, then it should continue without error.
If it times out, you may want to check that your server is not being blocked by a firewall, security policy, or NAT VM setting, etc.
Now you can do administrative tasks (as the SYSTEM user!).
Connecting to a HaaS HANA:
If you want to connect to a HANA system that is provided by the SAP Cloud Foundry HANA as a Service(HaaS), the server config is a bit tricky and
non-obvious.
For details on how to get and setup a HaaS instance, see
SAP Cloud Platform, SAP HANA ServiceGetting Started Guide
First find the fully qualified hostname and port of your HANA instance by logging into your SAP Cloud Platform account and navigating to the space in which your HaaS service instance is defined.
Click on "Open Dashboard" link next to your service instance.
Take note of the full hostname and port.
Back in eclipse, add another system.
This time put just the full hostname in the Host Name field, 00 in the instance, and
H00:
Port in the Mode field. Click Next.
Enter the SYSTEM user and password that you set when creating the instance. Be sure to check the Connect using SSL checkbox. Click Next.
Double check the hostname and port. Click Finish.
You should now have a H00 system defined.
Side-Note: If you want to connect to HANA1 systems provisioned in Neo, add the URL
https://tools.hana.ondemand.com/neon when installing software and
Then select the SAP Cloud Platform Tools.
This creates a new server option, Add Cloud System...
I won't show the details of logging in and add a system here.
Now that your eclipse can connect to HANA systems in on-premise and/or HaaS instances, let's continue setting up for python development.
If you're interested in a different development language, the following may be of help, but you'll have to research which eclipse plugins and procedures are most appropriate for you.
Setting up the Python support:
So assuming we want to do python development, let's load the PyDev eclipse plugin via the Eclipse Marketplace. Under the Help menu, select Eclipse Marketplace.
In the Find box type
pydev and click Go. Click the install button.
Confirm, agree to the license, and click Finish.
Let eclipse restart to load the new plugin.
What about Multi-Target Application development?
SAP promotes a concept for micro-service enterprise development called Multi-Target Applications(MTA). You can find out more about MTA development
here.
Currently there is no public eclipse MTA plugin, but we can at least have some help in editing the mta.yaml descriptor file in our project. Since this file is yaml, add a yaml editor via the Eclipse Marketplace feature. I'm using one called YEdit.
You may want to install other eclipse plugins for your needs, but these make up a basic working set for python development.
Understanding where your application will be deployed and what python will be available there.
One of the problems with python is that of too much python instead of not enough python. Many systems come with some version of python already installed. Others may have more that one version installed. When you start deploying to your XSA or CF system, then yet other versions come into play. Problems often come when some versions of python get mixed up with other versions.
The best solution I've found it to really understand what assumptions are made when python runs and under what environment it's running. Take the time to read up on all the PYTHON environment variables that affect how python runs and where it looks for it's library files and how to manage packages with pip the python packaging tool. Please read up on the environment
here.
Setting up Python locally.
Again, you're likely to have python in some form already on your local system. Open a terminal and type python --version and see what it finds.
python --version
If you plan on working locally as much as possible, it's important to make sure you're using the same version of python across all the places you plan to develop and deploy your application. If you don't have the version locally that you want, you can use the Anaconda project to install and maintain it as well as act as a central repository for other python tools and documentation.
https://anaconda.org/
While you can install whatever version of python you want on your local machine and you can build whatever version of python runtime for XSA deployments, in CF you are limited to the versions the platform provides. So if CF is a required deployment destination, you'll want to check the version that is available on the landscape that you'll be targeting. To do this, run the cf buildpacks command and check the version of the buildpack against the Cloud Foundry site to get the list of python versions associated with that buildpack. As of this writing it's version 1.6.20.
cf buildpacks | grep python
On the us10 landscape, here's the output.
buildpack position enabled locked filename
...
python_buildpack 6 true false python-buildpack-cflinuxfs2-v1.6.20.zip
https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.6.20
From that page, I see that python version 3.6.5 is available.
So for the purposes of this post, I'll use python version 3.6.5.
Getting the hang of your local python installation with a PyDev project.
At the src folder create a new PyDev package called "pkg".
In the __init__.py file place this code. You may have to do a pip install flask to get the dependencies loaded locally in your system.
pip install flask
Paste this in your __init__.py file.
from flask import Flask
from flask import request
import os
app = Flask(__name__)
port = int(os.getenv("PORT", 9099))
@app.route('/')
def hello_world():
output = '<strong>Hallo World! I am instance ' + str(os.getenv("CF_INSTANCE_INDEX", 0)) + '</strong></br>\n'
return output
if __name__ == '__main__':
# Run the app, listening on all IPs with our chosen port number
# Use this version in production
#app.run(host='0.0.0.0', port=port)
# Enable server.py reload on changes
# This also returns server errors in the html output (security risk)
app.run(debug=True, host='0.0.0.0', port=port)
Run your python by right-clicking on the init file and picking Python Run.
Your local python server will start up and run the code. Notice the messages in the console.
Copy the URL to your browser and test.
Notice that there is a spelling error in the word hello.
Terminate the running python.
Now double-click next to line 10 and set a breakpoint.
Debug your python by "Debug As" "Python Run".
Reload your browser and notice that it seemingly hangs, return to eclipse.
Switch to the Debug perspective.
Notice not that code line 10 is highlighted and you can now do debugging tasks like stepping through the code and inspecting variables. Click the run arrow to continue from debugging.
Even without explicitly invoking the debugger, we can run the code and make changes that are automatically realized. This is due to passing the debug=True when starting the flask server.
Rerun the python. Edit the word "Hallo" to "Hello" and save the file.
Notice in the Console that the file was automatically reloaded.
Go back to your browser and reload the page. Now Hello is spelled correctly.
This is a feature of the Flask http server library and will work no matter where the python code is deployed. It's a handy first thing to try when debugging python, but be sure to turn it off when your code is ready for production.
Setting up a sample MTA project with python in eclipse.
Since there is currently no eclipse new project wizard for an MTA style project, the easiest thing to do is to use the WebIDE to frame up a sample project. If you don't have easy access to the WebIDE, then git clone this project into your eclipse by following these steps.
https://github.com/alundesap/mta_python_dev_env
Do a File -> Import...
Import a Git Project from Git.
Clone it from a URI.
Enter the repo URI.
https://github.com/alundesap/mta_python_dev_env
Select the master branch.
Confirm the local destination.
The repo has a .project file so it will be accepted as an existing Eclipse project.
Confirm and Finish.
You should now see the sample MTA project in your Project tab.
While eclipse doesn't by default help automate any of the development tasks, we can still edit the files and interact with the underlying git repo. Depending on our needs, this may be enough. However, we can more effectively work on our python module within eclipse by setting up some external tools that trigger the XS or CF command line interfaces.
Getting the XS Command Line Interface(CLI).
The easiest way to get the xs cli is to follow the
this post to get the HXE Download Manager and use it to download and install the Clients for your workstation platform. Also, if you don't have access to a HANA system it will show you how to run one on your laptop.
Setting up a HANA Express Python Machine Learning API Demo VM
The assumption here is that our python module will need to connect to a HANA database.
The MTA way of things requires that we create a HDI container by "deploying" our db module.
There are several scripts in the tools folder of the project repo that accompanies this blog post.
You can get a copy of the project repo by performing a git clone.
git clone https://github.com/alundesap/mta_python_dev_env.git
Note: Currently the scripts in the tools folder of the project assume that your workstation is Linux or Mac and expect the bash shell executable to be in
#!/bin/bash . If you are using a windows workstation, you'll need to modify the scripts to work in your environment. However, if you are running HANA in a VM on your workstation, you can clone the repo there and run the scripts within the Linux environment it provides.
Assuming that you have a HANA system with XSA up and running and available to you, we will run the tools/hdi_deploy script to build an HDI container to mimic what the WebIDE tool does during a build operation of an SAP HANA DB module.
If you don't have a HANA XSA system available, you can follow this blog post to get one set up.
Setting up a HANA Express Python Machine Learning API Demo VM
We'll assume that you have an XSA system available and that the desired python runtime version is built and installed and available for our application.
If needed, shell into your HANA system as the hxeadm user (for HXE based system) <sid>adm if you're installation used a different <sid> (System ID).
If you want to instead run the xs cli locally and work from the file location where your project was imported into git, then find that folder by looking a the Location in the project's properties.
Open a terminal in that folder.
Check that your xs cli is pointing to the proper api endpoint. .
xs api
For HXE VM systems it should look like this(ymmv).
API endpoint: https://hxehost:39030/
Get logged in with the XSA_ADMIN user.
xs login -u XSA_ADMIN
Make sure you're in the proper org and space. For HXE systems this will work.
xs target -o HANAExpress -s development
Double check that you're in the project folder top level. The files should look like this.
ls -1
README.md
db
mta.yaml
python
tools
web
Run tools/hdi_deploy script like this. This runs it in xs mode and processes the files in the db folder.
tools/hdi_deploy xs db
If all goes well you should see some output like this.
Follow the directions at least once in order to grant your SYSTEM user access to the HDI container you just created. You may need to install the HANA Client tools to run the hdbsql command on your local workstation. Look at the
tools page for instructions on how to get and install them. The client package for your architecture can also be downloaded with the
HXE Download Manager.
Now that we've deployed the HDI module from the command line, let's make an easy way to trigger it from within eclipse. To do this we'll define an external tool. From the run menu, pick External Tools, then External Tools Configurations...
Location:
${workspace_loc:/${project_name}/tools/hdi_deploy xs db}
Working Directory:
${workspace_loc:/${project_name}}
Arguments:
xs bs
Give it a Name: HDI Deploy
Select the environment tab and define variables JAVA_HOME, NODE_PATH, and PATH. These will be provided to the external tool (not your terminal).
Name: JAVA_HOME
Value: /Users/i830671/Documents/sapjvm_8
Name: NODE_PATH
Value: /usr/local/lib/node_modules
Name: PATH
Value: /Applications/XSA/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin
Click Apply then Run.
Now you can launch the HDI deploy script from your eclipse and see the output in the console.
Now let's move on to the python module.
Some of the libraries our python module uses are provided by SAP (not PyPi.org yet) so we'll need to get them.
Getting the MTA python helper libraries.
Note that you need a valid S-user to connect to the Software Downloads area.
https://launchpad.support.sap.com/#/softwarecenter
Search for PYTHON in the Downloads area(or use this link).
https://launchpad.support.sap.com/#/softwarecenter/search/PYTHON
Download the
XS_PYTHON00_1-70003433.ZIP file and then copy it into your project.
Unzip the file and then rename it sap_dependencies (matches the SAP documentation).
unzip XS_PYTHON00_1-70003433.ZIP -d sap_dependencies
In order to facilitate local testing of our demo app, let’s install the python libraries locally, but before we do that we need to make sure our command line environment is pointing to the proper version of python so that the libraries get installed in the right place.
Open Eclipse, Preferences and PyDev, Interpreters, Python Interpreter Libraries tab.
Notice that the python executable will be python3.6 and pip will therefore be pip3.6 as well. Your situation will differ but you can test it's version.
python3.6 --version
pip3.6 --version
Set your terminal's PYTHONPATH to the in your eclipse python interpreter folders. You'll have to adjust the paths to your system as the path here is from mine.
export PYTHONBASE=/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6
export PYTHONPATH=$PYTHONBASE:$PYTHONBASE/lib-dynload:$PYTHONBASE/site-packages
This is just so that pip will put the needed dependencies for our project in the right site-packages locally. Make sure you're in the project top level directory and then change into the python directory.
cd python
pip3.6 install -r requirements.txt --find-links ../sap_dependencies
Note: You may have to invoke pip3.6 with sudo.
Now, let's set up a python run configuration making sure to use this python environment. What we are going for is this scenario. Your python is edited and running locally and it's accessing HANA on a locally provided VM but doesn't incur the overhead of an xs deploy operation.
This situation provides the full debugging functionality of eclipse but in order to act like an MTA deployed module, we need to re-create the environment in which it runs. We can do this be providing the same environment variable that are available to it when it's deployed.
Setting up the Python Run Configuration in eclipse.
Open up the Run Configurations... dialog from the menu or clicking next to the run icon.
Give it the name devEnvPython, Browse... and select the project mta_python_dev_env_proj. For Main Module, click Browse... and expand python and select server.py.
Your configuration so far should look like this. Be sure the PYTHONPATH is the same as when we exported earlier.
Click on the Environment tab. Select New.. and enter Name: PORT and Value: 8089 or some other number you think is free on your workstation.
Select New... again and this time Name: VCAP_SERVICES. The value of the variable will be specific to your situation and you can get it by running the get_VCAP_SERVICES script in the tools folder. Make sure you are in top level of the project and then run this command.
tools/get_VCAP_SERVICES xs db
Cut and past the result or in the meantime, you can use cut and copy this as a placeholder.
{
"hana": [
{
"name": "python-dev-env-hdi",
"label": "hana",
"tags": [
"hana",
"database",
"relational"
],
"plan": "hdi-shared",
"credentials": {
"schema": "XXXX3DE4018F463E9A740E1C660F0809",
"hdi_password": "XXX0x5Z3zSQ_KUlfUGpK3vI1ZL9abl3y0MRDy",
"password": "XXX3s42T0kbaX5RS",
"tenant_name": "HXE",
"driver": "com.sap.db.jdbc.Driver",
"port": "39015",
"host": "hxehost",
"db_hosts": [
{
"port": 39015,
"host": "hxehost"
}
],
"hdi_user": "XXXX3DE4018F463E9A740E1C660F0809_3SMLZS1ZNU5C0AUOYMEM3P21A_DT",
"user": "XXXX3DE4018F463E9A740E1C660F0809_9WT5KJ5HENW1YL314PE0ETP3G_RT",
"url": "jdbc:sap://parvus.lcfx.net:30015/?currentschema=XXXX3DE4018F463E9A740E1C660F0809"
}
}
]
}
For convenience, you can Duplicate the HDI Deploy External Tools configuration and change the Location to end in tools/get_VCAP_SERVICES. That way you can stay in eclipse while setting up the python run configuration.
Your screen should look like this. Note that only the first line with the { will show in the Value for VCAP_SERVICES but it's there.
Now click Run. If all works well your python will be launched and you'll get the following in the console.
Browse to this URL. If you click on the /env link you'll see the value of VCAP_SERVICES you entered above.
You can set breakpoints and debug and inspect variables all while connecting to your HANA data source. This can be a great environment for doing core development of your python module, but you won't get the benefit of exercising it as a module of the larger MTA project. So you will have to disable request validations and since your requests won't be coming from an app-router, you won't be able to test things like authorization and scope logic. I recommend you mock this up in your code and work out the details when you can test your full MTA project in XSA or Cloud Foundry.
Deploying to XSA.
As we move where our python codes run further away from our local machine, we need to adapt our strategy a bit, but we can still do effective development and debugging. The next mode of development is to deploy our MTA project to the XSA system running on the local VM.
We can do this in one of two ways. The recommended way is to use an SAP provided command like tool to package our project as a single file, a Muti-Target Application aRchive(MTAR). The tool is provided as a java jar file that can be found on this page.
Although it's on the cloud tools page, it can be used to package for XSA or Cloud Foundry deploys.
One of the benefits of the MTA Builder is that it supports the MTA python module type and handles it correctly by collecting up all the files in the python folder into the archive. The MTAR bundling function of the WebIDE does not currently (as of this writing) perform this properly and will fail when creating an MTAR file (that contains a python module) by right-click building at the project level.
I find the MTA Builder tool cumbersome to invoke so I've created a wrapper script to call it more simply. Also this way you can adjust the wrapper script if the file name of the jar file changes and then you don't have to adjust your other tools to accommodate. Or you can rename the jar file to mta.jar.
Once you've downloaded it, place it in your project's tools folder and rename it to mta.tar. If you'll be using it for other projects, you'll want to put it somewhere in your path so that you can invoke it from any project, but for now we'll use it from the tools folder.
cd tools
mv mta_archive_builder-1.1.7.jar mta.jar
cd ..
Now you should be able to invoke it like so.
tools/mta --version
To package or not to package?
While most tutorials and examples provided by SAP package the project as an MTAR file and then perform a deploy on it to the target environment, this isn't explicitly necessary. As long as you perform the same steps manually that a deploy operation is doing, you will get the same result (short of the deploy being registered with the system). This is essentially what the scripts that we've been invoking from eclipse are doing.
Where to package?
You can set up the MTA Builder on your local workstation or on the XSA system (VM or other hosted environment). However you must be aware of the implications of each situation. When you set it up on your local workstation and run the pip packaging tool, it detects the local architecture and puts the architecture specific versions of the libraries in your project. Since you're likely to be on a Windows workstation or Mac workstation, the result are binary versions of the libraries that don't work on Linux once deployed. This is the reason it's handy to have an XSA(Linux) system available to you to perform the lib dependency assembling.
Assuming you have access to a Linux environment, ssh into it as a normal user and clone a copy of the project repo.
git clone https://github.com/alundesap/mta_python_dev_env.git
Change into the project folder and then into the python module folder. This is where the requirement.txt file that lists the dependencies resides. We will be "vendoring" our python dependencies. This means that we will use pip to grab all the required dependencies and place them in a local folder in our project. This folder is then picked up in the MTAR packaging process.
mkdir -p vendor
pip download -d vendor -r requirements.txt --find-links ../sap_dependencies
Once you've done this, you can move the vendor folder to your local workstation and perform deploys from there since the vendor folder contains the Linux versions of the dependencies.
cd ..
tools/py_deploy xs python
When the python is finished deploying, find it's url with this.
xs app python-dev-env_python --urls
Browse to the URL returned and test that the python is now running in XSA. You can test the database connection but not yet the authorization link since we haven't yet deployed the app-router.
Set up an external tool config for building the python module. Open the External Tools Configurations and duplicate the HDI Deploy configuration.
Rename it "Py Deploy". Click "Browse Workspace..." expand the mta_python_dev_env_proj and select the tools folder, then select py_deploy.
Also change the Arguments from "xs db" to "xs python". Your configuration should look similar to this.
Click Apply.
Note: As mentioned above and worth mentioning again here. When you trigger the python deploy from the local machine, the expectation is that the
target (not local) architecture libraries are in the vendor folder. Be sure you've copied the vendor folder from the Linux system where you downloaded the python dependencies.
Double check that this is the case by expanding the vendor folder and checking the hdbcli. It should have linux in the name not windows or macos.
Create an External Tools Configuration for the tools/web_deploy script and a NodeJS run configuration if you want to develop that module with eclipse as well. I won't get into the NodeJS specifics, but you should be able to follow a similar approach as we did with the python. The web_deploy script takes three arguments, "xs web python" the last being a module to set an internal route to.
With the app-router running and requiring authorization, you can exercise the request routing in the python module.
Building and Deploying the MTA project.
Up to this point, we have been pushing individual modules of our project into XSA. I like to call this piecemeal deployment(my own term). The scripts I've provided do all that's needed to get the job done while allowing you to trigger them from eclipse and save time by only deploying one module at a time. You'll eventually want to build and deploy the whole project at once into an MTAR file. To do this, use the MTA builder utility we set up earlier. The MTA builder will pick up the mta.yaml file in the current folder and use it to carry out the deployment of the individual modules in the proper order and create the required service instances and bind the modules to them. We'll create a folder called target to store the resulting .mtar files.
mkdir -p target
tools/mta --build-target XSA --mtar target/python-dev-env.mtar build
Now that the XSA version of the mtar file had been created, deploy it with the xs deploy command.
xs deploy target/python-dev-env.mtar --use-namespaces --no-namespaces-for-services
We use the
--no-namespaces-for-services parameter so that our existing services will be used.
Once you get your xs deploy working to your satisfaction, you can create an External Tools Configuration to make it more easily triggered from within eclipse.
If all goes well you should be able to see your app as a deployed mta.
xs mtas
I you align your module names to that produced by the xs deploy operation, you can mix both the deployment method and the scripts triggered from eclipse. It's often handy to first do a full deploy of your project, then work on the python module from eclipse.
To undeploy the project, use the xs undeploy command.
xs undeploy mta_python_dev_env
Debugging with your python deployed to XSA.
Even when your XSA system in running in a VM on your workstation, your python code is not directly available to eclipse for debugging. The most basic form of debugging is to watch the logs of the running python module. Use the xs logs command for this.
xs logs python-dev-env_web
This may give you a hint at what's failing but the problem may also be deeper.
In this sample project the python module is using the Flask library to provide its web server(every microservices module needs one). The Flask library provides a mechanism to reload itself if any of the python code is changed. If you look at the last line of the python/server.py file, you'll see where it's specified.
Note: Be sure to disable Flask debugging before deploying your code to production.
app.run(debug=True, host='0.0.0.0', port=port)
When a project is deployed to XSA, the system uses process separation to isolate each module from the system and each other. It can be tricky to figure out where the deployed files currently reside.
If you're using a HANA Express system, this should work for you.
Note, you have to be ssh'd into the server for this to work!
echo "Changing to running app directory." ; pyguid=$(xs app python-dev-env_python --guids | grep RUNNING | tr -s " " | cut -d " " -f 12) ; echo $pyguid ; cd /hana/shared/HXE/xs/app_working/hxehost/executionroot/$pyguid/app
Otherwise, you'll need to replace HXE with your <sid> hxehost with your hostname.
Once your in the deployed app's folder you can edit the server.py file and save it. Automagically your python will be reloaded with the new changes. This is very effective to trying things out quickly but you need to be aware of a couple pitfalls. First, understand that any changes will be lost if your app is reloaded. So if you discover what code needs to change to make things work, copy/paste it into your eclipse server.py file for the next deployment. Second, you have to run the above each time you do a deploy or your app gets restaged(like when you inadvertently cause a seg-fault).
A better approach is to edit the files in eclipse then let a process pick up the changed file and send it into the correct place where your python module is deployed in XSA. This approach is described for NodeJS running Cloud Foundry in this blog post.
For this you'll need to ssh into your HXE server as the hxeadm (or <sid>adm) without a prompt for a password. There are various ways to set this up, but I usually use the ssh-copy-id command.
ssh-copy-id -i ~/.ssh/id_rsa.pub hxeadm@hxehost
Once you can ssh in with getting prompted for a password, we need to use the ssh client to form a reverse tunnel connection to the app running in XSA. First get your eclipse in a debug listening state.
Switch to the debug perspective. Manually run the debug server.
You should see this in the eclipse console window.
Debug Server at port: 5678
Now use ssh to create the tunnel from the server to the local workstation on port 5678.
ssh -nNT -p 2009 hxeadm@hxehost -R 5678:localhost:5678
What this command does is create a TCP tunnel from the remote hxehost on port 5678 to the local workstation on port 5678.
Now that we have our eclipse based debugging server (our workstation) listening for debugging connections and a tunnel from our XSA server to our workstation, we need our python application to "reach out" to debugging server and initiate a connection. To to this we add a debugging library to our python module and call a function to accomplish this.
First we need to make sure the pydevd library is listed in our requirements.txt file.
Then we need to define a function to call it in our python file.
def attach(port, host):
try:
import pydevd
pydevd.stoptrace() #I.e.: disconnect if already connected
# pydevd.DebugInfoHolder.DEBUG_RECORD_SOCKET_READS = True
# pydevd.DebugInfoHolder.DEBUG_TRACE_BREAKPOINTS = 3
# pydevd.DebugInfoHolder.DEBUG_TRACE_LEVEL = 3
pydevd.settrace(
port=port,
host=host,
stdoutToServer=True,
stderrToServer=True,
overwrite_prev_trace=True,
suspend=False,
trace_only_current_thread=False,
patch_multiprocessing=False,
)
except:
import traceback;traceback.print_exc()
...
attach(5678,"localhost")
...
Redeploy the python and then browse to it's URL.
In order for remote debugging to work, the python source file must be BOTH deployed on the server and in eclipse and they must be identical or the step tracing won't work. Also, the pydevd library must be told paths to the source files on the server and in eclipse. You can edit the pydevd_file_utils.py file on the server, but it's difficult to find once deployed. The easier way is to set a special environment variable that the pydev library picks up. Since XSA uses process separation, you have to set the environment in the same context that the app is running. To do this I've defined a path in the python to take a query param and set it into the environment.
https://<your host:port>/python/set_env?PATHS_FROM_ECLIPSE_TO_PYTHON=[["<path to files on eclipse>","."]]
You'll have to adjust it to your situation.
Check it with this.
https://<your host:port>/python/env
When things look OK, start the debug server, make sure your tunnel is up, and attach the running app to the debugger.
https://<your host:port>/python/attach
Now you can set breakpoint and exercise the code to hit it and start debugging.
Deploying to Cloud Foundry.
Now we move to our final deploy destination, Cloud Foundry. In this scenario we will deploy our python module into Cloud Foundry while retaining the same debugging functionality that we've been using so far. Here's what the scenario looks like. Notice in this case the local workstation has a minimal tool footprint an may be the only option for those who can't run HANA in a VM locally.
Flask debugging on Cloud Foundry.
In order to edit your CF deployed python module, you can use the cf command line tool. In Cloud Foundry, each module is run in it's own container that provides a ssh server with it. We can leveraget this with the cf ssh command. First you need to enable your deployed python module for ssh connections.
If you don't have the cf cli program and mta cf plugin, download and install them for your platform by following the instructions here.
Also, you can install the Cloud Foundry eclipse plugin.
Make sure you're connected to the proper CF landscape. I have an account in us10.
cf api https://api.cf.us10.hana.ondemand.com
Login with your SAP SSO credentials.
cf login -u cf_user@somewhere.com
Pick your org and space where you have a HANA As A Service instance running.
For details in setting up HaaS, check here.
SAP Cloud Platform, SAP HANA ServiceGetting Started Guide
Run the create_uaa, create_hdi, hdi_deploy, py_deploy, and web_deploy scripts.
tools/create_uaa cf <org> <space>
tools/create_hdi cf <org> <space>
echo "Customize your hdi_deploy,py_deploy and web_deploy scripts if you are NOT using HXE.
tools/hdi_deploy cf db
tools/py_deploy cf python
tools/web_deploy cf web
Once you know they are working well for you, you can create eclipse External Tool Configurations for them so that you can trigger them from within eclipse.
cf enable-ssh python-dev-env_python ; cf restart python-dev-env_python
Now you should be able to cf ssh into the container running your python.
cf ssh python-dev-env_python
You should get a new prompt that starts with vcap and ends with :~$ and looks like this.
vcap@75b874d3-b4e8-4723-6fe5-eb0f:~$
You can edit your server.py file and immediately test the results upon saving the file.
vi app/server.py
Exit to return to your local shell.
exit
Remote debugging on CF.
Now that we know how to ssh into our python module's container, how do we set up the tunnel to our workstation like we did for XSA?
First as above, we need to set the PATHS_FROM_ECLIPSE_TO_PYTHON in the containers environment. Use the python app URL like so.
https://<python-dev-env-python>.cfapps.us10.hana.ondemand.com/python/set_env?PATHS_FROM_ECLIPSE_TO_PYTHON=[["<path to files on eclipse>","/home/vcap/app"]]
Double check with python/env.
https://<python-dev-env-python>.cfapps.us10.hana.ondemand.com/python/env
Now we need to use the native ssh client in order to form a reverse tunnel. The cf ssh command only allows a local tunnel(-L) not a reverse(-R). To do this we get a one-time password from cf and use that to login via ssh to our container where our python module is running. To for the ssh command properly we need the python module's guid. Get it like this.
cf app python-dev-env_python --guid
Now use this command but replace xxxxx5b4-e94b-4681-9ee5-ca334f1xxxxx with the value you just got. Also make sure to change the landscape if you're not using us10. The /0 means the zeroth instance of your module. You will be prompted for the password. I've chained the commands together since you only get one chance to enter it correctly.
cf ssh-code ; ssh -nNT -p 2222 cf:xxxxx5b4-e94b-4681-9ee5-ca334f1xxxxx/0@ssh.cf.us10.hana.ondemand.com -R 5678:localhost:5678
If everything works right, the console will just hang there. The tunnel is now open and you can kill it with Ctrl-C.
Go back into eclipse and switch to the Debug perspective. Under Pydev, start the Debug Server.
And finally, browse to python/attach to get the python app to initiate a connection with the eclipse debug server running on your workstation.
https://<python-dev-env-python>.cfapps.us10.hana.ondemand.com/python/attach
A debug connection should appear in the Debug window in eclipse.
Clear the existing breakpoints and set a breakpoint on a line in the server.py file. I put one in the handler for the /python/env route.
No reload the window with the python/env path. It should seemingly hang the browser. Switch back to eclipse. You should now see that the line is highlighted and you can continue to step forward and inspect the variables in your python code.
Stop the debugger server when you're finished.
Final thoughts.
While I've been using python as an example language, eclipse can be configured to support any number of other development languages. In cloud foundry, check to see what runtimes are installed and what versions are supported. As of this writing, here's what us10 is reporting.
Fir instance, for PHP consult the Cloud Foundry release docs page.
https://github.com/cloudfoundry/php-buildpack/releases
Look for the release version in the filename. For php it's v4.3.59
https://github.com/cloudfoundry/php-buildpack/releases/tag/v4.3.59
Default binary versions:
I hope you've found this rather long post to be helpful. The accompanying MTA example project can be found here.
git clone https://github.com/alundesap/mta_python_dev_env.git
Please alert me to any errors and/or omissions.
Thanks,
-Andrew