FAQs for SAP Data Hub, developer edition

schneidertho · ‎2017 Dec 06

This page is used to collect frequently asked questions related to SAP Data Hub, developer edition. We will update it regularly.

Where do I find the SAP Data Hub, developer edition?

You find an overview in this blog post. And you can get started by looking at this tutorial.

Can the developer edition be used with Docker Toolbox?

Yes, you can run the SAP Data Hub, developer edition with Docker Toolbox (although we recommend to use Docker for Windows respectively Docker for Mac, since Docker Toolbox is considered a legacy solution).

Ensure that the Linux VM used to run Docker is sufficiently sized (at least 2 cores, recommended 4 cores, 8 GB of RAM). You can adjust the sizing via Virtualbox.

Publishing the services running inside the Docker container to 127.0.0.1 (localhost) does not work. Instead you can publish and access them to the docker-machine ip.

ERROR: could not handle api call, failure reason : execution of scheduler plan failed

If this error occurs while storing data in SAP Vora (during turorial Store sensor data in SAP Vora), check that you have correctly maintained the connection between the Kafka Consumer operator and the ToString Converter operator.

You need to connect the message port of the Kafka Consumer operator to the ininterface port of the ToString Converter operator.

Where do I find logs inside the Docker container?

You find the logs related to SAP Data Hub (and SAP Vora) inside the Docker container in the directory /var/log/vora.

How can I access SAP Data Hub, developer edition from "other" hosts?

You have set up SAP Data Hub, developer edition on a host different from your local computer or inside a VM (via Docker Toolbox). You cannot access SAP Data Hub.

You are running the Docker container for SAP Data Hub, developer edition with docker run -ti --publish 127.0.0.1:8090:8090 --publish 127.0.0.1:8998:8998 --publish 127.0.0.1:9225:9225 --publish 127.0.0.1:50070:50070 --name datahub --hostname datahub --network dev-net datahub run --agree-to-sap-license --hdfs --livy as per this tutorial.

When using this tutorial to set up SAP Data Hub, developer edition you bind the container ports to the loopback IP address (aka “localhost") to restrict access. Hence you can only access it on the host it is set up on.

For accessing SAP Data Hub, developer edition in conjunction with Docker Toolbox, look at “Can the developer edition be used with Docker Toolbox?”.

For accessing SAP Data Hub, developer edition on “another” host (different from your local computer), you can do without binding the container ports to an IP address. For details refer to the README file which you have downloaded with the developer edition. Keep in mind: SAP Data Hub, developer edition is not intended to provide a multi-user environment.

ERROR: No matching manifest for unknown...

You try to use SAP Data Hub, developer edition in conjunction with Docker for Windows and get an error "No matching manifest for unknown...". Please ensure that you use Linux (and not Windows) containers.

cbethune · ‎2017 Dec 29

Hello, I was able to successfully setup SAP Data Hub as covered in the tutorial at https://www.sap.com/developer/tutorials/datahub-docker-setup.html and the status loop seems to be running as expected.

However, I am not able to access any of the URLs from step 5 Smoke Test

http://localhost:8090 (SAP Data Hub - Data Pipelines)
http://localhost:9099 (Apache Zeppelin)
http://localhost:9225 (SAP Vora Tools)
http://localhost:50070 (Apache Hadoop User Interface)

I suspect there are some changes I need to make in the Virtual Box network settings.

I did notice that an IP of 172.18.0.2 is created in the Docker containter while observing the status loop so I tried with this IP instead of localhost but still get a "Site Cannot Be Reached - ERR_CONNECTION_REFUSED" error when trying to navigate to the URLs.

I have also tried changing the network adapters one at a time to use Bridged Mode. The default settings created in Virtual Box are:

Adapter 1: NAT

Adapter 2: Host Only Adapter (Virtual Box Host Only Ethernet Adapter #3)

Any advice on what I need to modify so that I am able to access the URL's and start working with Data Hub?

I am using Docker Toolbox since I am running Windows 7.

ayman_salem · ‎2017 Dec 30

in Host Only Adapter (Virtual Box Host Only ...) check which address you use, and also check that Host Only Adapter is active



And when you run the docker you must use --publish ex_port:port,  (ex: --publish 50001:8090: Here the port 8090 of the container will be mapped to port 50001).

Then, to access the "Data Hub" use http://192.168.99.100:50001

I hope that will help you.

former_member187175 · ‎2018 Jan 03

Hi Chris,

let me know if Ayman's solution works.

Here is what worked for me based on changes to the documented step 4:

1. find the docker-machine ip address:

$ docker-machine ip

192.168.99.100

2. Publish using the docker-machine ip address:

$ docker run -ti --publish 192.168.99.100:8090:8090 --publish 192.168.99.100:9099:9099 --publish 192.168.99.100:9225:9225 --publish 192.168.99.100:50070:50070 --name datahub --hostname datahub --network dev-net datahub run --agree-to-sap-license --hdfs –zeppelin

Now these work:

http://192.168.99.100:8090/

http://192.168.99.100:9099/#/

http://192.168.99.100:9225/web/

http://192.168.99.100:50070/

Regards,

Serge

ayman_salem · ‎2018 Jan 04

I just want to add that the "IP address 192.168.99.100" was assigned by the DHCP server it may vary in another installation of VirtualBox.

make sure that the DHCP server is active for the "VirtualBox Host-Only Adapter" that you are using for the "default" VM.

So, you can also run the datahub without IP in the command.

here's what I'm using:

$ docker run -ti --publish 50000:9225 --publish 50001:8090 --publish 50002:9099 --publish 50070:50070 --publish 8500:8500 --publish 14000:14000 --name devedition --hostname devedition --net dev-net datahub run --agree-to-sap-license --hdfs --zeppelin

Regards,

Ayman

cbethune · ‎2018 Jan 11

I ended up getting it running using your process here and the issued IP address. I may either set it to permanent or reconfigure for DHCP. First I need to try and acquire more RAM for my laptop so I can run it without risking a crash. I wanted to prove out I could get it running before investing in hardware upgrades. 🙂

cbethune · ‎2018 Jan 11

Thanks all for the suggestions. Note my reply above, I was able to get it running finally.

Abdelkader · ‎2018 Jan 23

Hi Thorsten,

Thanks for the tutorial, I was able to run it until the last

step :https://www.sap.com/developer/tutorials/datahub-pipelines-storeinvora.html

where i get the following message :

"Error during init of 'com.sap.vora.hdfsLoader': failed to ping SQL connection with DSN v2://vora:2202/?binary=true: timeout. Please ensure host is up and reachable"

It seems that port 2202 is not working. any hint how to debug this issue?

Thanks

Abdelkader

ayman_salem · ‎2018 Jan 23

Hallo Abdelkader,

you can check the status of the Vora services under http://<host/ip address>:8500

if all services are running, then check the firewall on your system (disable it or allow port access)

Ayman

DeepakVarandani · ‎2018 Mar 28

Hi Thorsten,

I have installed SAP Data Hub, developer Edition on my Windows 7 (8GB RAM) laptop using docker toolbox. I have successfully build the docker image of Dev edition, but not able to run docker container based on above datahub image. Please find below error screenshot for your reference.

 





Regards,

Deepak Varandani

schneidertho · ‎2018 Apr 05

Hi Deepak,

hard to say what goes wrong there. How much CPUs/RAM have you given to the Linux VM (see also question "Can the developer edition be used with Docker Toolbox?")?

Are you using the newest version of SAP Data Hub (i.e. when did you download it)? And does the problem stay the same even when you try a second time / after restarting your computer?

Cheers

Thorsten

Former Member · ‎2018 Apr 24

Hi Thorsten,

I am trying to install DH Developer Edition on my desktop using Docker but I am not able to build the image. See the error on the screenshot.

I hope you can help me, Thanks a lot.

David

schneidertho · ‎2018 Apr 24

Hi David,

two questions

1) which operating system / Docker version are you using?

2) do you build the image from inside a corporate (and have you set the necessary proxy in the Dockerfile)?

Cheers

Thorsten

Former Member · ‎2018 Apr 24

Thorsten I am using Windows 10 Pro with Docker version 18.03.

About the second question I am using my personal computer, I am not inside a corporate and I have set the necessary proxy in the Dockerfile using:

ENV http_proxy=http://myproxy:8080

ENV https_proxy=http://myproxy:8080

Thanks

schneidertho · ‎2018 Apr 26

Hi David,

but you go via a proxy? Have you tried without setting the proxy?

If that does not help, can you somehow (mail?) send me the complete log of the build process? Thanks.

Best regards

Thorsten

Former Member · ‎2018 Apr 27

Thorsten I think you can't post your mail here but could you send me a mail to df.boada914@gmail.com so I can send you the complete log of the build process

Thank you very much.

former_member468744 · ‎2018 May 04

Hi,

I downloaded the latest version 1.2 but I'm getting an error using the following command:

docker build --tag datahub_1.2 --build-arg VORA_USERNAME=vora --build-arg VORA_PASSWORD=******** .

2018-05-04T22:08:49+0000 -------- executing LIVY_download --------

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to mirror.netcologne.de port 80: Connection refused

2018-05-04T22:08:49+0000 ERROR: Couldn't download Apache Livy

The command '/bin/sh -c /dev-edition-helper.sh LIVY_download' returned a non-zero code: 1

Kindly advise

cheers

G.

former_member468744 · ‎2018 May 05

ok, I solved it on my own.

I changed the file dev-edition-helper.sh in the directory DatahubDevEdition/files and replaced the existing line for the LIVY_URL with some other mirror from https://www.apache.org/dyn/closer.lua/incubator/livy/0.5.0-incubating/livy-0.5.0-incubating-bin.zip

For example:

#   LIVY_URL="http://mirror.netcologne.de/apache.org/incubator/livy/${LIVY_VERSION}/livy-${LIVY_VERSION}-bin.zip"

   LIVY_URL="http://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/incubator/livy/${LIVY_VERSION}/livy-${LIVY_VERSION}-bin.zip"

After that the build runs.

Cheers

Gerd

schneidertho · ‎2018 May 15

Hi Gerd,

the mirror had a problem on the last weekend. Cheers

Thorsten

sap_sharad · ‎2018 Jun 10

Hi Thorsten,

Thanks for creating such a wonderful blog.

I am stuck at the last step of the blog "Store sensor data in SAP Vora". After executing pipeline, I am getting error " Error during init of 'com.sap.vora.hdfsLoader': failed to ping SQL connection with DSN v2://vora:2202/?binary=true: timeout. Please ensure host is up and reachable".

Kindly advise.

Thanks.

sap_sharad · ‎2018 Jun 10

I solved the issue. Increased no of cpu for VM.

nguelokt · ‎2018 Oct 24

aov.thorsten.schneider

I am trying to create a Docker File in the SAP Data Hub Pipeline Modeler. I have created the folder and gave a name for my Docker File. The Docker File Build is started, but i am recieve an error that the docker image cannot build then “Cannot connect to the Docker daemon at unix:///var/run/docker.sock”.

Do you have a suggestion to slove this issue?

Thanks,

Tatiana

schneidertho · ‎2018 Oct 24

Hi Tatiana,

at the moment you are not able to do this with the developer edition. Normally (i.e. not in the developer edition) we will spin up containers when running pipelines. These will run on Kubernetes and they will be the runtime environment for the operators.

With the (current) developer edition we only deliver a single docker image / container. You are not able to define your own Dockerfiles for new operators. Simply spoken the container which runs the developer edition itself is also the runtime environment for all operators.

If you like to build own operators AND create dedicated Docker files for these, please use our trial edition.

We know that this behavior is not ideal, but it will stay like this at least for 2.3 / 2.4.

Sorry!

Cheers
Thorsten

nguelokt · ‎2018 Oct 31

Thanks for the Information!

jcmendez · ‎2019 Jan 17

Hi.

I'm getting this same error.

Is Data Hub staying like this even with Developer Edition 2.3?

I'm trying to build my own operator and it is not possible.

Thanks.

schneidertho · ‎2019 Jan 18

Hi,

yes. This will stay like this at least for 2.3 and 2.4. You can alternatively look at the SAP Data Hub, trial edition which does not have this restriction.

Best regards
Thorsten

former_member574535 · ‎2019 Mar 19

Hi,

If I want to use pandas/numpy/scikit-learn python libraries, I need to first create a docker file in the datahub. If the datahub developer edition does not support creating new docker file, is there any workaroundthat allows me to use these python libraries?

Thanks!

Hobart

schneidertho · ‎2019 Mar 19

Hi Hobart,

you can try to install the libraries directly into the Docker OS, i.e. you exec bash inside the container and do the installation.

Cheers
Thorsten

former_member607843 · ‎2019 Apr 09

Hey Thorsten,

after installing everything and getting it to run on Docker for Linux, I integrated a Hadoop Cluster to Read and Write the Data to.

Now I would love to try the Leonardo MLF Client, but when it comes to configuring it with the rigth API’s from the SAP Business Hub I really am not able to make it work.

Any hints or documentation you could refere me to?

Many thanks in advance!

Julian

schneidertho · ‎2019 Apr 09

Hey Julian,

question: I think we have an example graph for MLF (I am not sure, if it is delivered as part of the developer edition... need to check, but will not find the time today).

Will try to look into or ask somebody to look into.

BR

Thorsten

former_member607843 · ‎2019 Apr 10

Hey Thorsten,

Thank you for looking into it!

Looking forward to hearing from you

Julian

terry_penna · ‎2019 Apr 19

Running Set up for SAP Data Hub, developer edition 2.4 tutorial and I am getting an error when I run the following docker command per the tutorial:

PS C:\SAP\dh_dev2_4\DatahubDevEdition> docker build --tag datahub .
Sending build context to Docker daemon 1.028GB
Step 1/14 : ARG BASEIMAGE="sapdatahub/dev-edition-base:15.0-01"
Step 2/14 : FROM ${BASEIMAGE}
Get https://registry-1.docker.io/v2/sapdatahub/dev-edition-base/manifests/15.0-01: unauthorized: incorrect username or password

I am logged onto the website https://developers.sap.com/tutorials/datahub-docker-v2-setup.html and in those steps it does not give me a user name and password and the docker command does not ask for one?

schneidertho · ‎2019 Apr 28

Hi Terry,

no imediate idea.... I just tested on my laptop and all works as expected.

You should not need a username / password.

If the problem is still there, can you give some more details about the environment you are using? Windows 10... with Docker for Windows?

What happens when you run "docker pull sapdatahub/dev-edition-base:15.0-01"?

Cheers
Thorsten

i033659 · ‎2019 Apr 30

Hello,

This works for me as well. But how could I use the hostname instead of the IP-address of the docker machine? Using the hostname throws the error

docker: invalid publish opts format (should be name=value but got '<hostname>:8090:8090')

Thx, Ingo