Solved: Missing information in SAP Signavio Process Intell...

MKreitlein · ‎2022 Jul 15

Dear colleagues (especially directed to SAP Signavio contacts),

I thought about writing a Blog first, but since there are still unsolved questions, I thought it is better to place it here.

I've been working with "try and error" quite a while now to get an ETL setup in Signavio Process Intelligence up and running to an on-prem system (S/4HANA) ... and in parallel I've been wondering why the so-called "documentation" is purely text-based without any helpful image?!

First of all... and this is the weirdest part of it all: SAP Signavio is also a "Process Modeling" Tool... why is there not a single process chart throughout the whole documentation? You don't use yourselves what you preach 😞

Second... I participated in two 4x4hours bootcamps so far, and still the technical ETL setup for connecting to a real SAP system has still not been part of the agenda... is there any plan in the future, when this will happen?

Third... it took me also weeks, and the course SIG220 and the 2nd bootcamp to verify, until I realized that creating an Extractor in SIG PI is not possible if you don't open an incident and request it. The menu item is just not there 😞 Wondering already, for what reason you have to request the ETL feature set (which at least is described in the "documentation") and now separately the extractor menu item - which is not described in the "documentation".

Speaking of it... what is also not described there is the exact spots of these two topics:

I've been writing to "SAP Signavio Docs" (the feedback mail address at the bottom of each page), twice during the last 5 weeks, to clarify the missing information, and to please update the page with it ... and also asking if I need to open an Incident for it... Answer:

Unfortunately, we didn’t get the necessary information from the person in charge yet. (after 5 weeks)

We forwarded your emails to make the urgency clearer.

We apologize for not being clearer in our initial reply about the timeline. (right, which timeline?)

We also have no insight in the ways available to you for escalating this, and unfortunately we weren’t able to find another point of contact for you to get the information.

Hm, hard to believe.

By the incident about the missing extractor menu item, I learned that regarding the timeline, there are obviously only quarterly documentation releases and that it may take until Q3 for an update 😞 ... too bad.

Last but not least: You are probably curious how far I got with the On-Prem Extraction?

- In Signavio Process Intelligence the setup is prepared as far as possible... this worked better than expected.

- The Windows Server has been setup by my server colleagues, where the Extraction setup needs to be done, now.

And here I'm stuck at the moment, without any hint regarding the error message:

Proxy configured: true
S3 Client connection pool initialized with poolSize 100 and timeout 60000 milliseconds
Started PiEtlExtractorApplicationKt in 7.409 seconds (JVM running for 9.217)
Can't establish websocket connection to server (retries = 1, next attempt in 1000 ms)

And this goes on and on, until you stop it.

Via wikipedia I found a small script to call the websocket... after the opening: socket.onopen = function ()

I directly see the socket.onerror = function (errorEvent) result.

No idea why, since I was told that the internet proxy is working and in Chrome we also could see:

If any one has an idea or a hint or a solution how to overcome this error, please feel free to share.

And also if you find any better documentation with potential error messages, or guidelines, please let me know.

Ah finally, before I forget: I also tried to find a Guided Answer for this Signavio ETL.... and indeed you find 14 "matches" if you type in "Signavio"... but I think you will guess it: Not a single one exists, which provides guided answers for troubleshooting connection setup issues in this topic 😞

@SAP: In my opinion, there is plenty of room for improvement.

Thanks for reading,

Martin

dgohel · ‎2022 Sep 27

Hi m.kreitlein ,

Caused by: java.lang.IllegalStateException: Unable to obtain the schema for table: T001W

The above error tells that the table "T001W" is not configured in the Data Source, and thats why the application is not able to find it.

MKreitlein · ‎2022 Oct 04

Hello Darshan,

thanks for the info ... it seems it could have been the root cause, if you can confirm that an once triggered extraction cannot retrieve the table information anymore, if it was included afterwards?

We added the T001W into the Datasource afterwards (we found out, we missed it first), but it continued to failed.

I tried the whole ETL process once more today, and finally it worked.

I would recommend to include this issue into this page as well:

https://documentation.signavio.com/suite/en-us/Content/process-intelligence/etl-troubleshooting-onpr...

Thanks and BR,

Martin

MKreitlein · ‎2022 Sep 21

Hello Darshan,

my OSS ticket has been and still is in progress at SAP Development for more than two weeks now 😞

The full log is the following... the strange thing is, in Procure-to-Pay, the T001W extraction worked fine, we have only 21 records there

c.s.p.e.e.o.EtlWebSocketClientService    : Received test connection request for SAP
c.s.p.e.e.s.s.ExtractionServiceImpl      : Testing the connection against source system: SAP
c.s.pex.sap.driver.SignavioSAPERPDriver  : Init driver
c.s.p.s.driver.SignavioSalesforceDriver  : Init driver
c.s.pex.sap.driver.SignavioSAPERPDriver  : Init driver
c.s.p.e.e.s.s.ExtractionServiceImpl      : CONNECTION SUCCEED!!!!
c.s.p.e.e.o.EtlWebSocketClientService    : Test connection request for SAP took: 1 seconds
c.s.p.e.e.o.EtlWebSocketClientService    : sending TestConnectionRequest reply to ETL bridge
c.s.p.e.e.onprem.EtlStompSession         : 7d65a2af-05df-d380-8bb5-0ea8645c6e75 true - message: success
c.s.p.e.e.o.EtlWebSocketClientService    : async handling ExtractionRequest
c.s.p.e.e.o.EtlWebSocketClientService    : Received extraction request for SAP
c.s.p.e.e.s.s.ExtractionServiceImpl      : Calling extract for table T001W with SQL: SELECT "MANDT" as "mandt", "WERKS" as "werks", "NAME1" as "name1" FROM T001W
, and parameters {}
c.s.p.e.e.o.EtlWebSocketClientService    : ExtractionJob failed | source-system: SAP
| exception: java.lang.IllegalStateException
| cause: null
| sql-query: [SELECT "MANDT" as "mandt", "WERKS" as "werks", "NAME1" as "name1" FROM T001W
]
| query-params: {}
| anonymize-columns: []


com.signavio.pi.etl.extractor.shared.exception.EtlException: ExtractionJob failed | source-system: SAP
| exception: java.lang.IllegalStateException
| cause: null
| sql-query: [SELECT "MANDT" as "mandt", "WERKS" as "werks", "NAME1" as "name1" FROM T001W
]
| query-params: {}
| anonymize-columns: []

        at com.signavio.pi.etl.extractor.shared.services.ExtractionServiceImpl.extractionJobException(ExtractionServiceImpl.kt:120)
        at com.signavio.pi.etl.extractor.shared.services.ExtractionServiceImpl.extract(ExtractionServiceImpl.kt:108)
        at com.signavio.pi.etl.extractor.onprem.EtlWebSocketClientService.extractionJob(EtlWebSocketClientService.kt:103)
        at com.signavio.pi.etl.extractor.onprem.EtlWebSocketClientService$$FastClassBySpringCGLIB$$c406d802.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
        at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Unable to obtain the schema for table: T001W
        at com.signavio.pi.etl.extractor.shared.services.TableSchemaRetriever.retrieveSchema(TableSchemaRetriever.kt:15)
        at com.signavio.pi.etl.extractor.shared.services.ExtractionServiceImpl.executeExtraction(ExtractionServiceImpl.kt:136)
        at com.signavio.pi.etl.extractor.shared.services.ExtractionServiceImpl.extract$getExtraction(ExtractionServiceImpl.kt:88)
        at com.signavio.pi.etl.extractor.shared.services.ExtractionServiceImpl.access$extract$getExtraction(ExtractionServiceImpl.kt:42)
        at com.signavio.pi.etl.extractor.shared.services.ExtractionServiceImpl$extract$1.invoke(ExtractionServiceImpl.kt:101)
        at com.signavio.pi.etl.extractor.shared.services.ExtractionServiceImpl$extract$1.invoke(ExtractionServiceImpl.kt:100)
        at com.signavio.pi.etl.shared.metrics.MetricsService.runWithTimer(MetricsService.kt:86)
        at com.signavio.pi.etl.shared.metrics.MetricsService.runWithTimer$default(MetricsService.kt:84)
        at com.signavio.pi.etl.extractor.shared.services.ExtractionServiceImpl.extract(ExtractionServiceImpl.kt:97)
        ... 11 common frames omitted

The main issue is this... and I have no idea that schema cannot be obtain, while all others could:

Caused by: java.lang.IllegalStateException: Unable to obtain the schema for table: T001W

BR, Martin

MKreitlein · ‎2022 Sep 05

Hello darshangohel,

yes, the old VC++ 2010 version solved my problem - see the posting, right above yours (I marked it as best answer, but then it got out of sort order, so I un-marked it again).

As of now we achieved loading the Invoice-to-Cash and Procure-to-Pay data into Sig. Proc. Intelligence.

The extraction of Order-to-Cash constantly fails with the very last table T001W:

Caused by: java.lang.IllegalStateException: Unable to obtain the schema for table: T001W

2022-09-01 11:52:11.865 DEBUG 6316 --- [        task-11] c.s.p.e.e.onprem.EtlStompSession         : 1faa9d90-8060-766d-e02f-8d9d26ff9de7 true - message: ExtractionResponse(job=JobInfo(executionId=de2259d6-c926-413a-9b5a-c5a144e158cf, jobId=0a963ad1-c8d0-4ed2-b74b-13a660322387, status=ERROR, tenantId=fb7d501fdbd545cc87cafec3aad9a4f3), message=ExtractionJob failed | source-system: SAP
| exception: java.lang.IllegalStateException
| cause: null
| sql-query: [SELECT "MANDT" as "mandt", "WERKS" as "werks", "NAME1" as "name1" FROM T001W]

The strange thing is... the preview works, and the same table has already been extracted successfully in the Procure-to-Pay process ...

As of now, no idea what a schema is that cannot be obtained, where to get it, where to create it, etc... the error message does not give any hint. And it is only 21 records in this table 😞

Also the official documentation lacks this error message.

BR, Martin

dgohel · ‎2022 Sep 05

Hi m.kreitlein,

Thanks for the information. Recently couple of our users have fixed the issue by doing 2 things,

Change the location of the "TMP" folder in the "start.bat" file (Line 54), which might causing the access rights issue

SET ETL_TMP_DIR=%USERPROFILE%\etltmp

Install proper version of MS VC++ Redistribution according to your Windows Server.

Let me know if this works for you.

Thanks

MKreitlein · ‎2022 Aug 30

Hello everyone, and also darshangohel

there are good news right now 🙂

Your posting made me think of... why not trying the old VC++ 2010 version (which is also required for SAP Analysis Office) ... even if this is not supported any longer?

However, the extraction server needs an old Java version (JDK 17 does not work, it needs openjdk-11.0.2), so why not an old VC++ version, too?

So I installed it... and you won't believe it... IT WORKED right now! I could extract the data from table T001.

Oh my god... and we tried the newest versions of VC++.

Honestly: In the Extractor zip File, which you download from Signavio Process Insights, you should include ALL required Files!!!

The old Java and the old VC++ version, so that everybody has the files in place which are required!

You are always told by SAP: USE THE NEWEST VERSION and reproduce the issue 😞 ... in this case, only the old versions work. Unbelieveable. This was a long fight to get this running.

BR, Martin

MKreitlein · ‎2022 Aug 29

Hello Darshan Gohel,

yes we have the newest version in place:

The error log shows:

org.apache.hadoop.fs.FileSystem          : https:// = class org.apache.hadoop.fs.http.HttpsFileSystem from /C:/Users/m.kreitlein/Downloads/pi-etl-extractor-onprem-v1.403.0/pi-etl-extractor-v1.403.0.jar
org.apache.hadoop.fs.FileSystem          : Looking for FS supporting file
org.apache.hadoop.fs.FileSystem          : looking for configuration option fs.file.impl
org.apache.hadoop.fs.FileSystem          : Looking in service filesystems for implementation class
org.apache.hadoop.fs.FileSystem          : FS for file is class org.apache.hadoop.fs.LocalFileSystem
org.apache.hadoop.fs.FileSystem          : Creating FS file:///: duration 0:00.047s
c.s.p.e.e.o.EtlWebSocketClientService    : ExtractionJob failed | source-system: SAP
| exception: org.apache.hadoop.util.Shell.ExitCodeException

FYI: I'm Windows Admin, so it should not be a problem of creating a folder or a file... but I have no idea what this message indicates.

I've been waiting for a week now, for any new feedback in my incident, but nobody wanted to see or trace the issue on first hand...

Will update any news here.

BR, Martin

dgohel · ‎2022 Aug 29

Hi manuelmeindl / m.kreitlein,

Just wanted to check if the customer have installed proper version of "Microsoft Visual C++ Redistributable"?

Hadoop creates problem with Windows Server sometimes and required additional libraries to work.

https://docs.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170&viewFallbackFr...

Can you kindly install the MS VC++ Redistributable, according to your Windows server version?

Thanks.

Darshan

dgohel · ‎2022 Aug 29

Hi manuelmeindl / m.kreitlein,

Just wanted to check if the customer have installed proper version of "Microsoft Visual C++ Redistributable"?

Hadoop creates problem with Windows Server sometimes and required additional libraries to work.

https://docs.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170&viewFallbackFr...

Can you kindly install the MS VC++ Redistributable, according to your Windows server version?

Thanks.

Darshan

dgohel · ‎2022 Aug 29

Hi manuelmeindl / m.kreitlein,

Just wanted to check if the customer have installed proper version of "Microsoft Visual C++ Redistributable"?

Hadoop creates problem with Windows Server sometimes and required additional libraries to work.

https://docs.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170&viewFallbackFr...

Can you kindly install the MS VC++ Redistributable, according to your Windows server version?

Thanks.

Darshan

manuelmeindl22 · ‎2022 Aug 24

Hi Martin,

I cannot help you directly, but I have a colleague who encountered a similar or the same error. He is advised to share the logs with our support team. To still help you as much as I can I recommend: Can you share your logs from the on-premise extractor as well within the ticket?

Best
Manuel

MKreitlein · ‎2022 Aug 16

Hello amartiushov,

and all who are interested ... there are - finally - some good news:

I got feedback via my incident, to test the newest version: 1.403.0

Indeed, with this version I am able to establish the Websocket connection now, and also I'm able to connect to my on Premise system and trigger a data preview of the tables, e.g. T001 or T001W. 🙂

Currently I'm stuck in the ETL itself. Extraction fails, even if preview worked.

Again, an error message without clear explanation:

2022-08-16 11:14:10.156 DEBUG 9136 --- [        task-17] c.s.p.e.e.onprem.EtlStompSession         : 1716498d-3cfb-ed61-f5a4-964eea314551 true - message: ExtractionResponse(job=JobInfo(executionId=<ID hidden>, jobId=<ID hidden>, status=ERROR, tenantId=<ID hidden>), message=ExtractionJob failed | source-system: SAP
| exception: org.apache.hadoop.util.Shell.ExitCodeException
| cause: null
| sql-query: [SELECT "WAERS" as "waers", "KKBER" as "kkber", "LAND1" as "land1", "ORT01" as "ort01", "MANDT" as "mandt", "BUKRS" as "bukrs", "BUTXT" as "butxt" FROM T001
]
| query-params: {}
| anonymize-columns: []
, data=Data(correlationId=null, recordCount=null, mappings=[], deltaParameters={}, sqlParams={}), messageType=JOB_COMPLETED)
2022-08-16 11:14:10.156  INFO 9136 --- [        task-17] c.s.p.e.e.o.EtlWebSocketClientService    : ExtractionResponse(executionId = <ID hidden>, jobId = <ID hidden>, status = ERROR) sent to ETL bridge
2022-08-16 11:14:39.507 DEBUG 9136 --- [nnection-reaper] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 60000 MILLISECONDS

(I replaced the IDs with <ID hidden> )

I'm curious if I can get this last issue solved via the same ticket or a new one 😉

BR, Martin

former_member836644 · ‎2022 Jul 29

Hi m.kreitlein,

I see that you have been using onPrem version 1.375.0, I really encourage you to update to the latest version 1.389.0. We were recently fixing some aspects of proxy connectivity.

former_member836644 · ‎2022 Jul 28

Hi m.kreitlein,

Couple of ideas came to my mind:
- please update to the latest onPrem extractor (you can download it from https://editor.signavio.com/g/statics/pi-etl/on-prem-extractors).

- please make sure that proxy itself is properly configured to allow wss traffic (may be as a first step, prepare everything for successful connection to the bridge without proxy, and as the second step try to introduce a proxy in between).

Best regards, Aleksandr

former_member836644 · ‎2022 Jul 26

Hi m.kreitlein,

Note. You are right telnet tool is trying to establish TCP connection and verifies whether there are any firewalls which make the connection not possible. HTTP protocol works on top of TCP, WebSocket works on top of HTTPS (see OSI model, if you need more details).

On your private PC you have seen

<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
</body>
</html>

this is an indication that connection was !established!. And I guess that you have pressed some button, which was interpreted as a Request in scope of open connection, that is why you have seen this HTML text.

On your company PC you can not establish a connection. Because one of these problems:

- IP address is blocked (whitelisting of the IPs 15.197.210.108 and/or 3.33.194.113 should help)

- port 443 should be opened

There is no such thing as "enable" protocol, you just whitelist IP, open ports and then particular tools will use it to connect.

former_member836644 · ‎2022 Jul 26

Hi m.kreitlein,

Thanks a lot for the documentation improvement proposal, we will fix it.

Back to the networking investigations.

Please execute commands, which are trying to connect to particular IPs and ports:

telnet 3.33.194.113 443

telnet 15.197.210.108 443

If you are not able to establish a connection, then the problem is on networking side, and I suggest to check whitelisted IP addresses and ports in your infrastructure.

former_member836644 · ‎2022 Jul 25

Hi m.kreitlein,

Sorry that it takes you so much time to configure on-premise extractor. The problem looks to be on network side, but nevertheless lets make sure that configuration of the application is correct. You should have to:

- Create on-premise connector here https://editor.signavio.com/g/statics/pi-etl/on-prem-extractors

- Copy parameters for this on-premise extractor and paste them to config.env file of downloaded zip

So if above is true lets jump into network check list:

- as written HERE it is required to be able to ping two IP addresses (may be it will be required to whitelist them on your infrastructure side, using e.g. firewall)

- also make sure that port 443 is open on your firewall side

I hope this helps.

MKreitlein · ‎2022 Jul 20

Hello Mike,

thanks for the reply. Can you let me know how to prove if the traffic gets really through or not?

My Firewall team insists it is the case, and I can only trust them without further knowledge.

The only thing I have is, when I execute my Javascript - the first message is never displayed... but after around 10 seconds, only the second message is displayed:

<html>
<body>
<script>
var socket = new WebSocket("wss://spi-etl-bridge.eu-prod-cloud-os-eu.suite-saas-prod.signav.io");

socket.onopen = function () {
    document.write("Connection established successfully");
};

socket.onerror = function (errorEvent) {
    document.write("Error! The Connection was closed, unexpectedly!")
};
</script>
</body>
</html><br>

I'm not able to ping the two IP addresses: https://documentation.signavio.com/suite/en-us/Content/process-intelligence/etl-on-premises-connecto... ... but like told, the network guys have their own knowledge and told me "to ping or not to ping" is no indicator at all?!

So, what to tell them?

Thanks, Martin

ZhangMingquan · ‎2022 Jul 19

Hi Martin,

It seems the extractor is trying to connect to the cloud via proxy, please make sure the firewall port 443 is open and the traffic could get through and reach each other: extractor <-> proxy <-> cloud.

Best Regards,

Mike

By Category

Related Content

Activity Groups

Industry Groups

Influence and Feedback Groups

Interest Groups

Location Groups

Customer Only Groups

Forums

Related Resources

Products

Learning and Support

About

My Account

My Account

Missing information in SAP Signavio Process Intelligence ETL Setup to on-Prem (S/4HANA)

Know the answer?

Need more details?

Accepted Solutions (1)

Accepted Solutions (1)

Answers (18)

Answers (18)