2024 Aug 27 12:48 PM - edited 2024 Aug 27 12:48 PM
Hello,
I have a loading process that is not working as expected. It has been done as many other processes are working correctly, but this one is not.
The problem is that in the Data Monitoring, in the Remote Queries section, the parameterised SQL statement is shown. This is
SELECT ... FROM ... where "_load_date" > CAST( ? AS TIMESTAMP)
, where the ? will be replaced by a timestsamp which is calculated in a table function SQL view.
While I can know the calculated TIMESTAMP, I don't see the one that is actually sent to the data source, and it looks like something is wrong with that, since I'm getting data that doesn't match the where filter.
I haven't seen any way to know what time stamp was sent in the where, neither in the monitoring, nor in the data flow logs, which makes me totally lost and the process out of control.
Does anyone have any idea where I could find it?
Thanks.
XP
2024 Aug 27 1:04 PM
2024 Aug 27 1:08 PM
Thanks @JimSpath, but we work with remote systems that we can't always have that level of access, and if we can it could take days.
Ideally, I would like to be able to see it in the DSP or in the HANA Cockpit.
2024 Aug 28 9:34 AM
You should be able to check the query which is sent to remote source in Datasphere at remote query monitor in Data integration.
2024 Aug 28 10:17 AM - edited 2024 Aug 28 10:18 AM
Thanks @Yogesh_Rehadwa , but in the monitor the parameterised queries are shown as I described, with ? and not the value.
2024 Aug 28 10:16 AM
I have not found the answer to my question, but I have found the bug that prompted it
In this particular case I am reading from a remote Google Big Query source.
Parameterised query parameters are not logged there either because they might contain sensitive information.
So I have not been able to find out what exactly is being sent in the parameter.
But after testing, I have found the error.
There is a difference between GBQ and DSP data for the TIMESTAMP field.
In the GBQ table I have ‘2024-06-12 10:44:06.350283 UTC’ (read using GBQ console) and that same data read from the DSP remote table gives ‘2024-06-12 10:44:06.350282’. Note the difference in the last decimal place.
This makes the system inconsistent (data is not a exact COPY, and DIFFERS in the source system and in DSP) so queries do not return the expected data.
I will have to open another incident in SAP regarding GBQ integration.