on 05-23-2024 12:46 PM
Hello everyone,
I am still relatively new to the Datasphere and am currently familiarizing myself with Python scripts for data transformation. I am encountering a problem in the mapping, which does not occur without a Python script with pure mapping of field in Datasphere.
My source has 56 fields, which should be extended by 2 additional calculated fields. If I write the data 1:1 (without script) into a target table, the data transformation works. As soon as the script is inserted and the data are passed through the script, the following error message appears:
I cannot identify from the error message which column is causing the problem. Is there any way to identify it, or do I have to extend the dataflow column by column and deploy and run the script again and again to determine which column is causing the error?
Hi,
First, the issue is caused by some invalid datatime value, you just need to check the related columns. it's not very convenient to debug the Python script in Datashpere, you can try to debug it in other applications, like Jupyter Notebook, with some sample data.
Second, you can try some other node to extend the columns you want if it's possible.
Thanks,
Simon
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hey Simon,
thanks for your idea. I hadn't even thought about the Jupyter notebook. I'll give that a try.
However, after further research, it seems to me that the problem lies in an incompatibility of the data type. In the source (AS400 database) there is a column of the data type "timestamp". If I use the Dataflow, the data from the remote table can be transferred to the local table without any problems.
If I add a script without additional code (only "return data") to the Dataflow, the error message appears.
The error therefore occurs regardless of the content of the Python script.
The solution can therefore only be that I check all columns for possible incompatibilities before calling the Python script and before the dataflow reaches the script.
---
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
User | Count |
---|---|
84 | |
10 | |
8 | |
7 | |
6 | |
5 | |
5 | |
5 | |
4 | |
4 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.