topic Re: strange data conversion problem in Application Development and Automation Discussions

strange data conversion problem

Former Member — Fri, 21 Jul 2006 17:16:06 GMT

Hi all,

I have a strange data conversion problem. I have a few fields of type QUAN, with runtime values of Var1 - "499.000" and Var2 - "12.500". I divide Var1 by Var2 and push the result in another field of type QUAN, Var3. The result should be '39.920'

The problem is, the result in Var3 is '0.040'. The value getsw rounded off to 40 and then gets assigned to this variable. The funny part is that the value gets copied to Var3 directly without conversion, and ends up in the decimal part of the variable.

I tried this on another program and it worked fine. But here, there seems to be some other problem. I tried with packed decimals as well but got the same result.

Another strange thing is that usually if you assign a value like '123.456' from a packed decimal of dec. length 3 to another of length 2 the result would be '123.46', but when I do it in this program, it becomes '1234.56'. I fail to understand why this is happening.

I believe this has something to do with the program's attributes/environment. The same code works perfectly in other programs.

Please Help!!

Thanks in advance,

Madhur

Re: strange data conversion problem

RichHeilman — Fri, 21 Jul 2006 17:19:48 GMT

Your right it might have something to with the program attributes. You need to check the "Fix Point Arithimatic" Check box on the attributes screen on the program. If it is not checked, then this is why you are seeing this behavior. You may be able to just click this checkbox on and it will solve the problem, but there may be a reason why it is off. If you don't check the box, you will need to handle manully which means that you will need to multiple by 100 or 1000 depending on the number of decimals places.

Regards,

Rich Heilman

Re: strange data conversion problem

Former Member — Fri, 21 Jul 2006 17:23:26 GMT

Hi Madhur,

Check in the program attributes, if the Fixed point arithmatic is checked.

Sometimes, such kind of errors happen if the checkbox is not checked then all QUAN, packed decimals will treated as integers only.

Cheers,

Vikram

Pls reward for helpful replies!!

Re: strange data conversion problem

Former Member — Fri, 21 Jul 2006 17:27:15 GMT

Hi Rich/Vikram,

Thanks a ton.

Regards,

Madhur