topic Floating Point Conversion in Application Development and Automation Discussions

Floating Point Conversion

Former Member — Mon, 15 Oct 2007 02:17:42 GMT

Hi,

I have a question on how integers are converted to floating point variables. I have a formula that involves a floating point variable but it reacts differently in different boxes.

data: x_floating_pt type f.

data: x_var1 like xxxx ---> This has a data type of quan 13 dec 3

data: x_type_p type p decimals 0.

x_var1 = 100.00

x_floating_pt = x_var1.

add x_floating_pt to x_type_p.

In one box, x_type_p will result into 100. While in another box, it will result into 100,000 (which requires me to divide it by 1000 to get the correct result).

Thanks.

Re: Floating Point Conversion

Former Member — Mon, 15 Oct 2007 02:27:02 GMT

Hi, DWade,

What do you mean by "different boxes"?

I'm not sure, if the different situations are in different programs, maybe it is caused by the different program attribute set on "fixed point arithmetic".

Re: Floating Point Conversion

Former Member — Mon, 15 Oct 2007 03:36:01 GMT

Thanks for your reply ! yes, the fixed point checkbox change does it