topic Re: Open Dataset Query in Application Development and Automation Discussions

Open Dataset Query

Former Member — Wed, 27 Jan 2016 09:17:47 GMT

Hi Experts,

I am just curious, is there any time limitation as in up to what time limit the file can remain open. The development scenario is that I am opening a large file using OPEN DATASET command , then the file contents are read using READ DATASET command. As soon a first x data is read from the file, parallel processing of the records starts(processing includes database commit and rollback) . After the x records are processed, I go for next x records, read and processing. The process is suppose to run for 12-15 hours.

I understand opening the file for a long time can restrict the access of the file for other programs, but is there any chance I can loose data from the file due to long run time. My goal is to reduce the memory intake by processing the file data in packets, rather than reading all the file data into an internal table and start processing.

Please advice.

Re: Open Dataset Query

Former Member — Wed, 27 Jan 2016 10:22:24 GMT

Since you opened your file for reading (I suppose) I think the data is not put at risk. However in my opinion it would be neater to open/close for each processing step if it takes that much time.

Have you considered to saving the position and next time use OPEN DATASET with addition AT POSITION?

Re: Open Dataset Query

Former Member — Wed, 27 Jan 2016 10:27:58 GMT

Hi Gabor,

Yes, I have considered AT POSITION but just wanted to know if long run time could lead to data loss.

Re: Open Dataset Query

Former Member — Wed, 27 Jan 2016 12:34:35 GMT

Hi Ritesh,

From my experience i have not run into a timeout scenario with OPEN DATASET in long running programs or data loss. I think what your wanting todo is possible and you should not have any issues with that as long as you can guarentee that no other programs will attempt to change the file your working on until your finished.

Now having said that i agree with Gabor that it would be more efficient to use the AT POSITION code and do a open/close data set as this will ensure your resources stay free.You could even look into doing parallel processing with this method if you wanted to help speed up processing time.

Geoffery

Re: Open Dataset Query

Former Member — Wed, 27 Jan 2016 14:04:46 GMT

Hi Geoffery,

Thanks for the input.

As for the AT POSITION syntax, SAP recommends to use this when we are opening the file in binary mode, but in my case I am opening the file in text mode. So, I am in a fix whether to use it or not.

Re: Open Dataset Query

Former Member — Wed, 27 Jan 2016 14:24:45 GMT

Hi Ritesh,

Do you have a specific requirement to do this in text mode? Because it seems like what you are wanting to do would be best done using the default mode in sap which is binary.

Geoffery

Re: Open Dataset Query

Jelena_Perfiljeva — Wed, 27 Jan 2016 22:10:31 GMT

I would suggest to run this scenario by your Basis team. "I will use a file to avoid using memory" sounds like something we would do in the good old mainframe days when hardware resources were at premium. In these days of in-memory databases such approach seems questionable, to say the least.

As for data loss - yes, it is, of course, possible if a program gets interrupted unexpectedly. Just like if anything was in memory and not in the database it would be lost as well. Files do not have any magical properties.

Re: Open Dataset Query

Former Member — Thu, 28 Jan 2016 08:05:11 GMT

Thanks All. Appreciate your inputs.

Re: Open Dataset Query

Former Member — Thu, 28 Jan 2016 08:08:26 GMT

Hi Geoffery, its just that I am enhancing an existing logic and the file contents are being read in text mode.