Processing 10 Million Records

[sanjay_ram]

Hi,

I need read 10 million records from a transparent table and process it. However if i read that many records into an internal table, the system crashes due to memory issue. How can i solve this issue?

[sanjay_ram]

Hi,

I need read 10 million records from a transparent table and process it. However if i read that many records into an internal table, the system crashes due to memory issue. How can i solve this issue?

[former_member598787]

Hello Sanjay,

Use Open cursor and get the records based on PACKAGE SIZE. Also it would be helpful if your program is run in background.

Refer to below links :

https://help.sap.com/doc/abapdocu_751_index_htm/7.51/en-us/abapopen_cursor.htm

https://answers.sap.com/questions/1757819/smallest--simplest-example-of-open-cursor-concept-.html

Regards,

Juby

[matt]

SELECT data FROM transparent_table INTO TABLE internal_table PACKAGE SIZE 100000.
  Do processing of the 100000 records just read.
ENDSELECT. 

No need for an OPEN CURSOR - although that's exactly how it works in the background.

[Sandra_Rossi]

But OPEN CURSOR WITH HOLD is needed if there's a database commit needed at the processing of each package.

[matt]

Not necessarily. Sometimes CURSOR WITH HOLD fails. (E.g. during RSA3). In these cases, you simply use a service connection.

https://help.sap.com/doc/abapdocu_751_index_htm/7.51/en-US/abenopensql_multiconnect.htm

Service Connections to the SAP Standard Database

In addition to the secondary database connections stored in database table DBCON, service connections to the SAP standard database can also be used. The name of this connection is R/3*name, which consist of the prefix R/3* and a user-defined name. A service connection automatically uses all the settings of the standard database connection.

You use the CONNECTION keyword with a connection beginning with R/3* and you can do commits on that quite freely.

[Sandra_Rossi]

+1 to you, I always forget this damn service connection.

I prefer OPEN CURSOR though, old habit of not using SELECT...ENDSELECT.

I was trying to see the advantages/drawbacks between the two in the ABAP documentation, there's almost nothing documented for SELECT...ENDSELECT (let's say it's very much condensed), but the one for OPEN CURSOR is easy to find and it takes sufficient place to describe it.

I also see that SAP puts the same effort to make OPEN CURSOR evolve identically to SELECT.

My conclusion is that both of them are identically recommended.

Maybe SAP should abandon one instead of maintaining both, but then OPEN CURSOR would win because (I think) it has more features (I think of reading two cursors in parallel ; okay extremely rare).