Block extraction of large data volumes

During the extraction, all data of an RFC call is transferred at once to the Java client using SAP JCo. However, large data volumes can result in memory overflow (OutOfMemoryError). For example, this happens when the primary keys for the document tables and document header tables are being determined. In this case, the data must first be buffered in the R/3 system and then transferred to the Java client a block at a time. The block size is specified by the -cpd parameter. There are three different block modes, which differ in terms of the way in which the data is buffered in the R/3 system.

Block mode D

Mode D is recommended for block extraction of large data sets from the R/3 system. The selected data is stored in the /IDS/INDX database table. If the connection to the Extractor fails or the extraction operation is aborted manually, the data is not automatically deleted from the database table. You need to delete the buffered data manually. To do this, display the content of the /IDS/INDX table using transaction SE16. Then select all the relevant entries and delete them using the Delete all... option in the main Table entry menu.

s_sap_6_sap_screenshot

Example

You want to extract more than a million entries from the CPDOS table (change document items). If you do not use block extraction, the Java memory will overflow and extraction will be aborted.

Executing the following command line extracts the data in blocks (block parameter in bold):

runsap2ppm -systemconfig Systemconfig.xml -tableconfig TableconfigCDPOS.xml

-outfile eventsCDPOS -nozip -begindate 19700101 -rfc_blockmode D -cpd 10000

By specifying the -rfc_blockmode parameter with the D argument, you start block extraction, which writes all data from the CPDOS database table to the output file. You use the -cpd parameter to specify the size of the blocks to be processed. In the example, 10000 documents are processed simultaneously. The default value is 5000.

The progress of the block extraction is logged as specified, e.g.: 

I: 09.05.06 11:30:38: [XML] Extracting data from...

...

...

I: 09.05.06 11:32:30: [XML] "CDPOS" table:

10,000 of 1,036,995 rows extracted.

I: 09.05.06 11:32:30: [XML] Extracting attributes for

10000 source system events...

I: 09.05.06 11:32:39: [XML] "CDPOS" table:

20,000 of 1,036,995 rows extracted.

I: 09.05.06 11:32:39: [XML] Extracting attributes for

10000 source system events...

...

By specifying -progress no, you can suppress the additional log messages for the block modes, which show the progress of the block extraction.

Block mode G

The selected data is stored in an internal table of the function group. The data is held in the R/3 server memory. If the connection to the Extractor fails or the extraction is aborted manually, the data is automatically deleted in the R/3 system. These specifications apply from SAP Release 4.7, i.e., from SAP Kernel Release 6.10. 

Block mode M

The selected data is stored in the ABAP memory using export to memory id. The data is held in the R/3 server memory. If the connection to the Extractor fails or the extraction is aborted manually, the data is automatically deleted in the R/3 system. These specifications apply from SAP Release 4.7, i.e., from SAP Kernel Release 6.10.