What is the file size limit when using the FTP adapter in SAP Integration Suite ?

[glenmisquith]

What is the file size limit when using the FTP adapter in SAP Integration Suite ?

[alex_bundschuh · Accepted solution]

Hi,

in theory, there is no file size limit because FTP and SFTP adapter support streaming, however when using streaming you need to ensure that all flow steps within your integration low support streaming, see https://help.sap.com/docs/CLOUD_INTEGRATION/368c481cd6954bdfa5d0435479fd4eaf/396941a8c12441ab999f0f3...

There is however a general recommendation to keep the payload size below 40 MB for an optimized overall performance although larger sizes may be supported, however this always depend on your specific integration flow design and requirement, see https://help.sap.com/docs/CLOUD_INTEGRATION/368c481cd6954bdfa5d0435479fd4eaf/491c80d16c3547c3b124cf3...

Alex

[benjaminnehring · Accepted solution]

Hi alexander.bundschuh,

can you give a bit more detail on what it means for the SFTP sender adapter to be streaming enabled? Let's say we have a large text file on a SFTP server, but the lines of the file can be processed independently. If we now configure an Iterating Splitter after the SFTP sender adapter with the following configuration:

Expression Type: Line Break

Grouping: 50

Streaming: enabled

Parallel Processing: disabled

Is my understanding then correct that the SFTP sender adapter would start reading the first 50 lines of the file and hands it over to the Iterating Splitter. After the splitter is finished with processing this batch of data, the SFTP sender adapter reads lines number 51 to 100 and hands it over again to the Iterating Splitter for processing. This cycle continues until the file is completely read. Is that what is happening in the streaming case in CPI?

BR, Benjamin

[alex_bundschuh · Accepted solution]

Hi Benjamin,

the outcome is the same like without streaming, with streaming however the runtime already starts processing even if the whole message hasn't been fully stored in the memory, so streaming is less memory-intensive

Alex

[benjaminnehring · Accepted solution]

Hi alexander.bundschuh,

ok, thanks for the clarification. So by using a streaming enabled splitter, the complete file would also be loaded into the memory. If we want to process arbitrary large files, this is only possible when all IFlow steps support streaming.

BR, Benjamin

[benjaminnehring · Accepted solution]

Hi alexander.bundschuh,

in the PO SFTP sender adapter there is the option to split a large csv file with File Content Conversion and the parameter Recordsets per Message into multiple messages. According to note

2663129 - Performanceverbesserung für den Modus des SFTP-Adapters zur Konvertierung des Dateiinhalts

this can also be configured to be memory efficient, meaning the file is always only partially read into the memory. Is such a feature also planned to be made available for the CPI SFTP sender adapter?

BR, Benjamin

[otto_frost4 · Accepted solution]

REMOVE COMMENT

[otto_frost4 · Accepted solution]

sftp to sftp - using streaming

there is 150 GB free space at the sftp server 

trying to transfer a 11 GB file we hit the following error 

Error Details org. apache. camel. StreamCacheException: Error during type conversion from type: org. apache. camel.component. file. remote. RemoteFile to the required type: org. apache. camel. StreamCache with value RemoteFile[dummy4.zip] due Error during type conversion from type: java. lang. String to the required type: org. apache. camel. StreamCache with value [Body is not logged] due java. io. IOException: error 28 - No space left on device, cause: java. io. IOException: error 28 - No space left on device

Conclusion - sap ci has some internal undocumented limit on the file size
The max size successully transferred 2.8 GB
If the 40 MB is the offically supported max size it seriously limits the usability of the product. It is the year 2025 and already 1995 most products supported file sizes above 40 MB. SAP should not be delivering a product which such a small limit in the year 2025.

Tested on suse openssh server and azure blob storage - the same problem on both servers.


see sap note
2648415 - Integration Flow Runtime Error "java.io.IOException: No space left on device"
no solution to the problem in the sap note except
create a ticket under BC-CP-IS-OPS to check the disk issue

A minimum requirement is that SAP has documented what is the limitation. Streaming implies unlimited file size but that is not the case.
It seems like the file transferred using streaming is actually stored at the CI tenant - if this is the case  then the file size is limited to the available storage space at the tenant - what is the available storage space at the tenant? Can the storage space be increased and how is the storage space increased? 


see https://camel.apache.org/manual/stream-caching.html (no clue about the max file size supported)

[maik_bosch · Accepted solution]

Please see https://help.sap.com/docs/cloud-integration/sap-cloud-integration/inspect-top-integration-flows-by-s... for some more information and you should see the 2 to 3 GB limit also here. Similar limits on our system

[AlexanderApel · Accepted solution]

Hey Alex, is SAP planning to increase the Payload in Integration Suite? We are facing more then one iFlows with huge Data Files with more than 1 GB per file. This is erveytime more than a challage to handle these files. Also we have the same Problem with our JDBC Adapter and SQL Responses over 1.5 Million records. In 2025 we should be able to handle these amounts of data. Best Regards, Alex

[Yogananda]

Hi glenmisquith

10MB is the size limit as far as I have tried..

[glenmisquith]

Hi Yogananda,

Will the FTP adapter throw an exception if the file size is above 10 Mb ? Is this size limit mentioned in help.sap.com or some other content ? I wasn't able to find any reference of this.

[Yogananda]

glenmisquith

yes it would indicate in message monitoring if the file size is exceeded for your iflow with failed status. Its not mentioned in help documentation about the file size limit.