Curl Easy Perform Error 52 faced in IBP Outbound Task runs in SAP CI-DS and its possible resolutions

[BSandeep]

1.What’s the error ?

What is the curl error 52 "empty reply from server"?

Curl gives this error when there is no reply from a server, since it is an error for HTTP not to respond anything to a request. The problem we have is that there is some piece of network infrastructure, like a firewall or a proxy, between us and the host in question. Getting this to work, therefore, will require us to discuss the issue with the people responsible for that hardware, i.e. the basis team. 

It can happen when server does not respond due to 100% CPU or Memory utilization.

This error has been encountered multiple times, like when someone was trying to access SonarQube API, and the server was not responding due to full memory utilization

Another common reason for an empty reply error is timeout. We can check all the hops from where the cron job is running from to your PHP/target server. There's probably a device/server/nginx/LB/proxy somewhere along the line that terminates the request earlier than we expect, resulting in an empty response.

We may have encountered the ‘curl (52) empty reply from a server error’ when we try to visit a website. This happens when the HTTP client does not connect to the intended web server. This response indicates that we are not using a secure HTTPS connection. There may be other issues causing the server to not process our request.

2.Why or when does it occur? What is the reason?

• If we are extracting Key Figure data from IBP to another System, the data is aggregated, calculated and processed by the IBP calculation scenarios. The calculation scenarios can be big depending on the amount of data is read and complexity of the key figure calculations (L-Code). Due to the architecture of the system the data extraction jobs have a timeout of 10 minutes. If a calculation scenario query doesn’t finish within 10 minutes, the connection breaks and no data is passed. The curl error appears.
• Occasional problems: Someday data can be extracted within that limit, some other days not. What is the reason for it? The runtime of the query does depend on overall system load by other IBP activities.

• A Mismatched Certificate – You may get an empty response if the SSL certificate is mismatched. The solution to this is to check whether the HTTPS certificate actually works with the simple tools available.

• Your HTTPS Request Is Targeting a Non-HTTPS Port – Another possible reason why we may be receiving the error is that our HTTPS connection is targeting an HTTP port. No HTTPS client can establish any encrypted session with an HTTP port. In this case, most browsers will indicate that one is also trying to access an unsafe website. There is a simple command that allows us to test if a connection is secure so that we can take the right steps.

          Sometimes the reason for this mismatch is firewall rules that are wrongly configured. We may be                using the wrong port, or some services that should be enabled to use the HTTPS port are not                      enabled. Resolve the issue on the user terminal.

• The Server May Have Closed the TCP Connection – If our server returns an empty response, it means the server has terminated the TCP connection. This is usually caused by the HTTP request being delivered to an HTTPS port. This error can occur if we try submitting an HTTPs request using  an HTTP port. It is the opposite of the case discussed above. Because HTTPS clients cannot create an encrypted session with a plain HTTP port, such a request is not possible.

          Solution to above point - Start by checking if our HTTPS client uses the right protocol. Only use               the HTTPS port for all HTTPS connections. By default, HTTPS connections use TCP port 443.                     However, the unsecure protocol HTTP uses port 80. We can change ports on the Window                           Defender Firewall as a user to force connections through TCP port 443. On the other hand,                         website owners need to configure server applications like Apache to ensure their sites use the                     same port.

• We are Using an Older Version of Libcurl - If our curl is using an outdated version of libcurl, We are bound to get an empty response from the server. This can be resolved by updating the libcurl. If we want to upgrade your curls, we can download the latest official curl release at curl.se and select our OS. If we have the latest version but are still facing issues, check if curl has a port assignment and consider using another port.

3.Solution approach to resolve this error ?  

When resolving the “ curl (52) empty reply from server” error, start with the basics , such as checking out the ports before looking at the certificates and the curl version in use. They are the common issues where the site is duly updated.

4.Is it an SAP specific or general error ?

It's a general error related to response from https server.

5.What can be done in this case, from a CI-DS perspective?

 1.Instead of running 1 big job to extract data, use multiple jobs.  Split the size of the jobs by applying filters and schedule jobs in sequence. E.g. filter for certain locations, products or business segments.

Furthermore, filter on time horizon and extract at needed weekly / monthly /...time granularity.

It is important the filter in CI-DS  is passed on to IBP Calculation Scenario : How can we check that ? On the CI-DS Data Flow: go to View History -> Monitor Log

Compare the row count between top and bottom line. (Note : there can be multiple lines) IF there is a difference you should have a closer look on the filters and transformation steps in CPI Data Flow.

 2. Reduce the number of Key Figures. if the data extraction job is running long, we can also split by Key figures. One troubleshooting guide is to separate stored Key Figures from those which have a complex calculation logic.

Note : Be Careful, when extracting data of key figures -That have an L-Code calculation or, External key figures (part of order-based planning) \

1. Plan early for performance testing. Make sure that we do have proper performance testing of the extraction prior productive use / Go Live of the system. Consider that over the time the planning area size may increase, and that the extraction still is well below the 10 min timeout.

2. Permanent solution offered by SAP : as per the latest update, SAP has suggested that the agent must be kept or installed at its highest version 1.0.011 or more (check if there’s a minimum version or more edit of this line) and if we are having an older version , then this curl error is bound to occur sometime or the other, so it’s better to upgrade the agent version as per SAP’s suggestion. For more information, please check with SAP help support.

Ques is, why does the permanent solution work ??

It’s because of the architecture change in recent times. The IBP – CI DS architecture has changed from following JDBC to following WebSocket RFC protocol. These are the following benefits from switching to WebSocket RFC protocol in CPI DS IBP setup:

Benefits of WebSocket over JDBC

The WebSocket RFC communication protocol provides the same set of features as JDBC, and the following additional benefits:

• SAP Data Services Agent communicates directly with the SAP IBP database instead of going via the SAP Cloud Integration for data services server. This simpler architecture increases system stability. and makes it easier to find the root cause if an issue occurs.
• Data extraction is not limited to 200/400m records in one flow, so it's no longer necessary to split tasks or create multiple tasks due to system limitations.
• WebSocket RFC supports X.509 certificate-based authentication, which is recommended for productive systems and requires a personal security environment (PSE).
• Stability and reliability for long-running jobs are improved. As there is a direct connection between SAP IBP and SAP Data Services Agent, there are no more time-out errors for jobs when they need to query initial data for more than 15 minutes.
• Systems using the WebSocket RFC connection are suitable for hyperscaler deployment. This leads to more elasticity and speed.
• As opposed to the system setup using the SAP-managed communication arrangement, the new configuration supports the IN and NOT IN operators in filter conditions used when exporting data from SAP IBP using SAP Cloud Integration for data services. For more information about differences in filtering, see Best Practices for Extracting Data from SAP IBP.

6. Important SAP Notes on the concerned issue :

1.     2493042	IBP- HCI Integration - Best Practices for extracting data out of IBP
2.     2685841	CPI-DS task error: Error: <"curl_easy_perform":52:Server returned nothing (no headers, no data)>

7. References :

1. How to Fix the Error "curl 52 Empty Reply From Server" - code fight club
2. SAP IBP Data Extraction via SAP CPI-DS - Curl ERRO... - SAP Community – by @MichaelMack
3. SAP IBP Data Extraction via CPI-DS: How to best fi... - SAP Community – by @MichaelMack 
4. Troubleshooting tips while working with CPI-DS - SAP Community
5. cron - What is the curl error 52 "empty reply from server"? - Stack Overflow
6. elasticsearch - "curl: (52) Empty reply from server" / timeout when querying ElastiscSearch - Stack ...
7. What does curl do? - everything curl
8. 2685841 - CI-DS task error: Error: < "curl_easy_perform":52:Server returned nothing (no headers, no ...
9. 2493042 - IBP- CPI Integration - Best Practices for Extracting Data out of IBP - SAP for Me
10. Scenarios | SAP Help Portal
11. SAP Note: 2728370 - How to extract data in base UOM / CURR – signifies different ways of filtering data out of IBP using cpi ds filters, examples are covered in this and other KBA notes.
12. SAP Note: 2437604 - HCI Outbound task : Data multiplied by certain factor in the extract
13. SAP Note: 2459215 - IBP: Is it possible for extract Key Figure data of a specific Scenario via HCI?
14. 2947890 - IBP - Understanding the use of the calc-scenario in extraction tasks from IBP using CPI/DS... – most imp and reference to access all other important documents.
15. Solved: CPI-DS agent upgrade steps - SAP Community – agent upgrade steps, will alleviate the curl 52 issue.
16. Scenarios | SAP Help Portal