#Fatal#/System/Network# on the updater instance

[doris_karapici]

Dear all,

currently, we are having issues with the updater instance that unexpectedly shutdowns. After restarting for some days it looks fine but then again shutdowns.

We observed the following Fatal Error in the Updater:<br>#2.#2023 02 22 19:33:10:748#+0100#<strong>Fatal</strong>#/System/Network#
com.sap.SCC.infra.000021#IS-CC##C0000A8B045A03F20000001F00004038####<system>#######Thread[Socket[addr=abcdefghijkl/111.222.333.44,port=2100,localport=11549]-WRITE,5,main]#Plain##
Connection with the remote client abcdefghijkl/111.222.333.44:2100 is closing because an error occurred with the socket network. Reason is: <strong>Broken pipe</strong>.#

We did a screening of all JVM Memory settings and were all according to the Note 2006073. The only parameter that we were not sure about was on the updater that it was -Xss256k

During the period of this Fatal error, there was no high load on the system and the CC is a mono-host system.

We do not have a complicated charge logic and less than 10 counters in the charge plans. Any ideas, or suggestions would be welcomed.

Many thanks,

Doris

[francois_thimon · Accepted solution]

Hi Doris,

If the port number in the actual error log is 2100, then it seems that the connection is always broken while the updater is trying to write something to the dispatcher (because, by default, 2100 is the dispatcher's internal port, used to receive acknowledgements and miscellaneous queries from the other instances).
So, can you check the dispatcher's logs/traces to see if a matching error appears around the time of the disconnections?
Please also check if you have thread dumps generated automatically for the updater and the dispatcher. They might indicate if a particular problem cause the disconnections.

In some cases, connections may be forcibly closed by a network security mechanism (either because they've been open for too long, or because they've carried more bytes than the authorised limit).
You may look into this, but we have no evidence that this is your issue now.

Also, can you check whether the other instances are similarly affected? This may be an indication too.

Finally, if it helps, you may enable the "NET" debug traces on the dispatcher and the updater until the issue occurs again (it's probably best to shut down the other instances during this test, to avoid getting too many illegible logs).

Best regards.

François
SAP Convergent Charging Support