SAP 760 GUI Response Way too slow

[former_member697481]

SAP 760 GUI for Windows Patch 6 Server 2016 & Citrix PVS Streamed Shared VDI - "Too slow and it should be faster"

I'm not sure where to begin other than we are making poor progress with this GUI (used Patch 3, 4, 5, and now 6). I'm coming from the Citrix Architect perspective and although the backend hardware is heavy-duty it is considered in the development phase which apparently equates to no support from the SAP customer perspective. I was recently engaged in the last week as an escalation of last resort. I'm posting and asking for guidance for a project that is now deep in the red.

So, I'll start with the basics:

1. HANA DB on IBM 8808 44E Series Power8 w/ 48 cores - PowerVM. vFlexHANA 48 Cores 512 GB 1200 GB HD.
2. SAP S/4 Middleware Tier is a virtual machine on ESX 6.5 SUSE Linux Enterprise 12 (64-bit) virtual machine 4vCPU and 48GB of RAM on an 8 Host cluster of ProLiant BL460c Gen9 with 72 cores Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz and each host having 384GB of physical memory.
3. Citrix Provisioning Services shared VDI (2016) hosted exclusively in physical memory. Two data centers connected by dark fiber. Each DC running at 50% capacity and memory reservations.

100% of the business applications hosted in Citrix. Current SAP GUI 740. The Citrix base image is 2016 with the latest CUs and security patches as of this month. The project started in March and falling behind due to the application teams claim that the GUI is too slow. Keeping in mind there are no baselines given the fact this is a new implementation. This is more of an intuitive claim where the GUI does not populate fast enough when clicking in the GUI to populate the line item or options. So when they click on one of the > on the left the lines underneath do not populate quick enough. However, there is no expectation set to how quickly they should populate only that it "should be faster."

So, I'm going with this for now as we are attempting to role this out as the next generation of SAP GUI. I have metrics from the application side that I will post but my first question is about the GUI version itself. Reading the forums, there appears to be known issues with this GUI and Server 2016. However, I am not able to locate the best practices guide relative to optimization options for the GUI.

The Citrix VDI is multi-user VDI with Citrix 7.15 LTSR CU6. Divided between two active HA data centers connected by dark fiber. 50% capacity on both sides to accommodate the complete loss of 1 data center. It is important to understand that these are read-only streamed images hosted 100% in the physical memory of the ESX cluster. The entire Citrix session is hosted in RAM. Each VM has 48GB of vMem and 8vCPU. There are 100's of these VM's in memory and Citrix advanced load balance algorithm distributes the sessions across the pool of shared VDI. The memory is reserved for each VM and the ESX cluster is 50% utilized to accommodate both DC's if one fails.

From the Citrix perspective, this is not a CPU or Memory related issue. The issue seems to be more related to the middleware tier. The type of latency almost resembles a typical IO related scenario where the underlying storage is the bottleneck. CPU and Memory have not been a bottleneck for more than 10 years now.

Or, it could be a simple interoperability issue with SAP 760 GUI running on Server 2016 Remote Desktop Services (Data Center Edition). However, spinning up a new Windows 10 Dedicated VDI on the same network does not resolve the latency issues in the GUI.

One example of the latency is the OLE Flush call. These are taking 4-5 seconds or longer and I'm told this is "too long."

OLE_FLUSH_CALL: The function module OLE_FLUSH_CALL is used to communicate with SAPGUI. OLE_FLUSH_CALL time is GUI time. When OLE_FLUSH_CALL appears in traces, it means that there are elements on the screen (SAPGUI) which can only be populated/controlled through the OLE Automation (OLEA) interface. (ALV Grids are one example of such an element.) Many transactions and screens across all applications in SAP NetWeaver will call OLE_FLUSH_CALL. This means almost every trace of a DIALOG type task will contain OLE_FLUSH_CALL

Network time (server to Citrix and back) = GUI time - interpretation time 4232 ms – +/- 400 ms= +/- 3800 ms 3557 ms – +/- 400 ms= +/- 3000 ms Here we are talking about 3 – 4 seconds per call, though during some testing we had Ole_flush_call of more than 20 seconds.

I'm told this is too long. The backend environment is hosted by the third-party at the multi-datacenter. The data above is from my SAP support team internal. Thus far, we are unable to get any real information as to the underlying storage.

Example
Trace: one Ole_Flush_Call 9.8 seconds with Client

GUI time: 9.8 seconds. Roll (in+ wait) time is the wait time of the process to get a response back from the front end. DB request time was 1.5 seconds.

I'm told this is too long.

SAP 740 does not have this issue on the same platform. I assume they are using that as their expectation for performance.

I've been engaged due to I am the last resort escalation. There are no obvious performance-related issues.

The perception is that Citrix VDI must be the issue and that it is preventing the project from completing as scheduled.

I'm not familiar with this version of SAP 760 and was wondering if anyone can recommend the next steps or what to test.

The application teams and owners are supposed to be responsible for the application. The problem we often encounter is once it moves to Citrix and having everything in Citrix it is by default a Citrix issue. Gone are the days where it is a "network issue." If it is hosted in Citrix then Citrix is the defacto blame.

SAP Support will not provide support due to this being "development."

My application and development teams are new to SAP 760 as well. Current production is SAP 740 front-end on Citrix side.

The common denominator is the middleware server which is hosted on the ESX cluster whereas HANA is hosted on IBM.

Next steps, I have set up an isolated environment that is a copy of production with WireShark, Burp Proxy, Fiddler 4, Windows Debug Toolkit, Process Monitor, Process Explorer, and SAP 760 Update 6.

The GUI is "slow" whether it runs on Citrix shared VDI or a dedicated single-use Windows 10 VDI.

There have also been some odd issues with Font's and the font's changing in the GUI.

If I turn this isolated environment up and have a dedicated tester - what am I looking for in the best-case scenario?

I'm asking those who might have already done this in a like environment.

Server 2016 Shared VDI running Citrix 7.15 VDA - hosted exclusively in physical memory.

SAP GUI 7.60 Patch 6 (replacement to 740).

Or perhaps, you just have some ideas as to where to start.

I need to provide a direction and I can get all the teams on a call if necessary but I want to be able to drive that call in the right direction so we find root cause.

I am also considering turning up multiple dedicated VDI options with different versions of SAP GUI. I've seen in the blogs and forums where 760 has issues with certain things but I'm not sure which version is fully compatible with our new backend. This might be a compatibility issue where we need to downgrade the client. I know that 740 does not support the currently needed functionality but perhaps there is one between 740 Update X and 760 Update 6 that actually works on Server 2016 Data Center and/or Windows 10. I've spun up Windows 10 1903, 1906, and to 2004. The perception of slowness exists in all of them.

Unfortunately, the feedback of "it should run faster" is not helping. I could use some published statistics or KB or Blogs that touch on this specific topic. Or, other logging tools than what I've stated above. Nothing trumps Wireshark but Wireshark might show spurious retransmissions or lost packets and so forth but this is all on the same physical network. In other words, there are no routers or firewalls between Citrix & Middleware and DB tiers.

I'm sure there is information missing. If there is something I can obtain from the team that would help - let me know.

[Sandra_Rossi]

I confirm that it's much too long. My opinion is that "acceptable" is 1 second maximum.

I didn't read your whole post, sorry if my point is unrelated: did you check what happens with direct access to SAP GUI? If there's the same issue, maybe it's about host name resolution or your SAP server is vampirized by a ghost process (check CPU% used when "nothing" runs).

[daniel_mccollum]

a very detailed question, & disappointing that SAP isn't offering advice as a non-PRD system. It will clearly be productive at some point. Id suggest escalation with your account manager if it continues to resist your project consultants analysis.

It may be that you've already considered these suggestions, but these notes offer guidance on analyzing your performance constraints. (& yes even though its anecdotal, those numbers appear horribly slow)

https://launchpad.support.sap.com/#/notes/2880032
https://launchpad.support.sap.com/#/notes/851012

[daniel_mccollum]

Another Note that may be of relevance. https://launchpad.support.sap.com/#/notes/2888454

[former_member697481]

I appreciate the feedback as this continues to be a point of frustration. This is in a phase of pre-production where we have a new customer contract that begins the 1st quarter of next year. A key issue is resources where the internal resources and application owners are not familiar with the application. This first phase of testing is to gauge and functionality testing relative to the second phase of building out the customer production solution.

My primary request is simply to create a runbook and provide feedback based on specific validation testing. The Citrix team is not familiar with all aspects of this application. I will review the links provided and at this point, we have built a dedicated Silo to host SAP760 and Hana front-end components. When we hand it off to application owners to install the application piece they are not the ones that perform validation testing.

Keeping in mind that Citrix PVS streamed images are hosted in Memory. There are no disks or storage relative to IO. 48 GB of RAM per server instance and this is hosted in the virtual memory of the ESX host. I've modified the processes to run at a higher priority, CPU pinning, and several other tactics. It has no impact either way because there are no indications of performance-related issues on that platform. A few examples of known open issues. The biggest problem is obtaining feedback.

Below a summarized overview of the current issues and what is escalated. No feedback from SAP.

Delay in swapping between SAP screens when using the eWM monitor è

• Basis has done many traces with below observations

• Windows 10 reacts faster than Windows Server 2016
• On Windows Server 2016 double buffering enabled

Talking to PE, learned that this behavior is also reported from the side of the local users working in a production environment. (not just EWM, but also SSO on ECC6

• User working in SAP S/4 HANA eWM environment suddenly gets confronted with a change of fonts. (Yet no one can tell me what Font is needed or details)
• ??? Where can I find information about this font issue?

• The consequence is that the screens are partially filled or not readable characters with as the only resolution to reboot the computer, as logging out of SAP is not enough to get it resolved.

• Slowness experienced by multiple people when working with non-SAP application (VDI environment) - What does this mean?? Slowness is subjective - I need specifics. What process, what sequence, what is your expectation - how fast should it run? No answer.

• Excel : Cell update take long time around 15 seconds

• Word : During typing text display with a delay on the screen - During closure of Word application , 50% of the screen remains vissible for almost 1 min

• Slowness experienced talking to PE team, they confirm that everything works fine when using Microsoft 2016 applications on their local drive multiple people when working with the non-SAP application (VDI environment)

Any thoughts are appreciated.

[Sriram2009]

Hi Brain.

SAPGUI 760 with Citrix is having perfomance issue, if you are access the sapgui 760 without citrix what is the impact?

Regards

SS

[former_member697481]

One more, the customer sent a vague email that stated double buffer was "on here" and "off here" All of my searches for buffer and SAP point back to database or middleware tier.

This one

• Basis has done many traces with below observations

• Windows 10 reacts faster than Windows Server 2016
• On Windows Server 2016 double buffering enabled

But I cannot get clarity as to what double buffering means relative to SAP? To me, double buffer is hundreds of different things. C#, .NET and code related perhaps.

I simply cannot get a better explanation as to what double buffer is meant and any insight here would be greatly appreciated.

No performance or impact related information was provided.

[former_member697481]

Adding to this topic, I want to highlight a line item that was provided yesterday and where at no point in this project was Office Suite used synonymously with SAP and performance.

I'm preparing to role out a dedicated SAP 760 with Hana PVS image. And then I see:

1) 15 - 20 seconds to change color text in Excel, Update in Excel changing cell take about 15 seconds

2) Swapping none sap screens very slow

3) Word changing very slow Outlook pops up when typing in other application

4) When closing Word application half of the window remains visible for 50 seconds

I think we might have a potential big win here if without asking direct we can get feedback that ties the performance issues to Office suite – changes the context.

It makes sense and is a simple oversight on their part where the perception is that SAP is “too slow” when the reality is the interoperability and required or having made it a requirement / dependency for Office.

The 20 seconds on Word and so forth was previously stated as “too slow”.

Which is fine but at least now we know what “slow” means and Office suite is something we can tweak just for SAP silo.

--------end

I would just like to find one document that speaks to SAP and Best Practices and tweaks for Office and Office is being used to gauge performance from the very beginning of this project. My teams focus has been SAP and Hana.

For all we know, up until now the feedback is “too slow” and without context or substance. Until now, we have no references to Office and SAP 760 from the very beginning until now. It could be 80% of the slowness perception, we simply don’t know.

IMO, this is first priority is determining as to whether or not we need to include Office Suite applications in the scope of this project as a required dependency. If I reflect back on prior conversations and limited feedback the Office related issues make sense but for whatever reason it was stated as SAP performance issue and highly probable referencing Office application as if it were SAP 760.

It lines up with what has been said regarding 20 seconds and 30 seconds.

For whatever reason it is just now being brought up. The remaining items on that list other than “font” are references to backend processes.

[former_member697481]

S Sriram -

To answer your question the impact is that Citrix is preventing the SAP 760 and Hana project from meeting deadlines. This first S1 phase is internal people test driving the provided development environment to prepare for a new customer contract starting 1st quarter of next year.

Up until yesterday, the Office Suite had not been mentioned relative to performance.

The issue that is considered the most critical at this moment is the screenshot attached. On the right hand side is some type of Font issue that locks the environment up and requires a forced reboot of the VDI.

--------------

Per the customer, The drop downs and screens go weird, please see the examples below and what I need to do to resolve the issue. It seems to be something with how the VDI/Citrix reacts with SAP.

First notice the drop-down was giving unreadable results in transaction SPRO.

------------end

I don't see how this ties in to being a Citrix issue? Instead SAP issue but I'm always open-minded and that is why I'm posting this information. Our team, without SAP support, does not know SAP or Hana. They see this as a showstopper and rightly so - it freezes the application and session and it is not consistent but instead random.

Using my S-ID, I am just trying to find articles and any type of best practices relative to Citrix or installing this application. My team installed SAP with minimal guidance and no documentation. That process should be handing a server off for the application owners to install on Citrix but those individuals claim they do not know the application well enough in that respect and SAP the vendor has not provided any information.