Use SLES 15.2 To Automate SAP HANA High Availability

[williams_ruter3]

4 years ago, I wrote a blog explaining how to manually setup and configure SLES 12.2 with SAP HANA to run an automated HA solution from performance-optimized to resource optimized “SAP Hana TDI setup – SAP Hana HSR 3 Tiers with SAPHanaSR”
As a strategic partner of SAP, in its latest revision “15.2”, now SLES provides a fully integrated process to accelerate the deployment of SAP HANA in a highly available environment. Awesome right!!!

In this first article of a series of four, I will talk to you about how SLES HA Automation process and integration work with SAP HANA to facilitate the management of these systems, but also how to get the best of this marriage by leveraging enhanced native SLES tools specifically for SAP platform.

 

Some references

Before starting, it is important to make sure that you have the right information in regard to the technology, tools, and components used to achieve and realize some of the scenarios that I will cover. Here is a collection of guides, references, and SAP Notes to be reviewed.

SAP Notes

• 2235581 - SAP HANA: Supported Operating Systems


• 2684254 - SAP HANA DB: Recommended OS settings for SLES 15 / SLES for SAP Application 15


• 1999880 - FAQ: SAP HANA System Replication


• 2407186 - How-To Guides & Whitepapers for SAP HANA High Availability

Guides

• SUSE Linux Enterprise Server for SAP Applications 15 SP2


• SAP Hana Administration Guide 2.0 SP5


• SAP HANA System Replication


• FreeNAS® 11.3-U5 User Guide

 

What and how to articulate it?

I know you want to go straight to the config part of the solution ?, but I have to be diligent with you and at least show some updates from the components used in my previous blog since those updates/upgrades are the essences for the rest of the documentation.

VMware Platform

vSphere 6.0 to 6.7 – vSphere 6.7 contains major improvements which provide such as 2x better performance, 3x reduction in memory usage as well as 3x faster operation relating to VDRS.

FreeNAS Platform

FreeNAS 9.0 to 11.3-U5 – Not just a virtual appliance, the latest version of FreeNAS offer the ability to add virtual machines, Docker, and cloud synchronization support, it includes a mobile-friendly Angular web interface, improved iocage plugin system, real-time APIv2, and synchronization with many more cloud storage providers.

Now that I have talked about the 2 main pieces, let’s dive into SLES For SAP 15 SP2, as I mentioned earlier this revision of SLES provides an embedded mechanism to set up and configure SAP HANA to run in HA mode with auto-failover (scale-up).
I will not cover how to install SLES for SAP step by step neither how to install SAP HANA, but will cover the detail to realize the setup.

Operating System Core Install

By design, SLES 15.2 media gives you the ability to install your OS according to a specific product in order to install the necessary extension and module

The Basesystem Module and Server Application Module are selected de facto, make sure to select SAP Application Module which will auto-select the SLES HA Extension

Your System Role is an important point; however, you can choose only one role.

A very important point, that people don’t think about often, is the NTP server. You need to make sure to sync your server time.

 

Get your system ready for HA

Because the environment needs to be configured as a cluster, to guaranty the consistency of the following an SBD device needs to be attached on both server parts of the cluster.

This is where I use FreeNAS to create and share an iSCSI bloc to be attached

Now in order to leverage the SLES feature for SAP HANA, specifics package needs to be deployed since they don’t come natively in the core install if you don’t explicitly select them. The yast2-sap-ha for the cluster setup, and the yast2-hana-update to maintain your SAP system within the cluster.

 

Ready to run the procedure

My DNS is updated with the virtual ip and hostname given for my cluster

Both SAP Hana is installed according to the recommendation, my primary system is backed up
On my primary Hana I have created a key for backup

The PKI SSF data and file has been copied from the primary to the secondary node

• scp -p /usr/sap/HB0/SYS/global/security/rsecssfs/data/SSFS_HB0.DAT hb0adm@hanadb03:/usr/sap/HB0/SYS/global/security/rsecssfs/data/


• scp -p /usr/sap/HB0/SYS/global/security/rsecssfs/key/SSFS_HB0.KEY hb0adm@hanadb03:/usr/sap/HB0/SYS/global/security/rsecssfs/key/

Now from the primary, as root, I invoke “yast2” and select “HA Setup for SAP Products”

Select the “Scale UP: Performance-optimized”

On the communication layer, provide the ip range from the nic card to be used.

Edit both server ips and hostname and also make sure to select “Append to /etc/hosts”, because of the root key exchange between server, the password will be asked

The NTP server configured earlier should appear

It the following screen, I need to add the SBD device attached to the server

How to check the iSCSI? run the command “fdisk -l”

For the Watchdog, I will choose “softdog”

In this last step, I provide the system information such, SID, instance number, VIP, the site name and the behavior for taking over and auto registration

Review the config and hit Install

During the process, some package will be required to be installed

One completed a review of the log to make sure no error encored and click finish

Validate the cluster and SAP HANA replication, to do so I will open the HAWK interface but using the virtual hostname of my cluster on port 7630

At the OS level if I run the “crm status” the status of the resources will show green

Finally, I will register my cluster in my Hana Cockpit

I can see that my hanadb02 node the primary used as per the Cockpit

 

Test the failover

I now test the failover process by killing the hb0adm process on the hanadb02 (node1) and check the takeover on Hana Cockpit and the OS level (because HAWK is only deployed on node1)
I my primary node (hanadb02) I run my killer command ? “pkill -9 HB0”

 

On the logical order, after the pkill command is invoked, I run the “crm status” command and I can see that hanadb02 resources are stopped

On the Hana Cockpit, I can see that the replication is broken on Site1

After a few second the takeover is completed and we can see the hanadb03 acting as the master now

And finally, the Hana Cockpit shows the hanadb03 as well the master after refreshing without changing anything

 

Conclusion

The new embedded process provided by SLES is very good to automate build and task for SAP HANA, very straight forward and efficient if you are not a Linux savvy person to deal with multiple file manipulation and OS command.
In the next portion of this of articles, I will talk about the maintenance part of this process for SAP HANA, Cost Optimized configuration (QA/PRD), and the Chained setup (3 tiers).

Series

Part 1: Current
Part 2: Use SLES 15.2 To Automate SAP HANA HA Maintenance

[dvankempen]

Awesome, indeed, Williams. Thanks for the post.

[ronnyfiebig]

Thank you Williams for this post!

[williams_ruter3]

Hey Denys, it is a pleasure to share with others

[williams_ruter3]

You are very welcome Ronny.

[cosmo83]

If you are not separating the iscsi device on a separate physical channel. the whole setup doesnt make sense. Separating the fencing device channel is very important for your high availability cluster. Having sbd  as the fencing device  especially for HANA instances (Since they mostly will not be carrying a FC card) only to complete the architecture you are looking for is a bad idea.

[williams_ruter3]

Hello Phani,

Thanks for your input, if you pay attention to my current setup i used only one channel to simplify the process only.

I talk about the feature provided by SLES and about the component to be used, the focus was not the design here 😉

As you know the SLES HA support FC or Iscsi SAN, so your comment is valid from network segregation point of view.

However, i appreciate all the comment as long as they can help others in their reflection.

Thank you.

[cosmo83]

Iam just pointing out the issue. After seeing the same blunders being done in production setups.  Cloud providers are suggesting to use this mechanism without understanding the repercussions of the same.

I have at least 3-4 customers who have used this approach of using the iscsi devices on the same channel as data for the fencing devices. Whenever i have discussions with them on the reason for using the iscsi devices as fencing devices, they point me to Cloud Provider Documents / such blogs .

[fraseggar]

Hi Williams,

I've followed your intructions but in my case I can't Add any watchdogs to configure because I can't see any wathcdog in the list. However if I execute the comand "lsmod | grep do" I see that softdog is running.

Could  you help me, please ?

Thanks in advance

Regards

FSG

[williams_ruter3]

Hello Francesc,

Can you share the screen picture of the "watchdog setup" after that you clic on "Add", you must have a list that come up.

Also which version of SLES are you running ?

 

Williams

[fraseggar]

Hi,

We've installed SuSe15 SP4 over Intel X386. When I select "Add" I don't have any module to select.

I've the possibility to continue with the installation process but it crash with the next log error after select "Install". I'm sure the problem is that these script try to read the watchdog information and it's find a null instead of character.

 

2023-04-18 21:15:42 <1> agora25(34603) [Ruby] modules/Misc.rb(SysconfigRead):182 ."/etc/os-release"."VERSION_ID": '15.4'

2023-04-18 21:15:42 <1> agora25(34603) [Ruby] system/hana.rb(version):103 --- called SapHA::System::HanaClass.version(HPP) ---

2023-04-18 21:15:42 <1> agora25(34603) [Ruby] system/shell_commands.rb(su_exec_outerr_status):92 Executing ["HDB", "version"] as user hppadm

2023-04-18 21:16:24 <1> agora25(34603) [Ruby] wizard/overview_page.rb(block in main_loop):70 --- SapHA::Wizard::ConfigurationOverviewPage.main_loop: event={"EventReason"=>"Activated", "EventSerialNo"=>24, "EventType"=>"WidgetEvent", "ID"=>:next, "WidgetClass"=>:PushButton, "WidgetDebugLabel"=>"Install", "WidgetID"=>:next} ---

2023-04-18 21:16:24 <1> agora25(34603) [Ruby] modules/Misc.rb(SysconfigRead):182 ."/etc/os-release"."VERSION_ID": '15.4'

2023-04-18 21:16:24 <1> agora25(34603) [Ruby] system/hana.rb(version):103 --- called SapHA::System::HanaClass.version(HPP) ---

2023-04-18 21:16:24 <1> agora25(34603) [Ruby] system/shell_commands.rb(su_exec_outerr_status):92 Executing ["HDB", "version"] as user hppadm

2023-04-18 21:16:24 <1> agora25(34603) [Ruby] system/connectivity.rb(run_rpc_servers):222 --- SapHA::System::ConnectivityClass.run_rpc_servers ---

2023-04-18 21:16:24 <1> agora25(34603) [Ruby] system/connectivity.rb(run_rpc_server):72 --- SapHA::System::Host.run_rpc_server ---

2023-04-18 21:16:24 <1> agora25(34603) [Ruby] system/shell_commands.rb(exec_status):43 Executing command ["ssh", "-o", "StrictHostKeyChecking=no", "-f", "root@agora26", "systemd-cat /usr/bin/ruby /usr/share/YaST2/lib/sap_ha/rpc_server.rb"]

2023-04-18 21:16:24 <1> agora25(34603) [Ruby] modules/Progress.rb(New):340 Progress::New(SAP High-Availability Setup, 2, ["Configure local node", "Configure remote node [agora26]"])

2023-04-18 21:16:24 <1> agora25(34603) [Ruby] sap_ha/sap_ha_installation.rb(local_configuration):108 --- SapHA::SAPHAInstallation.local_configuration: configuring current node ---

2023-04-18 21:16:24 <1> agora25(34603) [Ruby] sap_ha/node_logger.rb(method_missing):161 Starting setup process on node agora25

2023-04-18 21:16:24 <1> agora25(34603) [Ruby] sap_ha/sap_ha_installation.rb(block in local_configuration):113 --- SapHA::SAPHAInstallation.local_configuration: configuring NTP Configuration ---

2023-04-18 21:16:24 <1> agora25(34603) [Ruby] modules/Misc.rb(SysconfigRead):182 ."/etc/os-release"."VERSION_ID": '15.4'

2023-04-18 21:16:24 <1> agora25(34603) [Ruby] sap_ha/node_logger.rb(method_missing):161 Appying NTP Configuration

2023-04-18 21:16:24 <1> agora25(34603) [Ruby] sap_ha/sap_ha_installation.rb(block in local_configuration):113 --- SapHA::SAPHAInstallation.local_configuration: configuring Watchdog Setup ---

2023-04-18 21:16:24 <1> agora25(34603) [Ruby] sap_ha/node_logger.rb(method_missing):161 Appying Watchdog Configuration

2023-04-18 21:16:24 <3> agora25(34603) [Ruby] system/watchdog.rb(watchdog?):98 Module '' is not a watchdog!

2023-04-18 21:16:24 <3> agora25(34603) [Ruby] system/watchdog.rb(watchdog?):98 Module '' is not a watchdog!

2023-04-18 21:16:24 <3> agora25(34603) [Ruby] sap_ha/node_logger.rb(method_missing):161 Cannot install a watchdog module

2023-04-18 21:16:24 <1> agora25(34603) [Ruby] system/shell_commands.rb(exec_outerr_status):68 Executing command ["/usr/sbin/modprobe", nil]

2023-04-18 21:16:24 <3> agora25(34603) [Ruby] clients/sap_ha.rb(rescue in run_installation):404 An error occurred during the installation

2023-04-18 21:16:24 <3> agora25(34603) [Ruby] clients/sap_ha.rb(rescue in run_installation):405 no implicit conversion of nil into String

2023-04-18 21:16:24 <3> agora25(34603) [Ruby] clients/sap_ha.rb(rescue in run_installation):406 ["/usr/lib64/ruby/2.5.0/open3.rb:199:in `spawn'", "/usr/lib64/ruby/2.5.0/open3.rb:199:in `popen_run'", "/usr/lib64/ruby/2.5.0/open3.rb:190:in `popen2e'", "/usr/lib64/ruby/2.5.0/open3.rb:349:in `capture2e'", "/usr/share/YaST2/lib/sap_ha/system/shell_commands.rb:69:in `exec_outerr_status'", "/usr/share/YaST2/lib/sap_ha/system/watchdog.rb:126:in `load'", "/usr/share/YaST2/lib/sap_ha/configuration/watchdog.rb:83:in `block in apply'", "/usr/share/YaST2/lib/sap_ha/configuration/watchdog.rb:81:in `each'", "/usr/share/YaST2/lib/sap_ha/configuration/watchdog.rb:81:in `apply'", "/usr/share/YaST2/lib/sap_ha/sap_ha_installation.rb:114:in `block in local_configuration'", "/usr/share/YaST2/lib/sap_ha/sap_ha_installation.rb:112:in `each'", "/usr/share/YaST2/lib/sap_ha/sap_ha_installation.rb:112:in `local_configuration'", "/usr/share/YaST2/lib/sap_ha/sap_ha_installation.rb:44:in `run'", "/usr/share/YaST2/clients/sap_ha.rb:402:in `run_installation'", "/usr/share/YaST2/clients/sap_ha.rb:227:in `block in initialize'", "/usr/lib64/ruby/vendor_ruby/2.5.0/yast/builtins.rb:546:in `eval'", "/usr/share/YaST2/modules/Sequencer.rb:247:in `WS_run'", "/usr/share/YaST2/modules/Sequencer.rb:318:in `block in Run'", "/usr/share/YaST2/modules/Sequencer.rb:310:in `loop'", "/usr/share/YaST2/modules/Sequencer.rb:310:in `Run'", "/usr/share/YaST2/clients/sap_ha.rb:246:in `main'", "/usr/share/YaST2/clients/sap_ha.rb:448:in `<module:Yast>'", "/usr/share/YaST2/clients/sap_ha.rb:41:in `<top (required)>'", "/usr/lib64/ruby/vendor_ruby/2.5.0/yast/wfm.rb:347:in `eval'", "/usr/lib64/ruby/vendor_ruby/2.5.0/yast/wfm.rb:347:in `run_client'", "/usr/lib64/ruby/vendor_ruby/2.5.0/yast/wfm.rb:207:in `call_builtin'", "/usr/lib64/ruby/vendor_ruby/2.5.0/yast/wfm.rb:207:in `call_builtin_wrapper'", "/usr/lib64/ruby/vendor_ruby/2.5.0/yast/wfm.rb:196:in `CallFunction'", "/usr/lib/YaST2/bin/y2start:60:in `<main>'"]

2023-04-18 21:16:24 <3> agora25(34603) [Ruby] clients/sap_ha.rb(rescue in main):254 Error occurred during the unattended installation: no implicit conversion of nil into String

2023-04-18 21:16:24 <1> agora25(34603) [ui] YPushButton.cc(setRole):170 Guessing function key F9 for YPushButton "Stop" at 0x7f7c64045a40 from button role YCancelButton

2023-04-18 21:16:24 <1> agora25(34603) [ui] YPushButton.cc(setFunctionKey):202 Guessing button role YOKButton for YPushButton "OK" at 0x7f7c64034b10 from function key F10

2023-04-18 21:16:34 <1> agora25(34603) [Interpreter] bin/y2start:60 Called YaST client returned.

2023-04-18 21:16:34 <1> agora25(34603) [ncurses] YNCursesUI.cc(~YNCursesUI):121 Stop YNCursesUI

2023-04-18 21:16:34 <1> agora25(34603) [ncurses] NCurses.cc(~NCurses):145 Shutdown NCurses...

2023-04-18 21:16:34 <1> agora25(34603) [ncurses] NCurses.cc(~NCurses):164 NCurses down

2023-04-18 21:16:34 <1> agora25(34603) [Y2Ruby] binary/YRuby.cc(~YRuby):117 Shutting down ruby interpreter.

2023-04-18 21:16:34 <1> agora25(34603) [Y2Perl] YPerl.cc(destroy):164 Shutting down embedded Perl interpreter.

2023-04-18 21:16:34 <1> agora25(34574) [ncurses] NCurses.cc(RedirectToLog):565 isatty(stderr)yes

2023-04-18 21:16:34 <1> agora25(34574) [ncurses] NCurses.cc(RedirectToLog):574 isatty(stdout)yes

2023-04-18 21:16:34 <1> agora25(34574) [Ruby] clients/menu.rb(Launch):307 Got 0 from /sbin/yast sap_ha  >&2

Thanks in advance