[HANA System Replication] end-to-end Client Reconnect

[nicholas_chang]

I've seen many posts on how to setup Hana System Replication and its takeover, however, none or few of the post that covers client reconnect after sr_takeover.

In order to ensure the client is able to find seamlessly the active HDB node (doesn't matter primary or secondary), we can either use IP Redirection or DNS Redirection. In this blog, i'll emphasize on simple IP redirection as it is much easier, faster and less dependancies compare to DNS redirection.

For details info on IP and DNS redirection, please refer to the guide:

http://scn.sap.com/docs/DOC-63221

Introduction to High Availability for SAP HANA

How to Perform System Replication for SAP HANA

First of all, we need to identify a virtual hostname/ip, create them in your DNS. Below is the sample virtual hostname/ip and physical hostname/ip used:

Virtual IP/Hostname: [10.X.X.50 / hanatest]

Primary Physical IP/Hostname: 10.X.X.20 / primary1

Secondary Physical IP/Hostname: 10.X.X.21 / secondary2

In normal operation, [10.X.X.50 / hanatest] is bind to Primary Physical Host - primary1

SAP instances, HTTP, BO, SAP DS and etc are connect to HDB via [10.X.X.50 / hanatest]

During any unplanned outage/ Disaster, [10.X.X.50 / hanatest] will be unbind from primary host and bind to Secondary Physical Host.

And below are the steps on mapping virtual IP [10.X.X.50] to its MAC address in Linux:

1) Bind virtual ip (10.X.X.50) to Primary Physical Host

primary1:/etc/init.d # ifconfig eth0:0 10.X.X.50 netmask 255.255.255.0 broadcast 10.X.X.255 up

2) Check eth0 entry:

primary1:~ # ifconfig

eth0      Link encap:Ethernet  HWaddr XX:XX:XX:XX:XX:C2

          inet addr:10.XX.XX.21  Bcast:10.XX.XX.255  Mask:255.255.255.0

eth0:0 Link encap:Ethernet  HWaddr XX:XX:XX:XX:XX:C2

inet addr:10.XX.XX.50  Bcast:10.XX.XX.255  Mask:255.255.255.0

UP BROADCAST RUNNING MULTICAST MTU:1500  Metric:1

3) Ping hanatest and it is resolvable.

PING hanatest (10.XX.XX.50) 56(84) bytes of data.

64 bytes from hanatest (10.XX.XX.50): icmp_seq=1 ttl=64 time=0.028 ms

64 bytes from hanatest (10.XX.XX.50): icmp_seq=2 ttl=64 time=0.038 ms

64 bytes from hanatest (10.XX.XX.50): icmp_seq=3 ttl=64 time=0.024 ms

4) For all HDBs' clients, connect using virtual hostname [hanatest]

a) SAP - hdbuserstore:

sidadm 52> hdbuserstore list

DATA FILE       : /home/sidadm/.hdb/XX/SSFS_HDB.DAT

KEY DEFAULT

  ENV : hanatest:30515

  USER: SAPSID

Login to SAP, and you'll see DBHOST is pointed to primary1

In DBACOCKPIT -> DB CONNECTION -> Ensure virtual host is used:

b) HANA Studio:

Sevices are running on Physical Host primary1

c) ODBC - connect using virtula host

d) HTTP - xsengine

http://hanatest:8005/

http://hanatest:8005/sap/hana/xs/admin

------------------Unplanned outage *DISASTER*:--------------------------------------------

During Disaster. we will:

i) Ensure primary HDB is down and not accessibile to avoid any split-brain

ii) Unbind virtual ip [10.X.X.50] currently binding to Primary Physical Host. in ifconfig, eth0:0 should not visible after you execute below command.

primary1:~ # ifconfig eth0:0 10.XX.XX.50 down

iii) clear ARP cache in client [optional]

iv) initiate -sr_takeover and wait HDB on secondary to be up and ready

v) Once HDB on secondary host is up and running, bind virtual ip [10.X.X.50] to Secondary Physical Host

secondary2:/etc/init.d # ifconfig eth0:0 10.XX.XX.50 netmask 255.255.255.0 broadcast 10.XX.XX.255 up

secondary2:~ # ifconfig

eth0      Link encap:Ethernet  HWaddr XX:XX:XX:XX:XX:C8

          inet addr:10.XX.XX.21  Bcast:10.XX.XX.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500 Metric:1

eth0:0 Link encap:Ethernet  HWaddr XX:XX:XX:XX:XX:C8

inet addr:10.XX.XX.50  Bcast:10.XX.XX.255  Mask:255.255.255.0

UP BROADCAST RUNNING MULTICAST MTU:1500  Metric:1 10)

vi) Ping hanatest and it is resolvable. virtual host [hanatest] is currently bind to Secondary Physical Host - secondary2

PING hanatest (10.XX.XX.50) 56(84) bytes of data.

64 bytes from hanatest (10.XX.XX.50): icmp_seq=1 ttl=64 time=0.028 ms

64 bytes from hanatest (10.XX.XX.50): icmp_seq=2 ttl=64 time=0.038 ms

64 bytes from hanatest (10.XX.XX.50): icmp_seq=3 ttl=64 time=0.024 ms

----------------------- End-to-End Client Reconnect Verification -----------------------------

once done, you can perform end-to-end client reconnect verification without the need to perform any changes.

a) SAP Instances after sr_takeover and running on secondary host:

a.i) Developer trace – SAP reconnect ok to secondary host:secondary2

B Connection 1 opened (DBSL handle 1)

B successfully reconnected to connection 1

B ***LOG BYY=> work process left reconnect status [dblink       2158]

M ThHdlReconnect: reconnect o.k.

M

M Tue Sep 29 13:50:13 2015

M ThSick: rdisp/system_needs_spool = false

C FDA DB protocol version from connection 0 = 1

B Tue Sep 29 13:55:16 2015

B Connect to XXX as system with hanatest:30515

C Try to connect as system/<pwd>@hanatest:30515 on connection 1 ...

C

C Tue Sep 29 13:55:17 2015

C Attach to HDB : 1.00.095.00.1429086950 (fa/newdb100_rel)

C fa/newdb100_rel : build_weekstone=0000.00.0

C fa/newdb100_rel : build_time=2015-04-15 10:44:35

C Database release is HDB 1.00.095.00.1429086950

C INFO : Database 'TST/05' instance is running on 'secondary2'

C INFO : Connect to DB as 'SYSTEM', connection_id=300064

a.ii) SAP status (HDB switched from primary1 -> secondary2)

b) HANA Studio

c) ODBC

d) HTTP

xsengine: http://hanatest:8005/

http://hanatest:8005/sap/hana/xs/admin

Hopefully this blog will serve as a reference for client reconnect strategy when setting up Hana system replication. Also, hopefully more consultant are aware of the three execellent guides above, which provided detailed info on client reconnect mechanism and hana system replication.

Cheers,

Nicholas Chang

[Muadib]

Amazing !!! Thanks Nicholas

I was precisely looking at more information on how to ensure SAP connections recovery after Hana failover.

I still have a question : we are in the process of implementing system replication on HANA SP11, we are still unsure on whether we would implement IP Redirection or DNS Redirection.

We are reviwing the white paper "SAP HANA – High Availability" : in the chapter dedictaed to IP rediretion it is mentionned that a L2 switch is necessary, I am no network expert, and even after googling it, I am quite confused about what is a L2 Switch.

A colleague told me that it would require some additional equipment to implement L2 Network Switches ...

We do not use a cluster management software.

In order to prevent brain split, Is it sufficient to run the command uou mention, or do we need someting else ?

Thanks

[Former Member]

Thanks for this 

[nicholas_chang]

Hi Raoul,

You can prevent brain-split as long as your primary is down and not accessible, and virtual hostname is moved to secondary on take over.

Despite cluster management software, you can do it manually or write a script to take down primary and move the virtual ip to secondary in the event of DR.

Cheers,

Nicholas Chang

[nicholas_chang]

Thanks @Michael Healy! And congrats on you recent acchivement! 😉

[annaeymnd]

Great post.

I use a little different methode. I dont like to play arround with ip-adresses and wanted to avoid split-brain problems.

I use a virtual "service"-name on the app servers which is defined in /etc/hosts file.

Like "<abapsid>db --> <physical_ip_of_Primary_hana>"

So, in case of a takeover I only have to switch IP-adresses in the hosts file on the app Servers. works instantly.

of course only recommended if your number of hana Clients (app Server) is not too high.

because as sap Administrator I have no general root access, I use sudoedit for editing the /etc/hosts

[nicholas_chang]

Hi Florian,

Thanks for sharing. It works fine for SAP on HANA. It may not work for external application like eg: SDA, BODS, and etc.

[former_member182657]

Hi Nicholas,

Thanks for sharing  a wonderful document with the community.This document was the true turning point for making my SAP HANA HA a successful setup.

Best Regards,

Gaurav Rana

[former_member220499]

Thanks for this amazing post. I want to share our aproach. Simply we added the two hostanames and ports of the primary and secondary HANA servers in our System Replication scenario in the hdbuserstore file. The application server connects to the primary, and, when is down, reconnect autmatically to the secondary. Like this:

hdbuserstore SET DEFAULT "<hostname_node1>:3<inst>15; … ;<hostname_node(n)>:3<inst>15" <sapsid> <password>

More info here: https://help.sap.com/viewer/b3ee5778bc2e4a089d3299b82ec762a7/2.0.00/en-US/ddbdd66b632d4fe7b3c2e0e6e3...

Best regards.