Hitachi / HDS - Best Practices Guide for using Hitachi LPAR

[Former Member]

1. Purpose

This document introduces how to construct production HANA environment on Hitachi Compute Blade CB520X in LPAR mode. The document consists of the following 3 topics:

1. How to tune EFI parameters.
2. How to setup HVM
3. How to install OSs

2. How to construct a HANA environment on CB520X LPAR

2.1. How to tune EFI parameters

This session will show how to tune EFI parameters on a HVM-mode blade. Basic-mode blade does not need this tuning.

1. Connect to CB500 SVP web console.
2. Login to the web console.
3. Click a blade figure in “Chassis Status”.
4. Click “Server Blade Action” in “Condition” tab, and then choose “Start remote console”.
5. Power-on the blade.
6. Push F1 to enter EFI setup menu.
7. Select “System Settings” -> “Processor”.
8. Set “C1E” to disable and “C-state” to disable.
9. Save and Exit the EFI settings.

2.2. How to setup HVM

2.2.1. How to enable/disable HVM-mode

This session will show how to enable/disable HVM-mode.

Note: This setting can be changed only when the blade is powered-off.

1. Connect to CB500 SVP web console.
2. Login to the web console.
3. Click a blade figure in “Chassis Status”.
4. Click “Settings” tab, then click “Show LP settings” button.
5. Click “Change “LP mode” button.
6. Switch “Basic”/”LP mode” to disable/enable HVM-mode, then confirm.

2.2.2. Initial HVM Setup and Optimization for HANA

This session will show how to setup HVM when the blade is powered-on first time.

1. Connect to CB500 SVP via SSH or Telnet.
2. Login (Username and password are the same as the one for CB500 Web console).
3. Enter following command to power-on.

$ poweron blade <blade number> -F

1. Enter following command to connect the serial console of the blade.

$ change console -b <blade> -F

1. Expand SSH/Telnet window to show HVM screen. (80 characters x 25 characters are needed for the window).
2. After the blade completely booted, set HVM IP address/HVM VNIC System Number, and then press F10 key.

Note: HVM VNIC System Number must be unique number on your system.

1. Enter an IP address of the terminal PC to “LP CLI1 IP Address”.
2. Press Esc button to exit from the configuration window.
3. Select “HVM Options” by using cursor key, and then press Enter key.
4. Choose “PhyCPU C-State (>= C3)”, and then change it to “Disable”.
5. Press Esc button to exit from “HVM Options”.
6. Press F9 button to save the configuration.
7. Press Ctrl-b to exit the serial console.

2.2.3. How to create LPAR (One CPU socket per LPAR, on 2-blade SMP config)

This session will show how to create LPARs which have one CPU socket per a LPAR.

1. Connect to CB500 SVP via SSH or Telnet.
2. Login (Username and password are the same as the one for CB500 Web console).
3. Enter following command to connect the serial console of the blade.

$ change console -b <blade> -F

1. Expand SSH/Telnet window to show HVM screen. (80 characters x 25 characters are needed for the window).
2. Press Ctrl-t to show HVM screen.
3. Select “Physical Processor Configuration” by using cursor key, and then press Enter key.
4. Press F1 to add “Processor group”.
5. Do No.7 twice additionally.
6. Change processor group of each CPUs as follows:
   1. Processor#0 - #29 are in Group#0. (All CPUs in NUMA Node#0 are in Group#0.)
   2. Processor#30 - #59 are in Group#1. (All CPUs in NUMA Node#1 are in Group#1.)
   3. Processor#60 - #89 are in Group#2. (All CPUs in NUMA Node#2 are in Group#2.)
   4. Processor#90 - #119 are in Group#3. (All CPUs in NUMA Node#3 are in Group#3.)
7. Press Esc key to exit from “Physical Processor Configuration”.
8. Select “Logical Partition Configuration” by using cursor key, and then press Enter key.
9. Press F6 to add LPAR.
10. Enter name of the LPAR, processor scheduling(Scd), the number of processor(Pro), processor group(Grp), memory size(Mem), memory node(MN), and idle detection (ID) as follows:

LPAR#	Name	Scd	Pro	Grp	Mem	MN	ID
1	Any	Dedicate(D)	30	0	260096 (MB)	0	N
2	Any	Dedicate(D)	30	1	260096 (MB)	1	N
3	Any	Dedicate(D)	30	2	260096 (MB)	2	N
4	Any	Dedicate(D)	30	3	260096 (MB)	3	N

(Other parameters are not needed to change from default.)

1. Press Esc key to exit from “Logical Partition Configuration”.
2. Select “PCI Device Assignment” by using cursor key, and then press Enter key.
3. Allocate PCI Devices as follows: (Change scheduling mode to D+, press F10 key, and then assign the devices.)

PCI Dev#	3	4	5	6
Slot# (See the bottom of the window)	E<primary blade number>1	E<primary blade number>2	E<primary blade number>3	E<primary blade number>4
Device	Emulex BE3 Controller #1	Hitachi Fibre Channel	Emulex BE3 Controller #0	Hitachi Fibre Channel
Scheduling mode	Dedicated (D+)	Dedicated (D+)	Dedicated (D+)	Dedicated (D+)
LPAR#1	A	A	*	*
LPAR#2	*	*	A	A
LPAR#3	*	*	*	*
LPAR#4	*	*	*	*

PCI Dev#	8	9	10	11
Slot# (See the bottom of the window)	E<non primary blade number>1	E<non primary blade number>1	E<non primary blade number>2	E<non primary blade number>4
Device	Emulex BE3 Controller #0	Emulex BE3 Controller #1	Hitachi Fibre Channel	Emulex BE3 Controller #0
Scheduling mode	Dedicated (D+)	Dedicated (D+)	Dedicated (D+)	Dedicated (D+)
LPAR#1	*	*	*	*
LPAR#2	*	*	*	*
LPAR#3	A	*	A	*
LPAR#4	*	A	*	A

(Other devices are not needed to change from default.)

1. Press Esc key to exit form “PCI Device Assignment”.
2. Press F9 key to save these configuration.

2.2.4. How to create LPAR (Two CPU socket per LPAR, on 2-blade SMP config)

This session will show how to create LPARs which have two CPU sockets per a LPAR.

1. Connect to CB500 SVP via SSH or Telnet.
2. Login (Username and password are the same as the one for CB500 Web console).
3. Enter following command to connect the serial console of the blade.

$ change console -b <blade> -F

1. Expand SSH/Telnet window to show HVM screen. (80 characters x 25 characters are needed for the window).
2. Press Ctrl-t to show HVM screen.
3. Select “Physical Processor Configuration” by using cursor key, and then press Enter key.
4. Press F1 to add “Processor group”.
5. Change processor group of each CPUs as follows:
   1. Processor#0 - #59 are in Group#0. (All CPUs in NUMA Node#0 and #1 are in Group#0.)
   2. Processor#60 - #119 are in Group#1. (All CPUs in NUMA Node#2 and #3 are in Group#1.)
6. Press Esc key to exit from “Physical Processor Configuration”.
7. Select “Logical Partition Configuration” by using cursor key, and then press Enter key.
8. Press F6 to add LPAR.
9. Enter name of the LPAR, processor scheduling(Scd), the number of processor(Pro), processor group(Grp), memory size(Mem), idle detection (ID), and guest NUMA(NUMA) as follows:

LPAR#	Name	Scd	Pro	Grp	Mem (Node setting will be appeared by setting NUMA to Y)	ID	NUMA (Press F11/F12 to change screen)
1	Any	Dedicate(D)	60	0	Total: 520192 (MB) Node0: 260096 (MB) Node1: 260096 (MB)	N	Y
2	Any	Dedicate(D)	60	1	Total: 520192 (MB) Node2: 260096 (MB) Node3: 260096 (MB)	N	Y

(Other parameters are not needed to change from default.)

1. Press Esc key to exit from “Logical Partition Configuration”.
2. Select “Logical Processor Configuration” by using cursor key, and then press Enter key.
3. Change processor assignment of each logical processors as follows:
   1. Logical processor#0 - #59 are assigned to physical processor #0 - #59, respectively.
   2. Logical processor#60 - #119 are assigned to physical processor #60 - #119, respectively.
4. Press Esc key to exit from “Logical Processor Configuration”.
5. Select “PCI Device Assignment” by using cursor key, and then press Enter key.
6. Allocate PCI Devices as follows:

PCI Dev#	3	4	5	6
Slot# (See the bottom of the window)	E<primary blade number>1	E<primary blade number>2	E<primary blade number>3	E<primary blade number>4
Device	Emulex BE3 Controller #1	Hitachi Fibre Channel	Emulex BE3 Controller #0	Hitachi Fibre Channel
Scheduling mode	Dedicated (D+)	Dedicated (D+)	Dedicated (D+)	Dedicated (D+)
LPAR#1	A	A	*	*
LPAR#2	*	*	*	*

PCI Dev#	8	9	10	11
Slot# (See the bottom of the window)	E<non primary blade number>1	E<non primary blade number>1	E<non primary blade number>2	E<non primary blade number>4
Device	Emulex BE3 Controller #0	Emulex BE3 Controller #1	Hitachi Fibre Channel	Emulex BE3 Controller #0
Scheduling mode	Dedicated (D+)	Dedicated (D+)	Dedicated (D+)	Dedicated (D+)
LPAR#1	*	*	*	*
LPAR#2	A	*	A	*

(Other devices are not needed to change from default.)

1. Press Esc key to exit form “PCI Device Assignment”.
2. Press F9 key to save these configuration.

2.3. How to install OSs

This session will show how to install and configure OSs.

Note: HVM can attach USB device to a LPAR, thus OS installation will be done serially.

1. Connect to CB500 SVP web console.
2. Login to the web console.
3. Click a blade figure in “Chassis Status”.
4. Click “Server Blade Action” in “Condition” tab, and then choose “Start remote console”.
5. Click “Tools”, and then choose “Launch Virtual Media”.
6. Click “Add Image…”, and then choose ISO HANA installer media.
7. Click “Mapped” check-box of the media to connect the media via virtual CD drive.
8. On HVM screen (the serial screen of HVM), press F3 to activate a LPAR
9. On remote-KVM which is launched at No.4, select “Continue”
10. Change all ports of Hitachi Fibre channel to bios-enable state as follows:

Shell> drivers

            T  D

D          Y C I

R          P F A

V  VERSION  E G G #D #C                                  DRIVER NAME                                              IMAGE NAME

== ======== = = = == ==                      ===================================                       ===================

3C 0000000A D - - 3  -                                  Platform Console Management Driver     ConPlatform

3D 0000000A D - - 3  -                                  Platform Console Management Driver      ConPlatform

…

77 10010208 B X - 1  1                                  Hitachi PCI-X/PCIe Fibre channel Dr      ScsiBusFive

Shell> drvcfg 77

Configurable Components

  Drv[77]  Ctrl[81] Lang[eng]

  Drv[77]  Ctrl[82] Lang[eng]

Shell> drvcfg –s 77 81

hfccfg>select

  HBA FC Port List:

      Num Seg Bus Dev Func  current WWPN      original WWPN

      --- --- --- --- ----  ----------------  ----------------

      1 - 00  03 00  00  2389000087xxxx10

      2 - 00  03 00  01  2389000087xxxx20

      c - cancel

  select Number -->1

1. hfccfg.2389000087xxxx10>set

  Base Settings:

    ** Boot Function = Disabled

change? (y/[n]) -->y

          0 - Disabled -[default]

          1 - Enabled

          c - cancel

        please select -->1

    ** Connection Type = Auto Detection

      please Enter -->

    ** Data Rate = Auto Detection

      please Enter -->

    ** Spinup Delay = Disabled

      change? (y/[n]) -->

    ** Login Delay Time = 3sec

      change? (y/[n]) -->

    ** Persistent Bindings = Enabled

      change? (y/[n]) -->

    ** Force Default Parameter for adapter driver = Disabled

      change? (y/[n]) -->

    ** Select Boot Device = Enabled

      change? (y/[n]) -->

    ** << Boot Device List >> (LUN:decimal)

          1 - WWPN:50060E801033E570  LUN:0000

          2 - WWPN:0000000000000000  LUN:0000

          3 - WWPN:0000000000000000  LUN:0000

          4 - WWPN:0000000000000000  LUN:0000

          5 - WWPN:0000000000000000  LUN:0000

          6 - WWPN:0000000000000000  LUN:0000

          7 - WWPN:0000000000000000  LUN:0000

          8 - WWPN:0000000000000000  LUN:0000

change? (y/[n]) -->

  Base Settings is completed

1. hfccfg.2389000087xxxx10>save

  Save configuration data.

        y - execute

        n - cancel

  please select (y/[n]) -->y

  Now executing....

  done.

1. hfccfg.2389000087xxxx10> select

  HBA FC Port List:

      Num Seg Bus Dev Func  current WWPN      original WWPN

      --- --- --- --- ----  ----------------  ----------------

      1 - 00  03 00  00  2389000087xxxx10

      2 - 00  03 00  01  2389000087xxxx20

c - cancel

  select Number -->2

1. hfccfg.2389000087xxxx20>set

  Base Settings:

    ** Boot Function = Disabled

      change? (y/[n]) -->y

          0 - Disabled -[default]

          1 - Enabled

          c - cancel

        please select -->1

    ** Connection Type = Auto Detection

      please Enter -->

    ** Data Rate = Auto Detection

      please Enter -->

    ** Spinup Delay = Disabled

      change? (y/[n]) -->

    ** Login Delay Time = 3sec

      change? (y/[n]) -->

    ** Persistent Bindings = Enabled

      change? (y/[n]) -->

    ** Force Default Parameter for adapter driver = Disabled

      change? (y/[n]) -->

    ** Select Boot Device = Enabled

      change? (y/[n]) -->

    ** << Boot Device List >> (LUN:decimal)

          1 - WWPN:50060E801033E570  LUN:0000

          2 - WWPN:0000000000000000  LUN:0000

          3 - WWPN:0000000000000000  LUN:0000

          4 - WWPN:0000000000000000  LUN:0000

          5 - WWPN:0000000000000000  LUN:0000

          6 - WWPN:0000000000000000  LUN:0000

          7 - WWPN:0000000000000000  LUN:0000

          8 - WWPN:0000000000000000  LUN:0000

      change? (y/[n]) -->

  Base Settings is completed

1. hfccfg.2389000087xxxx20>save

  Save configuration data.

        y - execute

        n - cancel

  please select (y/[n]) -->y

  Now executing....

  done.

1. hfccfg.2389000087xxxx20>exit

Shell> reconnect –r

Shell> map –r

Shell> exit

1. Select “Boot Maintenance Manager” as follows.
2. Select “Boot Options” -> “Add Boot Option” -> “NO VOLUME LABEL, [Acpi …/USB(xxx)/…/CDROM(xxx)]]” -> <EFI> -> <BOOT> -> “bootx64.efi”.
3. Enter “Input the description” to set the name, and then select “Commit Changes and Exit”.
4. Select “Boot Options” -> “Change Boot Order”.
5. Configure “Change the order” to set No.13 device to the first priority of the boot devices (select by enter key, and change the order by +/- keys), and then select “Commit Changes and Exit”.
6. Enter Esc key to get back to No.11 screen.
7. Select “Continue” to boot from HANA installer.
8. Set parameters to install OS and HANA as follows:
9. Password: Hitachino1
10. Deployment: Scale-UP
11. HANA install: yes
12. Hostname/Domainname/IPaddress/etc…(Network): the parameters given by SAP
13. HANA System ID: H0<LPAR#> (ex. LPAR#1: H01, Baremetal: H00)
14. HANA System Number: 0<LPAR#> (ex. LPAR#1: 01, Baremetal: 00)
15. HANA Password: Hitachino1

Note: Please do not unmap HANA install media until login prompt appeared. Otherwise, HANA will not be installed.

1. Login to the OS as root.
2. Add the following kernel options by using “yast” command. (Select “System” -> “Boot Loader” -> the one that is default setting for booting the kernel -> “Optional kernel command line parameter”)

nmi_watchdog=0 mce=0 pci=noaer scsi_mod.scan=yes no_timer_check console=tty0 console=ttyS1,115200 apic=verbose

1. Add the following setting in /etc/securetty.

ttyS1

1. Add the following setting in /etc/inittab.

S1:12345:respawn:/sbin/agetty 115200 ttyS1

1. Edit the following setting in /etc/rc.d/boot.local.

Old: echo never > /sys/kernel/mm/transparent_hugepage/enabled

New: echo always > /sys/kernel/mm/transparent_hugepage/enabled

1. If the blade is in HVM mode, enter the following commands.

# /opt/hitachi/drivers/hba/hfcmgr –p hfcldd0 ir msix

# mkinitrd

1. Reboot the OS.

2.4. How to use ISO image

This session will show how to use ISO image by using Remote KVM.

1. Connect to CB500 SVP web console.
2. Login to the web console.
3. Click a blade figure in “Chassis Status”.
4. Click “Server Blade Action” in “Condition” tab, and then choose “Start remote console”.
5. Click “Tools”, and then choose “Launch Virtual Media”.
6. Click “Add Image…”, and then choose ISO media.
7. Click “Mapped” check-box of the media to connect the media via virtual CD drive.
8. Select “PCI Device Assignment” on HVM screen (the serial screen of HVM).
9. Choose PCI Device#0 and press F5 key to assign USB device to the LPAR.