ABAP_trial 2022 HDB start problems: log full still active in log partition

[WRoeckelein]

Hi all,

HDB of my #abap_trial 2022 hangs on startup, in /hana/shared/HDB/HDB02/vhcala4hci/trace/daemon__0000783__children.trc I found these messages

[935]{-1}[-1/-1] 2024-06-12 09:57:58.194184 i EventHandler EventManagerImpl.cpp(00962) : Could not set event 'LogFullEvent[volumeid=1,eventid=1,part=/hana/log/HDB/mnt00001/hdb00001/]' to state 'handled'
[962]{-1}[-1/-1] 2024-06-12 09:57:58.194220 i Logger LogSegment.cpp(03175) : log partition disk usage 100% >= log_disk_usage_limit 95%, do not create new log segment
[962]{-1}[-1/-1] 2024-06-12 09:57:58.194229 i Logger LogSegment.cpp(03294) : Retry of log segment creation failed, log full still active in log partition /hana/log/HDB/mnt00001/hdb00001/, awaiting free segment or additional disk space

df says

Filesystem 1K-blocks Used Available Use% Mounted on
overlay 263112772 261397492 0 100% /
tmpfs 65536 0 65536 0% /dev
tmpfs 10240832 0 10240832 0% /sys/fs/cgroup
shm 65536 0 65536 0% /dev/shm
/dev/sdd 263112772 261397492 0 100% /etc/hosts
tmpfs 10240832 0 10240832 0% /proc/acpi
tmpfs 10240832 0 10240832 0% /sys/firmware

Some ideas on how to overcome this? @julieplummer20 any hint?

Thanks,

  Wolfgang

[WRoeckelein]

to be deleted

[julieplummer20 · Accepted solution]

Hi Wolfgang,

I talked to a colleague.

He suggested that you "may have performed a large number of DB transactions, and that the Log Disk actually full.

How big is the Log Disk?
Have you tried ALTER SYSTEM RECLAIM LOG on HANA Console?"

I'll be honest; this is not my specialism. Have you tried this?
Best wishes,

Julie.

[WRoeckelein · Accepted solution]

Hi Julie and all at ABAP_Trial,

this was a bit more complicated than expected.

This is a specific problem of Docker on Windows running on WSL2.

All docker containers there use one WSL2 instance behind the curtain.
All overlay filesystems use thus one virtual disk located at %LocalAppData%\Docker\wsl\data\ext4.vhdx .
This disk is hard limited to approx 250GB. When this is reached, df -h inside a docker container will report 100% used (see screenshot above) and thus 0% available.

Increasing this is a multi step process. This all is documented, however at various scattered places but not really in one place.

The following steps are needed (there might be quicker ways to that, proven shortcuts are welcome):

• You need a WSL2 distribution besides the two docker ones, eg the default Ubuntu
• Stop all docker virtual machines. The ABAP trial container should be stopped with "docker stop -t 7200 a4h", the rest can be stopped with "wsl.exe --shutdown"
• First expand the virtual drive, either with
  • diskpart, use the following steps
    • run diskpart and inside diskpart, run 
      • select vdisk file="%LocalAppData%\Docker\wsl\data\ext4.vhdx"
      • expand vdisk maximum=<sizeInMegaBytes>
      • exit
  • some other means like 
    • resize-vhd -Path %LocalAppData%\Docker\wsl\data\ext4.vhdx -SizeBytes 300GB
• Then increase the partition inside the virtual drive with the help of the Ubuntu distro
  • Normally (eg wsl updated from the store) you should be able to mount this directly with
    • wsl --mount --vhd %LocalAppData%\Docker\wsl\data\ext4.vhdx
  • however this did not work for me, so I had to first mount this in Windows with powershell
    • Write-Output "\\.\PhysicalDrive$((Mount-VHD -Path <pathToVHD> -PassThru | Get-Disk).Number)"
  • then use the output with
    • wsl --mount \\.\PhysicalDrive2
  • in both cases start default distro
    • wsl
  • and inside the default Ubuntu first find the partition with
    • mount | grep ext4
  • the increase and exit with
    • sudo resize2fs /dev/sdX <sizeInMegabytes>M
    • exit
  • then unmount depending on how you mounted either only with
    • wsl --unmount --vhd %LocalAppData%\Docker\wsl\data\ext4.vhdx
  • or with 
    • wsl.exe --unmount \\.\PhysicalDrive2
  • together with unmount from windows with
    • Dismount-VHD -DiskNumber 2

[julieplummer20 · Accepted solution]

Hi Wolfgang, Many thanks for getting back in touch - this is really helpful information; I will discuss this with the team and incorporate this in the FAQs:

ABAP Cloud Developer Trial on Docker: FAQs 

Best wishes,

Julie Plummer

ABAP PM.

[julieplummer20]

Hi Wolfgang,

Perhaps I'm missing something, but default log segment count = 10240. Have you enlarged this, ie: 

1. Stop the Hana DB.
2. Increase the limit (e.g. <count> = 10240) :
   
   • hdblogdiag directory --enlarge <count> /hana/log/<sid>/mnt<id>/hdb<id>
   • The parameter <count> specifies the number of new entries which can be appended to the log segment directory.

3. After extending the directory the database can be restarted and additional <count> log segments will be allocated.

Just let me know if I have missed something.

Best wishes,

JUlie.

[WRoeckelein]

Hi @julieplummer20 ,

thanks for the quick answer. However despite several extensions already the error won't go away

[833]{-1}[-1/-1] 2024-06-12 13:55:40.178800 i Logger LoggerImpl.cpp(01435) : Replayed 323456B (0MB) of log in 0.045049 seconds; 6.84747MB/s; max known TID=67656961
[833]{-1}[-1/-1] 2024-06-12 13:55:40.178875 i Logger LoggerImpl.cpp(01010) : Starting logger with 8 log buffers per partition, 1024KB each, segment size 64MB, log mode normal, currentWritePos=0xfd479f80
[966]{-1}[-1/-1] 2024-06-12 13:55:40.178996 i Logger LogSegment.cpp(03175) : log partition disk usage 100% >= log_disk_usage_limit 95%, do not create new log segment
[966]{-1}[-1/-1] 2024-06-12 13:55:40.179001 w Logger LogSegment.cpp(03250) : Log full - log_disk_usage_limit 95% reached in log partition /hana/log/HDB/mnt00001/hdb00001/, current disk usage is 100%, awaiting free segment or additional free disk space
[966]{-1}[-1/-1] 2024-06-12 13:55:40.179008 i EventHandler EventManagerImpl.cpp(00680) : New event reported: 'LogFullEvent[volumeid=1,eventid=1,part=/hana/log/HDB/mnt00001/hdb00001/]'

ls -l /hana/log/HDB/mnt00001/hdb00001/ gives now

total 298388
0 -rw-r--r-- 1 hdbadm sapsys 0 Oct 16 2017 __DO_NOT_TOUCH_FILES_IN_THIS_DIRECTORY__
4 -rw-r--r-- 1 hdbadm sapsys 36 Jun 11 16:21 landscape.id
65536 -rw-r--r-- 1 hdbadm sapsys 67108864 Mar 3 2023 logsegment_000_00000016.dat
65536 -rw-r--r-- 1 hdbadm sapsys 67108864 Jun 11 19:51 logsegment_000_00000017.dat
65536 -rw-r--r-- 1 hdbadm sapsys 67108864 Jun 11 19:54 logsegment_000_00000018.dat
29076 -rw-r--r-- 1 hdbadm sapsys 29773824 Jun 12 13:53 logsegment_000_directory.dat
1328 -rw-r--r-- 1 hdbadm sapsys 1359872 Jun 11 19:57 logsegment_000_directory.dat_3_1718196457310367
7932 -rw-r--r-- 1 hdbadm sapsys 8122368 Jun 12 12:47 logsegment_000_directory.dat_3_1718199375099519
14540 -rw-r--r-- 1 hdbadm sapsys 14888960 Jun 12 13:36 logsegment_000_directory.dat_3_1718199825061521
21148 -rw-r--r-- 1 hdbadm sapsys 21655552 Jun 12 13:43 logsegment_000_directory.dat_3_1718200121402079
27752 -rw-r--r-- 1 hdbadm sapsys 28418048 Jun 12 13:48 logsegment_000_directory.dat_3_1718200424066429

Regards,

  Wolfgang