FILESYSTEMIO_OPTIONS under Oracle 11.2.0.2 on AIX 6.1

[reinhard_koch]

Hello Friends,

first I have updated a sandbox from oracle 11.2.0.4 to 11.2.0.1 and everything works fine.

But now I have installed patchset 11.2.0.2 and with unchanged mountoptions (cio) for sapdata-dirs and FILESYSTEMIO_OPTIONS=SETALL (unchanged too) I can't access the datafiles on os level for online backup (without RMAN) while the database is running. Error "A system call received a parameter that is not valid" is shown when using cp or dd. If I shutdown the database I can copy the files (very slow, due to the cio-mount option)

Only when I set the FILESYSTEMIO_OPTIONS to ASYNCH it is possible to do access the files when the system is online.

Has anybody of you a hint to resolve this issue?

Thanx in advance ...

Reinhard Koch

[reinhard_koch · Accepted solution]

Hello,

now it comes what I expected. The new installation SAP Netweaver 7 EHP 2 with Oracle 11.2.0.2 on AIX 6100-06-03-1048 shows the same problem. The datafiles can't be accessed when the database is running and filesystemio_options is set to "SETALL". I have no idea what I can further do.

@Volker: Have you already logged a call at oracle?

Greetings

Reinhard

.

[Former Member · Accepted solution]

I am having this same issue after upgrading 10.2.0.2 and 11.2.0.1 databases. I currently have notes open with SAP and Oracle. I will update this thread as soon as I can.

[Former Member · Accepted solution]

SAP and Oracle have escalated my tickets to their advanced support teams.

[volker_borowski2 · Accepted solution]

Hi,

I have a call open as well. But no solution yet.

May be the new bundle patch which should be available within the next week helps.

Volker

[Former Member · Accepted solution]

All,

I happened to come across SAP Note 948294.

AIX 6.1 with Oracle >= 11.2.0.2:

AIX 6.1 introduced a new open flag O_CIOR which is same as O_CIO, but this

allows subsequent open calls without CIO. The advantage of this enhancement

is that other applications like cp, dd, cpio, dbv can access database files

in read only mode without having to open them with CIO.

Starting with Oracle 11.2.0.2 when AIX 6.1 is detected, Oracle will

use O_CIOR option to open a file on JFS2.

Therefore you should no longer mount the filesystems with mount option -o

cio

Please remove the mount option "-o cio" if you are running Oracle 11.2.0.2

on AIX 6.1 to avoid the following error messages:

cp: A system call received a parameter that is not valid.

0653-902 Cannot open the specified file for reading.

DBV-00100: Specified FILE not accessible

errno(22) A system call received a parameter that is not valid.

I am getting confirmation from Oracle, SAP, and IBM on whether or not we will suffer a performance hit by unsetting the CIO mount option.

Paul

[volker_borowski2 · Accepted solution]

Hi,

my call was directed to the same note. Will try to crosscheck today.

Volker

[thomas_franke2 · Accepted solution]

Hello,

I also opende a sap ticket for this. Also for the performance issue. The answer was (sorry it was in german).

nein, da wir weiterhin mit den Oracle Prozessen per CIO zugreifen,

allerdings mit dem seit AIX 6.1 möglich O_CIOR.

Bisher (vor der Kombination AIX 6.1 + Oracle 11.2.0.2) wurde der Zugriffüber O_CIO gemacht. Damit mußte das Filesystem entsprechend gemounted

werden (mit Option -o cio), um sicherzustellen, da0 auch die anderen

Programme wie cpio, dd, dbv, tar etc. ohne den Filesystemcache zugreifen

Dies ist nun nicht mehr nötig.

Wenn Oracle feststellt, daß AIX 6.1 als Betriebssystem genutzt wird,

werden die Datafiles automatisch mit O_CIOR geöffnet.

D.h. die Oracle Prozesse lassen dann einen lesenden Zugriff auf die

Dateien von anderen Prozessen zu (das bedeutet das R am Ende des O_CIOR)OHNE dass das Filesystem mit der CIO Mountoption gemounted sein muß.

Die Performance wird also nicht schlechter, da die Oracle Prozesse

weiterhin mit CIO arbeiten können.

Mit freundlichen Grüßen

Tanja Albrecht

Development Support

Support Solution Center SAP, Support & Services, Oracle Rot / Germany

[reinhard_koch]

Hi Paul, Hi Volker,

Thank you both to take part at this thread!

SAP note 948294 was updated yesterday. Now I have removed the cio-mountoption from my jfs2 filesystems and everything works fine.

I am wondering why this information cant be found in any oracle known issues document for 11.2.0.2.

Greetings

Reinhard

[Former Member]

Per Oracle:

=== ODM Action Plan ===

Thanks for the information.

Bug 9310972: ENHANCEMENT : INTRODUCING O_CIOR FLAG WHEN OPENING DATAFILES IN AIX 6.1

Bug:9310972 is an enhancement to call O_CIOR instead of O_CIO on 11.2.0.2.

In this case, JFS2 filesystem should be mounted without option -o cio.

Since this is a valid enhancement, it should not cause any performance issues.

=== ODM Action Plan ===

Confirmed with developer of this enhancement.

1. CIO option should not be enabled when mounting file system on Oracle 11.2.0.2 on AIX 6.1.

2. For Oracle O_CIO and O_CIOR should behave exactly the same in terms of performance. Disabling cio on file system level will not cause any performance issues since calling O_CIOR acts the same as calling O_CIO.

Although....I cannot find anything in Oracle's Bug database for this. Must be unreleased....which makes tons of sense.

[volker_borowski2]

Hi,

did an online db-verify successfully after removing cio option.

So operation is back to normal.

Could not check any performance aspects up to now.

But I think it will be ok this way.

Thanks all for sharing as well. Good feeling not to be alone on this

Volker

[Former Member]

Hi Reinhard,

Very interesting! I don't have any answers but was just wondering what a truss of the cp or dd command shows?

It might give a clue as to what system call is failing and why...

[reinhard_koch]

Hello Somckit,

here is an output from a try with the cp command:

truss cp cntrlGKS.dbf cntrlGKS.dbf_test
execve("/usr/bin/cp", 0x2FF22878, 0x20012818)    argc: 3
sbrk(0x00000000)                                = 0x30002458
vmgetinfo(0x2FF219A0, 7, 16)                    = 0
sbrk(0x00000000)                                = 0x30002458
sbrk(0x00000008)                                = 0x30002458
__libc_sbrk(0x00000000)                         = 0x30002460
getuidx(4)                                      = 0
getuidx(2)                                      = 0
getuidx(1)                                      = 0
getgidx(4)                                      = 0
getgidx(2)                                      = 0
getgidx(1)                                      = 0
__loadx(0x01480080, 0x2FF21360, 0x00000A50, 0x2FF21ED0, 0x00000000) = 0xD04E5128
__loadx(0x01480180, 0x2FF21360, 0x00000A50, 0xF042846C, 0xF042839C) = 0xF04EAEA0
__loadx(0x07080000, 0xF042843C, 0xFFFFFFFF, 0xF04EAEA0, 0x00000000) = 0xF04EBDF4
__loadx(0x07080000, 0xF042837C, 0xFFFFFFFF, 0xF04EAEA0, 0x00000000) = 0xF04EBE00
__loadx(0x07080000, 0xF042844C, 0xFFFFFFFF, 0xF04EAEA0, 0x00000000) = 0xF04EBE30
__loadx(0x07080000, 0xF042838C, 0xFFFFFFFF, 0xF04EAEA0, 0x00000000) = 0xF04EBE3C
__loadx(0x07080000, 0xF042840C, 0xFFFFFFFF, 0xF04EAEA0, 0x00000000) = 0xF04EBE0C
__loadx(0x07080000, 0xF04283AC, 0xFFFFFFFF, 0xF04EAEA0, 0x00000000) = 0xF04EBE24
__loadx(0x07080000, 0xF042841C, 0xFFFFFFFF, 0xF04EAEA0, 0x00000000) = 0xF04EBE48
__loadx(0x07080000, 0xF042842C, 0xFFFFFFFF, 0xF04EAEA0, 0x00000000) = 0xF04EBE78
__loadx(0x07080000, 0xF04283BC, 0xFFFFFFFF, 0xF04EAEA0, 0x00000000) = 0xF04EBE60
__loadx(0x07080000, 0xF04283CC, 0xFFFFFFFF, 0xF04EAEA0, 0x00000000) = 0xF04EBED8
getuidx(4)                                      = 0
getuidx(2)                                      = 0
getuidx(1)                                      = 0
getgidx(4)                                      = 0
getgidx(2)                                      = 0
getgidx(1)                                      = 0
__loadx(0x01480080, 0x2FF21360, 0x00000A50, 0x2FF21ED0, 0x00000000) = 0xD04E5128
getuidx(4)                                      = 0
getuidx(2)                                      = 0
getuidx(1)                                      = 0
getgidx(4)                                      = 0
getgidx(2)                                      = 0
getgidx(1)                                      = 0
__loadx(0x01480080, 0x2FF21360, 0x00000A50, 0x2FF21ED0, 0x00000000) = 0xD04E5128
getuidx(4)                                      = 0
getuidx(2)                                      = 0
getuidx(1)                                      = 0
getgidx(4)                                      = 0
getgidx(2)                                      = 0
getgidx(1)                                      = 0
__loadx(0x01480080, 0x2FF21360, 0x00000A50, 0x2FF21ED0, 0x00000000) = 0xD04E5128
getuidx(4)                                      = 0
getuidx(2)                                      = 0
getuidx(1)                                      = 0
getgidx(4)                                      = 0
getgidx(2)                                      = 0
getgidx(1)                                      = 0
__loadx(0x01480080, 0x2FF21360, 0x00000A50, 0x2FF21ED0, 0x00000000) = 0xD04E5128
getuidx(4)                                      = 0
getuidx(2)                                      = 0
getuidx(1)                                      = 0
getgidx(4)                                      = 0
getgidx(2)                                      = 0
getgidx(1)                                      = 0
__loadx(0x01480080, 0x2FF21360, 0x00000A50, 0x2FF21ED0, 0x00000000) = 0xD04E5128
access("/usr/lib/nls/msg/en_US/cp.cat", 0)      = 0
_getpid()                                       = 401482
getuidx(2)                                      = 0
umask(0)                                        = 18
umask(18)                                       = 0
statx("cntrlGKS.dbf_test", 0x2FF22700, 176, 020) Err#2  ENOENT
statx("cntrlGKS.dbf", 0x2FF22548, 176, 020)     = 0
statx("cntrlGKS.dbf_test", 0x2FF225F8, 176, 020) Err#2  ENOENT
kopen("cntrlGKS.dbf", O_RDONLY|O_LARGEFILE)     Err#22 EINVAL
access("/usr/lib/nls/msg/en_US/libc.cat", 0)    = 0
_getpid()                                       = 401482
kopen("/usr/lib/nls/msg/en_US/libc.cat", O_RDONLY) = 3
kioctl(3, 22528, 0x00000000, 0x00000000)        Err#25 ENOTTY
kfcntl(3, F_SETFD, 0x00000001)                  = 0
kioctl(3, 22528, 0x00000000, 0x00000000)        Err#25 ENOTTY
kread(3, "0001 ù007007 I S O 8".., 4096)    = 4096
lseek(3, 0, 1)                                  = 4096
lseek(3, 0, 1)                                  = 4096
lseek(3, 0, 1)                                  = 4096
_getpid()                                       = 401482
lseek(3, 0, 1)                                  = 4096
lseek(3, 4365, 0)                               = 4365
kread(3, " A   s y s t e m   c a l".., 4096)    = 4096
close(3)                                        = 0
cpkwrite(2, " c p", 2)                          = 2
: kwrite(2, " :  ", 2)                          = 2
cntrlGKS.dbfkwrite(2, " c n t r l G K S . d b f", 12)   = 12
: kwrite(2, " :  ", 2)                          = 2
A system call received a parameter that is not valid.kwrite(2, " A   s y s t e m   c a l".., 53)        = 53

kwrite(2, "
", 1)                              = 1
access("cntrlGKS.dbf_test", 0)                  Err#2  ENOENT
kfcntl(1, F_GETFL, 0x2FF22FF8)                  = 67110914
kfcntl(2, F_GETFL, 0x00000000)                  = 67110914
_exit(1)

Can you see anything which explains the behaviour?

Greetings

Reinhard

[Former Member]

Hi Reinhard,

The issue appears to be in this system call:

kopen("cntrlGKS.dbf", O_RDONLY|O_LARGEFILE) Err#22 EINVAL

This is the kopen system call. I don't have access to an AIX 6.1 system at the moment, but you can man kopen to see what typical EINVAL errors might occur.

It looks to me like it's trying to open the cntrlGKS.dbf file and receives the error.

What does the truss output look like when it works?

Maybe search the IBM bug site?

Hope this helps.

[stefan_koehler]

Hello Reinhard,

if you try "cp" while running an oracle database on a JFS2 filesystem with filesystemio_options SETALL - this behaviour is normal (explained in detail in metalinknote #832536.1).

The error you receive is the following:

#define EINVAL  22      /* Invalid argument                     */

The question would be why kopen is used and not open and why the kopen call has the wrong arguments as you can see.

Regards

Stefan

[volker_borowski2]

Hi Reinhard,

I now have the issue at an additional customer as well.

Same error when "dd"ing a datafile.

When you say "unchanged" from 11.2.0.1. did you install OS fixes prior to your upgrade to 11.2.0.2.

If the only change was the oracle patchset, it looks oracle related.

If you did apply osfixes, it might be aix related as well.

Since 11.2.0.2 took so long to get approvement and even metalink has nothing significant on this as well,

I currently assume an AIX cause for this due to newest OS fixes.

BTW we are on 6100-06-03-1048.

We still need to do some additional tests, but I am pretty confident, that we will soon (today) log a call for this.

Volker

[volker_borowski2]

>

> Since 11.2.0.2 took so long to get approvement and even metalink has nothing significant on this as well,
> I currently assume an AIX cause for this due to newest OS fixes.
> BTW we are on 6100-06-03-1048.
> Volker

Humm,

have found a 10.2.0.4 system on the same OS level as our 11.2.0.2 system,

where online access to datafiles work.

So it is may be not AIX but Oracle related ?!??

Anyone else working on this issue?

Volker

[reinhard_koch]

Hello Volker,

es you are right. I have only applied the oracle patchset. The AIX os was completely untouched (at 6100-02-02-0849). So this issue must be related to oracle.

At the moment I am trying a new Netweaver 7.0 EHP2 with a completely new oracle 11.2.0.2 installation on a AIX 6100-06-03-1048. When the database is installed and running the first test I will do is the "dd" and I will report you how it works.

Reinhard

[volker_borowski2]

... a completely new oracle 11.2.0.2 installation on a AIX 6100-06-03-1048. 

This is how the system here was created.

No patching as all, this is why I was so interested, if it worked on the same os as your 11.2.0.1

Volker

[volker_borowski2]

You are not alone,

I have a customer with the same issue.

Reading datafile with dd fails, even backint (TDPR3) can not read the file.

No solution yet, beside switching to rman or offline backups.

Volker