Java Stack Does not start

[Former Member]

Hello Guys ,

I have got a serious issue with my Java stack. I am unable to start Java stack sucessfully. when i check trace files, I find that it was started but eventually stopped due to some error.

Here I am posting bootstrap trace file to investigate issue and to help me out.

-

---

trc file: "/usr/sap/N4S/DVEBMGS01/work/dev_bootstrap_ID14704950", trc level: 1, release: "700"

-

---

node name : bootstrap_ID14704950

pid : 12621

system name : N4S

system nr. : 01

started at : Tue Jun 15 13:56:20 2010

arguments :

arg[00] : /usr/sap/N4S/DVEBMGS01/exe/jlaunch

arg[01] : pf=/usr/sap/N4S/SYS/profile/N4S_DVEBMGS01_n4shost

arg[02] : -DSAPINFO=N4S_01_bootstrap

arg[03] : pf=/usr/sap/N4S/SYS/profile/N4S_DVEBMGS01_n4shost

arg[04] : -DSAPSTART=1

arg[05] : -DCONNECT_PORT=32910

arg[06] : -DSAPSYSTEM=01

arg[07] : -DSAPSYSTEMNAME=N4S

arg[08] : -DSAPMYNAME=n4shost_N4S_01

arg[09] : -DSAPPROFILE=/usr/sap/N4S/SYS/profile/N4S_DVEBMGS01_n4shost

arg[10] : -DFRFC_FALLBACK=ON

arg[11] : -DFRFC_FALLBACK_HOST=localhost

[Thr 182897318208] Tue Jun 15 13:56:20 2010

[Thr 182897318208] *** WARNING => INFO: Unknown property [instance.box.number=N4SDVEBMGS01n4shost] [jstartxx_mt. 841]

[Thr 182897318208] *** WARNING => INFO: Unknown property [instance.en.host=n4shost] [jstartxx_mt. 841]

[Thr 182897318208] *** WARNING => INFO: Unknown property [instance.en.port=3200] [jstartxx_mt. 841]

[Thr 182897318208] *** WARNING => INFO: Unknown property [instance.system.id=1] [jstartxx_mt. 841]

**********************************************************************

JStartupReadInstanceProperties: read instance properties [/usr/sap/N4S/DVEBMGS01/j2ee/cluster/instance.properties]

-> ms host : n4shost

-> ms port : 3900

-> OS libs : /usr/sap/N4S/DVEBMGS01/j2ee/os_libs

-> Admin URL :

-> run mode : NORMAL

-> run action : NONE

-> enabled : yes

**********************************************************************

**********************************************************************

Used property files

-> files [00] : /usr/sap/N4S/DVEBMGS01/j2ee/cluster/instance.properties

**********************************************************************

**********************************************************************

Instance properties

-> ms host : n4shost

-> ms port : 3900

-> os libs : /usr/sap/N4S/DVEBMGS01/j2ee/os_libs

-> admin URL :

-> run mode : NORMAL

-> run action : NONE

-> enabled : yes

**********************************************************************

**********************************************************************

Bootstrap nodes

-> [00] bootstrap : /usr/sap/N4S/DVEBMGS01/j2ee/cluster/instance.properties

-> [01] bootstrap_ID14704900 : /usr/sap/N4S/DVEBMGS01/j2ee/cluster/instance.properties

-> [02] bootstrap_ID14704950 : /usr/sap/N4S/DVEBMGS01/j2ee/cluster/instance.properties

**********************************************************************

**********************************************************************

Worker nodes

-> [00] ID14704900 : /usr/sap/N4S/DVEBMGS01/j2ee/cluster/instance.properties

-> [01] ID14704950 : /usr/sap/N4S/DVEBMGS01/j2ee/cluster/instance.properties

**********************************************************************

[Thr 182897318208] JLaunchRequestQueueInit: create named pipe for ipc

[Thr 182897318208] JLaunchRequestQueueInit: create pipe listener thread

[Thr 1074792800] JLaunchRequestFunc: Thread 1074792800 started as listener thread for np messages.

[Thr 1085282656] WaitSyncSemThread: Thread 1085282656 started as semaphore monitor thread.

[Thr 182897318208] Tue Jun 15 13:56:21 2010

[Thr 182897318208] SigISetDefaultAction : default handling for signal 17

[Thr 182897318208] NiInit3: NI already initialized; param 'maxHandles' ignored (1;202)

[Thr 182897318208] CPIC (version=700.2005.12.02)

[Thr 182897318208] [Node: server0 bootstrap] java home is set by profile parameter

Java Home: /usr/lib/java

[Thr 182897318208] JStartupICheckFrameworkPackage: can't find framework package /usr/sap/N4S/DVEBMGS01/exe/jvmx.jar

**********************************************************************

JStartupIReadSection: read node properties [bootstrap_ID14704950]

-> node name : server0 bootstrap

-> node type : bootstrap

-> node execute : yes

-> jlaunch parameters :

-> java path : /usr/lib/java

-> java parameters : -Djco.jarm=1

-> java vm version : J2RE 1.4.2 IBM J9 2.2 Linux amd64-64 j9xa64142ifx-20051125 (JIT enabled)

-> java vm vendor : IBM J9SE VM (IBM Corporation)

-> java vm type : <default>

-> java vm cpu : amd64

-> heap size : 203M

-> root path : /usr/sap/N4S/DVEBMGS01/j2ee/cluster

-> class path : ./bootstrap/launcher.jar

-> OS libs path : /usr/sap/N4S/DVEBMGS01/j2ee/os_libs

-> main class : com.sap.engine.offline.OfflineToolStart

-> framework class : com.sap.bc.proj.jstartup.JStartupFramework

-> registr. class : com.sap.bc.proj.jstartup.JStartupNatives

-> framework path : /usr/sap/N4S/DVEBMGS01/exe/jstartup.jar:/usr/sap/N4S/DVEBMGS01/exe/jvmx.jar

-> parameters : com.sap.engine.bootstrap.Bootstrap ./bootstrap ID014704950

-> debuggable : yes

-> debug mode : no

-> debug port : 60000

-> shutdown timeout : 120000

**********************************************************************

[Thr 1086335328] JLaunchIStartFunc: Thread 1086335328 started as Java VM thread.

**********************************************************************

JHVM_LoadJavaVM: VM Arguments of node [server0 bootstrap]

-> stack : 1048576 Bytes

-> arg[ 0]: exit

-> arg[ 1]: abort

-> arg[ 2]: vfprintf

-> arg[ 3]: -Djco.jarm=1

-> arg[ 4]: -Dsys.global.dir=/usr/sap/N4S/SYS/global

-> arg[ 5]: -Dapplication.home=/usr/sap/N4S/DVEBMGS01/exe

-> arg[ 6]: -Djava.class.path=/usr/sap/N4S/DVEBMGS01/exe/jstartup.jar:/usr/sap/N4S/DVEBMGS01/exe/jvmx.jar:./bootstrap/launcher.jar

-> arg[ 7]: -Djava.library.path=/usr/lib/java/jre/bin:/usr/lib/java/bin/../jre/bin:/usr/lib/java/jre/bin/j9vm:/usr/lib/java/jre/bin:/usr/sap/N4S/DVEBMGS01/exe:/usr/sap/N4S/DVEBMGS01/exe:/usr/sap/N4S/DVEBMGS01/exe:/tmp/sapinst_exe.11133.1149865079:/usr/sap/N4S/SYS/exe/run:/sapdb/programs/lib:/usr/lib:/usr/sap/N4S/DVEBMGS01/j2ee/os_libs:/usr/sap/N4S/DVEBMGS01/exe:/usr/sap/N4S/DVEBMGS01/exe:/usr/sap/N4S/DVEBMGS01/exe:/usr/sap/N4S/SYS/exe/run:/sapdb/programs/lib

-> arg[ 8]: -Dmemory.manager=203M

-> arg[ 9]: -Xmx203M

-> arg[ 10]: -DLoadBalanceRestricted=no

-> arg[ 11]: -Djstartup.mode=BOOTSTRAP

-> arg[ 12]: -Djstartup.ownProcessId=12621

-> arg[ 13]: -Djstartup.ownHardwareId=A0340344147

-> arg[ 14]: -Djstartup.whoami=bootstrap

-> arg[ 15]: -Djstartup.debuggable=yes

-> arg[ 16]: -DSAPINFO=N4S_01_bootstrap

-> arg[ 17]: -DSAPSTART=1

-> arg[ 18]: -DCONNECT_PORT=32910

-> arg[ 19]: -DSAPSYSTEM=01

-> arg[ 20]: -DSAPSYSTEMNAME=N4S

-> arg[ 21]: -DSAPMYNAME=n4shost_N4S_01

-> arg[ 22]: -DSAPPROFILE=/usr/sap/N4S/SYS/profile/N4S_DVEBMGS01_n4shost

-> arg[ 23]: -DFRFC_FALLBACK=ON

-> arg[ 24]: -DFRFC_FALLBACK_HOST=localhost

-> arg[ 25]: -DSAPSTARTUP=1

-> arg[ 26]: -DSAPSYSTEM=01

-> arg[ 27]: -DSAPSYSTEMNAME=N4S

-> arg[ 28]: -DSAPMYNAME=n4shost_N4S_01

-> arg[ 29]: -DSAPDBHOST=n4shost

-> arg[ 30]: -Dj2ee.dbhost=n4shost

**********************************************************************

[Thr 1086335328] *** ERROR => JHVM_LoadJavaVM: Cannot create Java VM (rc=-4) [jhvmxx_mt.c 542]

[Thr 1086335328] *** ERROR => Cannot load Java VM (<default>) (rc=-1) [jlnchxxi_mt. 739]

[Thr 1086335328] SigISetIgnoreAction : SIG_IGN for signal 17

[Thr 1086335328] JLaunchCloseProgram: good bye (exitcode = -1)

[Former Member]

Here is dev_jcontrol

**********************************************************************

JStartupReadInstanceProperties: read instance properties [/usr/sap/N4S/DVEBMGS01/j2ee/cluster/instance.properties:/usr/sap/N4S/DVEBMGS01/SDM/program/config/sdm_jstartup.properties]

-> ms host : n4shost

-> ms port : 3900

-> OS libs : /usr/sap/N4S/DVEBMGS01/j2ee/os_libs

-> Admin URL :

-> run mode : NORMAL

-> run action : NONE

-> enabled : yes

**********************************************************************

[Thr 182897318208] JControlExecuteBootstrap: enumerate the nodes after global bootstrap

**********************************************************************

Used property files

-> files [00] : /usr/sap/N4S/DVEBMGS01/j2ee/cluster/instance.properties

-> files [01] : /usr/sap/N4S/DVEBMGS01/SDM/program/config/sdm_jstartup.properties

**********************************************************************

**********************************************************************

Instance properties

-> ms host : n4shost

-> ms port : 3900

-> os libs : /usr/sap/N4S/DVEBMGS01/j2ee/os_libs

-> admin URL :

-> run mode : NORMAL

-> run action : NONE

-> enabled : yes

**********************************************************************

**********************************************************************

Bootstrap nodes

-> [00] bootstrap : /usr/sap/N4S/DVEBMGS01/j2ee/cluster/instance.properties

-> [01] bootstrap_ID14704900 : /usr/sap/N4S/DVEBMGS01/j2ee/cluster/instance.properties

-> [02] bootstrap_ID14704950 : /usr/sap/N4S/DVEBMGS01/j2ee/cluster/instance.properties

**********************************************************************

**********************************************************************

Worker nodes

-> [00] ID14704900 : /usr/sap/N4S/DVEBMGS01/j2ee/cluster/instance.properties

-> [01] ID14704950 : /usr/sap/N4S/DVEBMGS01/j2ee/cluster/instance.properties

-> [02] sdm : /usr/sap/N4S/DVEBMGS01/SDM/program/config/sdm_jstartup.properties

**********************************************************************

[Thr 182897318208] JControlExecuteBootstrap: execute bootstrap process [bootstrap_ID14704900]

[Thr 182897318208] [Node: dispatcher bootstrap] java home is set by profile parameter

Java Home: /usr/lib/java

[Thr 182897318208] JStartupICheckFrameworkPackage: can't find framework package /usr/sap/N4S/SYS/exe/run/jvmx.jar

**********************************************************************

JStartupIReadSection: read node properties [bootstrap_ID14704900]

-> node name : dispatcher bootstrap

-> node type : bootstrap

-> node execute : yes

-> jlaunch parameters :

-> java path : /usr/lib/java

-> java parameters : -Djco.jarm=1

-> java vm version : J2RE 1.4.2 IBM J9 2.2 Linux amd64-64 j9xa64142ifx-20051125 (JIT enabled)

-> java vm vendor : IBM J9SE VM (IBM Corporation)

-> java vm type : <default>

-> java vm cpu : amd64

-> heap size : 203M

-> root path : /usr/sap/N4S/DVEBMGS01/j2ee/cluster

-> class path : ./bootstrap/launcher.jar

-> OS libs path : /usr/sap/N4S/DVEBMGS01/j2ee/os_libs

-> main class : com.sap.engine.offline.OfflineToolStart

-> framework class : com.sap.bc.proj.jstartup.JStartupFramework

-> registr. class : com.sap.bc.proj.jstartup.JStartupNatives

-> framework path : /usr/sap/N4S/SYS/exe/run/jstartup.jar:/usr/sap/N4S/SYS/exe/run/jvmx.jar

-> parameters : com.sap.engine.bootstrap.Bootstrap ./bootstrap ID014704900

-> debuggable : yes

-> debug mode : no

-> debug port : 60000

-> shutdown timeout : 120000

**********************************************************************

**********************************************************************

JControlStartJLaunch: program = /usr/sap/N4S/DVEBMGS01/exe/jlaunch

-> arg[00] = /usr/sap/N4S/DVEBMGS01/exe/jlaunch

-> arg[01] = pf=/usr/sap/N4S/SYS/profile/N4S_DVEBMGS01_n4shost

-> arg[02] = -DSAPINFO=N4S_01_bootstrap

-> arg[03] = -nodeId=-1

-> arg[04] = -file=/usr/sap/N4S/DVEBMGS01/j2ee/cluster/instance.properties

-> arg[05] = -syncSem=14778432

-> arg[06] = -nodeName=bootstrap_ID14704900

-> arg[07] = -jvmOutFile=/usr/sap/N4S/DVEBMGS01/work/jvm_bootstrap_ID14704900.out

-> arg[08] = -stdOutFile=/usr/sap/N4S/DVEBMGS01/work/std_bootstrap_ID14704900.out

-> arg[09] = -locOutFile=/usr/sap/N4S/DVEBMGS01/work/dev_bootstrap_ID14704900

-> arg[10] = -mode=BOOTSTRAP

-> arg[11] = pf=/usr/sap/N4S/SYS/profile/N4S_DVEBMGS01_n4shost

-> arg[12] = -DSAPSTART=1

-> arg[13] = -DCONNECT_PORT=33065

-> arg[14] = -DSAPSYSTEM=01

-> arg[15] = -DSAPSYSTEMNAME=N4S

-> arg[16] = -DSAPMYNAME=n4shost_N4S_01

-> arg[17] = -DSAPPROFILE=/usr/sap/N4S/SYS/profile/N4S_DVEBMGS01_n4shost

-> arg[18] = -DFRFC_FALLBACK=ON

-> arg[19] = -DFRFC_FALLBACK_HOST=localhost

-> lib path = LD_LIBRARY_PATH=/usr/lib/java/jre/bin:/usr/lib/java/bin/../jre/bin:/usr/lib/java/jre/bin/j9vm:/usr/lib/java/jre/bin:/usr/sap/N4S/DVEBMGS01/exe:/usr/sap/N4S/DVEBMGS01/exe:/usr/sap/N4S/DVEBMGS01/exe:/tmp/sapinst_exe.11133.1149865079:/usr/sap/N4S/SYS/exe/run:/sapdb/programs/lib:/usr/lib:/usr/sap/N4S/DVEBMGS01/j2ee/os_libs:/usr/sap/N4S/DVEBMGS01/exe:/usr/sap/N4S/DVEBMGS01/exe:/usr/sap/N4S/DVEBMGS01/exe:/usr/sap/N4S/SYS/exe/run:/sapdb/programs/lib

-> exe path = PATH=/usr/lib/java/bin:/usr/sap/N4S/DVEBMGS01/j2ee/os_libs:/sapdb/programs/bin:/usr/lib/java/bin:.:/home/n4sadm:/usr/sap/N4S/SYS/exe/run:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin

**********************************************************************

[Thr 182897318208] JStartupICreateProcess: fork process (pid 25460)

[Thr 182897318208] Wed Jun 16 11:07:47 2010

[Thr 182897318208] *** ERROR => invalid return code of process [bootstrap_ID14704900] (exitcode = 255) [jstartxx_mt. 1642]

[Thr 182897318208] JControlExecuteBootstrap: error executing bootstrap node [bootstrap_ID14704900] (rc = 255)

[Thr 182897318208] JControlExecuteBootstrap: execute bootstrap process [bootstrap_ID14704950]

[Thr 182897318208] [Node: server0 bootstrap] java home is set by profile parameter

Java Home: /usr/lib/java

[Thr 182897318208] JStartupICheckFrameworkPackage: can't find framework package /usr/sap/N4S/SYS/exe/run/jvmx.jar

**********************************************************************

JStartupIReadSection: read node properties [bootstrap_ID14704950]

-> node name : server0 bootstrap

-> node type : bootstrap

-> node execute : yes

-> jlaunch parameters :

-> java path : /usr/lib/java

-> java parameters : -Djco.jarm=1

-> java vm version : J2RE 1.4.2 IBM J9 2.2 Linux amd64-64 j9xa64142ifx-20051125 (JIT enabled)

-> java vm vendor : IBM J9SE VM (IBM Corporation)

-> java vm type : <default>

-> java vm cpu : amd64

-> heap size : 1024M

-> root path : /usr/sap/N4S/DVEBMGS01/j2ee/cluster

-> class path : ./bootstrap/launcher.jar

-> OS libs path : /usr/sap/N4S/DVEBMGS01/j2ee/os_libs

-> main class : com.sap.engine.offline.OfflineToolStart

-> framework class : com.sap.bc.proj.jstartup.JStartupFramework

-> registr. class : com.sap.bc.proj.jstartup.JStartupNatives

-> framework path : /usr/sap/N4S/SYS/exe/run/jstartup.jar:/usr/sap/N4S/SYS/exe/run/jvmx.jar

-> parameters : com.sap.engine.bootstrap.Bootstrap ./bootstrap ID014704950

-> debuggable : yes

-> debug mode : no

-> debug port : 60000

-> shutdown timeout : 120000

**********************************************************************

**********************************************************************

JControlStartJLaunch: program = /usr/sap/N4S/DVEBMGS01/exe/jlaunch

-> arg[00] = /usr/sap/N4S/DVEBMGS01/exe/jlaunch

-> arg[01] = pf=/usr/sap/N4S/SYS/profile/N4S_DVEBMGS01_n4shost

-> arg[02] = -DSAPINFO=N4S_01_bootstrap

-> arg[03] = -nodeId=-1

-> arg[04] = -file=/usr/sap/N4S/DVEBMGS01/j2ee/cluster/instance.properties

-> arg[05] = -syncSem=14778432

-> arg[06] = -nodeName=bootstrap_ID14704950

-> arg[07] = -jvmOutFile=/usr/sap/N4S/DVEBMGS01/work/jvm_bootstrap_ID14704950.out

-> arg[08] = -stdOutFile=/usr/sap/N4S/DVEBMGS01/work/std_bootstrap_ID14704950.out

-> arg[09] = -locOutFile=/usr/sap/N4S/DVEBMGS01/work/dev_bootstrap_ID14704950

-> arg[10] = -mode=BOOTSTRAP

-> arg[11] = pf=/usr/sap/N4S/SYS/profile/N4S_DVEBMGS01_n4shost

-> arg[12] = -DSAPSTART=1

-> arg[13] = -DCONNECT_PORT=33065

-> arg[14] = -DSAPSYSTEM=01

-> arg[15] = -DSAPSYSTEMNAME=N4S

-> arg[16] = -DSAPMYNAME=n4shost_N4S_01

-> arg[17] = -DSAPPROFILE=/usr/sap/N4S/SYS/profile/N4S_DVEBMGS01_n4shost

-> arg[18] = -DFRFC_FALLBACK=ON

-> arg[19] = -DFRFC_FALLBACK_HOST=localhost

-> lib path = LD_LIBRARY_PATH=/usr/lib/java/jre/bin:/usr/lib/java/bin/../jre/bin:/usr/lib/java/jre/bin/j9vm:/usr/lib/java/jre/bin:/usr/sap/N4S/DVEBMGS01/exe:/usr/sap/N4S/DVEBMGS01/exe:/usr/sap/N4S/DVEBMGS01/exe:/tmp/sapinst_exe.11133.1149865079:/usr/sap/N4S/SYS/exe/run:/sapdb/programs/lib:/usr/lib:/usr/sap/N4S/DVEBMGS01/j2ee/os_libs:/usr/sap/N4S/DVEBMGS01/exe:/usr/sap/N4S/DVEBMGS01/exe:/usr/sap/N4S/DVEBMGS01/exe:/usr/sap/N4S/SYS/exe/run:/sapdb/programs/lib

-> exe path = PATH=/usr/lib/java/bin:/usr/sap/N4S/DVEBMGS01/j2ee/os_libs:/sapdb/programs/bin:/usr/lib/java/bin:.:/home/n4sadm:/usr/sap/N4S/SYS/exe/run:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin

**********************************************************************

[Thr 182897318208] JStartupICreateProcess: fork process (pid 25465)

[Thr 182897318208] Wed Jun 16 11:07:49 2010

[Thr 182897318208] *** ERROR => invalid return code of process [bootstrap_ID14704950] (exitcode = 255) [jstartxx_mt. 1642]

[Thr 182897318208] JControlExecuteBootstrap: error executing bootstrap node [bootstrap_ID14704950] (rc = 255)

[Thr 182897318208] JControlCloseProgram: started (exitcode = 255)

[Thr 182897318208] JControlCloseProgram: good bye... (exitcode = 255)