Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /oracle/db10g
System name:    AIX
Node name:      zn_vlsdbs_01
Release:        1
Version:        6
Machine:        00C20ED54C00
Instance name: znavls1
Redo thread mounted by this instance: 1
Oracle process number: 18
Unix process pid: 17039370, image: oracle@zn_vlsdbs_01 (CJQ0)

*** 2012-06-01 16:25:32.615
*** SERVICE NAMESYS$BACKGROUND) 2012-06-01 16:25:23.013
*** SESSION ID150.1) 2012-06-01 16:25:23.013
Waited for process J000 to initialize for 60 seconds
*** 2012-06-01 16:25:32.615
Dumping diagnostic information for J000:
OS pid = 33096188
loadavg : 9.16 11.62 9.51
swap info: free_mem = 551.00M rsv = 96.00M
           alloc = 13816.41M avail = 24576.00M swap_free = 10759.59M
       F S      UID      PID     PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY
TIME CMD
  240001 A   oracle 33096188        1   0  60 20 11c4d5f590 96460          16:24:34
    -  0:00 ora_j000_znavls1
skgpgpstack: read() for cmd /bin/sh -c '/usr/bin/procstack 33096188 2>&1' timed out a
fter 60 seconds
*** 2012-06-01 16:27:09.036
*** 2012-06-01 16:27:19.038
Waited for process J000 to initialize for 70 seconds
*** 2012-06-01 16:27:19.038
Dumping diagnostic information for J000:
OS pid = 33096188
loadavg : 2.97 8.90 8.71
swap info: free_mem = 2815.93M rsv = 96.00M
           alloc = 13846.67M avail = 24576.00M swap_free = 10729.33M
       F S      UID      PID     PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY
TIME CMD
  240001 A   oracle 33096188        1   0  60 20 11c4d5f590 96460          16:24:34
    -  0:00 ora_j000_znavls1
33096188: ora_j000_znavls1
0x09000000000ecee4  thread_wait(0x12c0000012c) + 0x244
0x00000001000fcb74  sskgpwwait(??, ??, ??, ??, ??) + 0x34
0x00000001000fa15c  skgpwwait(??, ??, ??, ??, ??) + 0xbc
0x000000010011e70c  kslges(??, ??, ??, ??, ??) + 0x54c
0x000000010012253c  kslgetl(??, ??, ??, ??) + 0x33c
0x00000001049f10b8  ksfglt(??, ??, ??, ??, ??) + 0x198
0x00000001000847f4  kghfrunp(??, ??, ??, ??, ??, ??, ??) + 0x794
0x000000010007a4c8  kghfnd(??, ??, ??, ??, ??, ??) + 0x7e8
0x0000000100098484  kghalo(??, ??, ??, ??, ??, ??, ??, ??) + 0xa24
0x0000000100005948  ksp_param_handle_alloc(??) + 0x168
0x000000010001d25c  kspcrec(??) + 0x1bc
0x0000000100141348  ksucre(??) + 0x408
0x0000000103082a70  kkjrdp() + 0x350
0x000000010430d150  opirip(??, ??, ??) + 0x4f0
0x0000000102d9b558  opidrv(??, ??, ??) + 0x458
0x000000010370c070  sou2o(??, ??, ??, ??) + 0x90
0x00000001000008b0  opimai_real(??, ??) + 0x150
0x0000000100000718  main(??, ??) + 0x98
0x0000000100000340  __start() + 0x70
*** 2012-06-01 16:28:18.750
*** 2012-06-01 16:28:29.619
Waited for process J000 to initialize for 80 seconds
*** 2012-06-01 16:28:29.619
Dumping diagnostic information for J000:
OS pid = 33096188
loadavg : 1.50 7.22 8.13
swap info: free_mem = 2515.27M rsv = 96.00M
           alloc = 13800.50M avail = 24576.00M swap_free = 10775.50M
       F S      UID      PID     PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY
TIME CMD
  240001 A   oracle 33096188        1   0  60 20 11c4d5f590 96464          16:24:34
    -  0:00 ora_j000_znavls1
33096188: ora_j000_znavls1
0x09000000000ecee4  thread_wait(0x12c0000012c) + 0x244
0x00000001000fcb74  sskgpwwait(??, ??, ??, ??, ??) + 0x34
0x00000001000fa15c  skgpwwait(??, ??, ??, ??, ??) + 0xbc
0x000000010011e70c  kslges(??, ??, ??, ??, ??) + 0x54c
0x000000010012253c  kslgetl(??, ??, ??, ??) + 0x33c
0x00000001049f10b8  ksfglt(??, ??, ??, ??, ??) + 0x198
0x00000001000847f4  kghfrunp(??, ??, ??, ??, ??, ??, ??) + 0x794
0x000000010007a4c8  kghfnd(??, ??, ??, ??, ??, ??) + 0x7e8
0x0000000100098484  kghalo(??, ??, ??, ??, ??, ??, ??, ??) + 0xa24
0x0000000100005948  ksp_param_handle_alloc(??) + 0x168
0x000000010001d25c  kspcrec(??) + 0x1bc
0x0000000100141348  ksucre(??) + 0x408
0x0000000103082a70  kkjrdp() + 0x350
0x000000010430d150  opirip(??, ??, ??) + 0x4f0
0x0000000102d9b558  opidrv(??, ??, ??) + 0x458
0x000000010370c070  sou2o(??, ??, ??, ??) + 0x90
0x00000001000008b0  opimai_real(??, ??) + 0x150
0x0000000100000718  main(??, ??) + 0x98
0x0000000100000340  __start() + 0x70
*** 2012-06-01 16:28:31.547
*** 2012-06-01 16:28:41.549
Waited for process J000 to initialize for 90 seconds
*** 2012-06-01 16:28:41.549
Dumping diagnostic information for J000:
OS pid = 33096188
loadavg : 1.54 6.95 8.02
swap info: free_mem = 2432.48M rsv = 96.00M
           alloc = 13792.54M avail = 24576.00M swap_free = 10783.46M
       F S      UID      PID     PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY
TIME CMD
  240001 A   oracle 33096188        1   0  60 20 11c4d5f590 96464          16:24:34
    -  0:00 ora_j000_znavls1
33096188: ora_j000_znavls1
0x09000000000ecee4  thread_wait(0x12c0000012c) + 0x244
0x00000001000fcb74  sskgpwwait(??, ??, ??, ??, ??) + 0x34
0x00000001000fa15c  skgpwwait(??, ??, ??, ??, ??) + 0xbc
0x000000010011e70c  kslges(??, ??, ??, ??, ??) + 0x54c
0x000000010012253c  kslgetl(??, ??, ??, ??) + 0x33c
0x00000001049f10b8  ksfglt(??, ??, ??, ??, ??) + 0x198
0x00000001000847f4  kghfrunp(??, ??, ??, ??, ??, ??, ??) + 0x794
0x000000010007a4c8  kghfnd(??, ??, ??, ??, ??, ??) + 0x7e8
0x0000000100098484  kghalo(??, ??, ??, ??, ??, ??, ??, ??) + 0xa24
0x0000000100005948  ksp_param_handle_alloc(??) + 0x168
0x000000010001d25c  kspcrec(??) + 0x1bc
0x0000000100141348  ksucre(??) + 0x408
0x0000000103082a70  kkjrdp() + 0x350
0x000000010430d150  opirip(??, ??, ??) + 0x4f0
0x0000000102d9b558  opidrv(??, ??, ??) + 0x458
0x000000010370c070  sou2o(??, ??, ??, ??) + 0x90
0x00000001000008b0  opimai_real(??, ??) + 0x150
0x0000000100000718  main(??, ??) + 0x98
0x0000000100000340  __start() + 0x70
*** 2012-06-01 16:28:43.489
*** 2012-06-01 16:28:53.491
Waited for process J000 to initialize for 100 seconds
*** 2012-06-01 16:28:53.491
Dumping diagnostic information for J000:
OS pid = 33096188
loadavg : 1.69 6.81 7.96
swap info: free_mem = 2379.35M rsv = 96.00M
           alloc = 13783.44M avail = 24576.00M swap_free = 10792.56M
       F S      UID      PID     PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY
TIME CMD
  240001 A   oracle 33096188        1   0  60 20 11c4d5f590 96464          16:24:34
    -  0:00 ora_j000_znavls1
33096188: ora_j000_znavls1
0x09000000000ecee4  thread_wait(0x12c0000012c) + 0x244
/oracle/db10g/admin/znavls/bdump/znavls1_cjq0_17039370.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /oracle/db10g
System name:    AIX
Node name:      zn_vlsdbs_01
Release:        1
Version:        6
Machine:        00C20ED54C00
Instance name: znavls1
Redo thread mounted by this instance: 1
Oracle process number: 18
Unix process pid: 17039370, image: oracle@zn_vlsdbs_01 (CJQ0)

*** 2012-06-01 16:25:32.615
*** SERVICE NAMESYS$BACKGROUND) 2012-06-01 16:25:23.013
*** SESSION ID150.1) 2012-06-01 16:25:23.013
Waited for process J000 to initialize for 60 seconds
*** 2012-06-01 16:25:32.615
Dumping diagnostic information for J000:
OS pid = 33096188
loadavg : 9.16 11.62 9.51
swap info: free_mem = 551.00M rsv = 96.00M
           alloc = 13816.41M avail = 24576.00M swap_free = 10759.59M
       F S      UID      PID     PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY
TIME CMD
  240001 A   oracle 33096188        1   0  60 20 11c4d5f590 96460          16:24:34
    -  0:00 ora_j000_znavls1
skgpgpstack: read() for cmd /bin/sh -c '/usr/bin/procstack 33096188 2>&1' timed out a
fter 60 seconds
*** 2012-06-01 16:27:09.036
*** 2012-06-01 16:27:19.038
Waited for process J000 to initialize for 70 seconds
*** 2012-06-01 16:27:19.038
Dumping diagnostic information for J000:
OS pid = 33096188
loadavg : 2.97 8.90 8.71
swap info: free_mem = 2815.93M rsv = 96.00M
           alloc = 13846.67M avail = 24576.00M swap_free = 10729.33M
       F S      UID      PID     PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY
TIME CMD
  240001 A   oracle 33096188        1   0  60 20 11c4d5f590 96460          16:24:34
    -  0:00 ora_j000_znavls1
33096188: ora_j000_znavls1
0x09000000000ecee4  thread_wait(0x12c0000012c) + 0x244
0x00000001000fcb74  sskgpwwait(??, ??, ??, ??, ??) + 0x34
0x00000001000fa15c  skgpwwait(??, ??, ??, ??, ??) + 0xbc
0x000000010011e70c  kslges(??, ??, ??, ??, ??) + 0x54c
0x000000010012253c  kslgetl(??, ??, ??, ??) + 0x33c
0x00000001049f10b8  ksfglt(??, ??, ??, ??, ??) + 0x198
0x00000001000847f4  kghfrunp(??, ??, ??, ??, ??, ??, ??) + 0x794
0x000000010007a4c8  kghfnd(??, ??, ??, ??, ??, ??) + 0x7e8
0x0000000100098484  kghalo(??, ??, ??, ??, ??, ??, ??, ??) + 0xa24
0x0000000100005948  ksp_param_handle_alloc(??) + 0x168
0x000000010001d25c  kspcrec(??) + 0x1bc
0x0000000100141348  ksucre(??) + 0x408
0x0000000103082a70  kkjrdp() + 0x350
0x000000010430d150  opirip(??, ??, ??) + 0x4f0
0x0000000102d9b558  opidrv(??, ??, ??) + 0x458
0x000000010370c070  sou2o(??, ??, ??, ??) + 0x90
0x00000001000008b0  opimai_real(??, ??) + 0x150
0x0000000100000718  main(??, ??) + 0x98
0x0000000100000340  __start() + 0x70
*** 2012-06-01 16:28:18.750
*** 2012-06-01 16:28:29.619
Waited for process J000 to initialize for 80 seconds
*** 2012-06-01 16:28:29.619
Dumping diagnostic information for J000:
OS pid = 33096188
loadavg : 1.50 7.22 8.13
swap info: free_mem = 2515.27M rsv = 96.00M
           alloc = 13800.50M avail = 24576.00M swap_free = 10775.50M
       F S      UID      PID     PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY
TIME CMD
  240001 A   oracle 33096188        1   0  60 20 11c4d5f590 96464          16:24:34
    -  0:00 ora_j000_znavls1
33096188: ora_j000_znavls1
0x09000000000ecee4  thread_wait(0x12c0000012c) + 0x244
0x00000001000fcb74  sskgpwwait(??, ??, ??, ??, ??) + 0x34
0x00000001000fa15c  skgpwwait(??, ??, ??, ??, ??) + 0xbc
0x000000010011e70c  kslges(??, ??, ??, ??, ??) + 0x54c
0x000000010012253c  kslgetl(??, ??, ??, ??) + 0x33c
0x00000001049f10b8  ksfglt(??, ??, ??, ??, ??) + 0x198
0x00000001000847f4  kghfrunp(??, ??, ??, ??, ??, ??, ??) + 0x794
0x000000010007a4c8  kghfnd(??, ??, ??, ??, ??, ??) + 0x7e8
0x0000000100098484  kghalo(??, ??, ??, ??, ??, ??, ??, ??) + 0xa24
0x0000000100005948  ksp_param_handle_alloc(??) + 0x168
0x000000010001d25c  kspcrec(??) + 0x1bc
0x0000000100141348  ksucre(??) + 0x408
0x0000000103082a70  kkjrdp() + 0x350
0x000000010430d150  opirip(??, ??, ??) + 0x4f0
0x0000000102d9b558  opidrv(??, ??, ??) + 0x458
0x000000010370c070  sou2o(??, ??, ??, ??) + 0x90
0x00000001000008b0  opimai_real(??, ??) + 0x150
0x0000000100000718  main(??, ??) + 0x98
0x0000000100000340  __start() + 0x70
*** 2012-06-01 16:28:31.547
*** 2012-06-01 16:28:41.549
Waited for process J000 to initialize for 90 seconds
*** 2012-06-01 16:28:41.549
Dumping diagnostic information for J000:
OS pid = 33096188
loadavg : 1.54 6.95 8.02
swap info: free_mem = 2432.48M rsv = 96.00M
           alloc = 13792.54M avail = 24576.00M swap_free = 10783.46M
       F S      UID      PID     PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY
TIME CMD
  240001 A   oracle 33096188        1   0  60 20 11c4d5f590 96464          16:24:34
    -  0:00 ora_j000_znavls1
33096188: ora_j000_znavls1
0x09000000000ecee4  thread_wait(0x12c0000012c) + 0x244
0x00000001000fcb74  sskgpwwait(??, ??, ??, ??, ??) + 0x34
0x00000001000fa15c  skgpwwait(??, ??, ??, ??, ??) + 0xbc
0x000000010011e70c  kslges(??, ??, ??, ??, ??) + 0x54c
0x000000010012253c  kslgetl(??, ??, ??, ??) + 0x33c
0x00000001049f10b8  ksfglt(??, ??, ??, ??, ??) + 0x198
0x00000001000847f4  kghfrunp(??, ??, ??, ??, ??, ??, ??) + 0x794
0x000000010007a4c8  kghfnd(??, ??, ??, ??, ??, ??) + 0x7e8
0x0000000100098484  kghalo(??, ??, ??, ??, ??, ??, ??, ??) + 0xa24
0x0000000100005948  ksp_param_handle_alloc(??) + 0x168
0x000000010001d25c  kspcrec(??) + 0x1bc
0x0000000100141348  ksucre(??) + 0x408
0x0000000103082a70  kkjrdp() + 0x350
0x000000010430d150  opirip(??, ??, ??) + 0x4f0
0x0000000102d9b558  opidrv(??, ??, ??) + 0x458
0x000000010370c070  sou2o(??, ??, ??, ??) + 0x90
0x00000001000008b0  opimai_real(??, ??) + 0x150
0x0000000100000718  main(??, ??) + 0x98
0x0000000100000340  __start() + 0x70
*** 2012-06-01 16:28:43.489
*** 2012-06-01 16:28:53.491
Waited for process J000 to initialize for 100 seconds
*** 2012-06-01 16:28:53.491
Dumping diagnostic information for J000:
OS pid = 33096188
loadavg : 1.69 6.81 7.96
swap info: free_mem = 2379.35M rsv = 96.00M
           alloc = 13783.44M avail = 24576.00M swap_free = 10792.56M
       F S      UID      PID     PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY
TIME CMD
  240001 A   oracle 33096188        1   0  60 20 11c4d5f590 96464          16:24:34
    -  0:00 ora_j000_znavls1
33096188: ora_j000_znavls1
0x09000000000ecee4  thread_wait(0x12c0000012c) + 0x244
0x00000001000fcb74  sskgpwwait(??, ??, ??, ??, ??) + 0x34
0x00000001000fa15c  skgpwwait(??, ??, ??, ??, ??) + 0xbc
0x000000010011e70c  kslges(??, ??, ??, ??, ??) + 0x54c
0x000000010012253c  kslgetl(??, ??, ??, ??) + 0x33c
0x00000001049f10b8  ksfglt(??, ??, ??, ??, ??) + 0x198
0x00000001000847f4  kghfrunp(??, ??, ??, ??, ??, ??, ??) + 0x794
0x000000010007a4c8  kghfnd(??, ??, ??, ??, ??, ??) + 0x7e8
0x0000000100098484  kghalo(??, ??, ??, ??, ??, ??, ??, ??) + 0xa24
0x0000000100005948  ksp_param_handle_alloc(??) + 0x168
0x000000010001d25c  kspcrec(??) + 0x1bc
0x0000000100141348  ksucre(??) + 0x408
0x0000000103082a70  kkjrdp() + 0x350
0x000000010430d150  opirip(??, ??, ??) + 0x4f0
0x0000000102d9b558  opidrv(??, ??, ??) + 0x458
0x000000010370c070  sou2o(??, ??, ??, ??) + 0x90
0x00000001000008b0  opimai_real(??, ??) + 0x150
0x0000000100000718  main(??, ??) + 0x98
0x0000000100000340  __start() + 0x70
*** 2012-06-01 16:28:55.454
*** 2012-06-01 16:29:05.456
Waited for process J000 to initialize for 110 seconds
*** 2012-06-01 16:29:05.456
0x00000001000847f4  kghfrunp(??, ??, ??, ??, ??, ??, ??) + 0x794
0x000000010007a4c8  kghfnd(??, ??, ??, ??, ??, ??) + 0x7e8
0x0000000100098484  kghalo(??, ??, ??, ??, ??, ??, ??, ??) + 0xa24
0x0000000100005948  ksp_param_handle_alloc(??) + 0x168
0x000000010001d25c  kspcrec(??) + 0x1bc
0x0000000100141348  ksucre(??) + 0x408
0x0000000103082a70  kkjrdp() + 0x350
0x000000010430d150  opirip(??, ??, ??) + 0x4f0
0x0000000102d9b558  opidrv(??, ??, ??) + 0x458
0x000000010370c070  sou2o(??, ??, ??, ??) + 0x90
0x00000001000008b0  opimai_real(??, ??) + 0x150
0x0000000100000718  main(??, ??) + 0x98
0x0000000100000340  __start() + 0x70
*** 2012-06-01 16:28:55.454
*** 2012-06-01 16:29:05.456
Waited for process J000 to initialize for 110 seconds
*** 2012-06-01 16:29:05.456
Dumping diagnostic information for J000:
OS pid = 33096188
loadavg : 1.68 6.56 7.85
swap info: free_mem = 2295.70M rsv = 96.00M
           alloc = 13771.25M avail = 24576.00M swap_free = 10804.75M
       F S      UID      PID     PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY
TIME CMD
  240001 A   oracle 33096188        1   0  60 20 11c4d5f590 96464          16:24:34
    -  0:00 ora_j000_znavls1
33096188: ora_j000_znavls1
0x00000001000820c4  kghfrmrg(??, ??, ??, ??, ??) + 0x124
0x0000000100095d9c  kghfre(??, ??, ??, ??, ??) + 0x57c
0x0000000100088228  kghfrh_internal(??, ??, ??) + 0x528
0x00000001000947b0  kghfrh(??, ??) + 0x10
0x00000001047cceec  kksFreeHeap(??, ??, ??) + 0xac
0x00000001047cd118  kksFreeHeapGetMutex(??, ??, ??, ??) + 0x198
0x0000000101f84468  kksCursorFreeCallBack(??, ??, ??) + 0x88
0x000000010030ba24  kglobf0(??, ??, ??, ??) + 0x264
0x00000001002f9868  kglhpd_internal(??, ??, ??, ??) + 0x228
0x00000001002fd3f0  kglhpd(??, ??, ??) + 0x10
0x0000000100086528  kghfrx(??, ??, ??) + 0x28
0x0000000100084ba4  kghfrunp(??, ??, ??, ??, ??, ??, ??) + 0xb44
0x000000010007a4c8  kghfnd(??, ??, ??, ??, ??, ??) + 0x7e8
0x0000000100098484  kghalo(??, ??, ??, ??, ??, ??, ??, ??) + 0xa24
0x0000000100005948  ksp_param_handle_alloc(??) + 0x168
0x000000010001d25c  kspcrec(??) + 0x1bc
0x0000000100141348  ksucre(??) + 0x408
0x0000000103082a70  kkjrdp() + 0x350
0x000000010430d150  opirip(??, ??, ??) + 0x4f0
0x0000000102d9b558  opidrv(??, ??, ??) + 0x458
0x000000010370c070  sou2o(??, ??, ??, ??) + 0x90
0x00000001000008b0  opimai_real(??, ??) + 0x150
0x0000000100000718  main(??, ??) + 0x98
0x0000000100000340  __start() + 0x70
*** 2012-06-01 16:29:07.277
Killing process (ospid 33096188):  (reason=x4 error=0)
... and the process is still alive after kill!

过段时间做一个公开课，关于怎么处理数据库hang住的公开课！

谢谢 相老师   我正在看您的免费视频   请问收费视频在哪里？

Oracle把很多需要定时运行的操作都以job的方式去执行，然后通过J000进程去实现。
在资源不足的情况下，会出现你上面的问题。
注意整体观察系统资源。

应该是系统资源不足，特别是内存和资源资源。
确认一下你的process、memory相关配置，特别是process先关配置。

数据库里面只有EMD_MAINTENANCE.EXECUTE_EM_DBMS_JOB_PROCS() 一个job

没有详细读，应该是你的一个job导致数据库hang住，关注一下那个job。

Fri Jun  1 05:30:07 2012
ALTER SYSTEM ARCHIVE LOG
Fri Jun  1 05:30:08 2012
Thread 1 cannot allocate new log, sequence 1947
Checkpoint not complete
  Current log# 2 seq# 1946 mem# 0: /dev/rdb_ro21_128m
Fri Jun  1 05:30:11 2012
Thread 1 advanced to log sequence 1947 (LGWR switch)
  Current log# 1 seq# 1947 mem# 0: /dev/rdb_ro11_128m
Fri Jun  1 12:57:31 2012
Thread 1 advanced to log sequence 1948 (LGWR switch)
  Current log# 2 seq# 1948 mem# 0: /dev/rdb_ro21_128m
Fri Jun  1 16:29:16 2012
kkjcre1p: unable to spawn jobq slave process
Fri Jun  1 16:29:16 2012
Errors in file /oracle/db10g/admin/znavls/bdump/znavls1_cjq0_17039370.trc:

Fri Jun  1 16:30:16 2012
PMON failed to acquire latch, see PMON dump
Fri Jun  1 16:31:04 2012
WARNING: inbound connection timed out (ORA-3136)
Fri Jun  1 16:31:38 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 16:32:02 2012
PMON failed to acquire latch, see PMON dump
Fri Jun  1 16:33:01 2012
WARNING: inbound connection timed out (ORA-3136)
Fri Jun  1 16:33:04 2012
WARNING: inbound connection timed out (ORA-3136)
Fri Jun  1 16:33:09 2012
WARNING: inbound connection timed out (ORA-3136)
Fri Jun  1 16:33:09 2012
WARNING: inbound connection timed out (ORA-3136)
Fri Jun  1 16:33:13 2012
WARNING: inbound connection timed out (ORA-3136)
Fri Jun  1 16:33:14 2012
WARNING: inbound connection timed out (ORA-3136)
WARNING: inbound connection timed out (ORA-3136)
Fri Jun  1 16:34:27 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 16:36:49 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 16:39:10 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 16:41:31 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 16:43:51 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 16:46:13 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 16:48:34 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 16:50:55 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 16:51:13 2012
IPC Send timeout detected. Receiver ospid 42336460
Receiver is waiting for a latch dumping latch state for receiver -17068
Fri Jun  1 16:51:13 2012
Errors in file /oracle/db10g/admin/znavls/udump/znavls1_ora_67043510.trc:
Fri Jun  1 16:51:16 2012
Errors in file /oracle/db10g/admin/znavls/bdump/znavls1_pz97_42336460.trc:
Fri Jun  1 16:51:16 2012
Trace dumping is performing id=[cdmp_20120601165116]
Fri Jun  1 16:53:16 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 16:53:38 2012
IPC Send timeout detected. Receiver ospid 10747918
Receiver is waiting for a latch dumping latch state for receiver -17068
Fri Jun  1 16:53:38 2012
Errors in file /oracle/db10g/admin/znavls/udump/znavls1_ora_67043510.trc:
Fri Jun  1 16:53:39 2012
Errors in file /oracle/db10g/admin/znavls/bdump/znavls1_pz98_10747918.trc:
Fri Jun  1 16:53:39 2012
Trace dumping is performing id=[cdmp_20120601165339]
Fri Jun  1 16:55:37 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 16:57:58 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:00:18 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:02:40 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:05:01 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:07:22 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:09:43 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:12:04 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:14:25 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:16:46 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:19:07 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:21:28 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:23:49 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:26:10 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:28:31 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:28:42 2012
Process q001 died, see its trace file
Fri Jun  1 17:28:42 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:28:54 2012
Process q001 died, see its trace file
Fri Jun  1 17:28:54 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:29:07 2012
Process q001 died, see its trace file
Fri Jun  1 17:29:07 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:29:18 2012
Process q001 died, see its trace file
Fri Jun  1 17:29:18 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:29:30 2012
Process q001 died, see its trace file
Fri Jun  1 17:29:30 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:29:42 2012
Process q001 died, see its trace file
Fri Jun  1 17:29:42 2012
ksvcreate: Process(q001) creation failed
Fri Jun  1 17:29:54 2012
Process q001 died, see its trace file
Fri Jun  1 17:29:54 2012
ksvcreate: Process(q001) creation failed

Redo thread mounted by this instance: 1
Oracle process number: 39
Unix process pid: 43843694, image: oracle@zn_vlsdbs_01 (P014)

*** SERVICE NAME) 2012-06-01 19:12:05.932
*** SESSION ID129.1) 2012-06-01 19:12:05.932
*** 2012-06-01 19:12:05.932
Start recovery for domain 0, valid = 0, flags = 0x5
*** 2012-06-01 19:12:06.169
KCRP: blocks claimed = 23, eliminated = 0
----- Recovery Hash Table Statistics ---------
Hash table buckets = 32768
Longest hash chain = 1
Average hash chain = 23/23 = 1.0
Max compares per lookup = 0
Avg compares per lookup = 0/23 = 0.0
----------------------------------------------
----- Recovery Hash Table Statistics ---------
Hash table buckets = 32768
Longest hash chain = 1
Average hash chain = 23/23 = 1.0
Max compares per lookup = 1
Avg compares per lookup = 349/349 = 1.0
----------------------------------------------
~

AIX  ERRPT信息
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
A924A5FC   0601163212 P S SYSPROC        SOFTWARE PROGRAM ABNORMALLY TERMINATED
[zn_vlsdbs_01][/]> errpt -aj A924A5FC
---------------------------------------------------------------------------
LABEL:          CORE_DUMP
IDENTIFIER:     A924A5FC

Date/Time:       Fri Jun  1 16:32:07 GMT+08:00 2012
Sequence Number: 30596
Machine Id:      00C20ED54C00
Node Id:         zn_vlsdbs_01
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSPROC         

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SOFTWARE PROGRAM

User Causes
USER GENERATED SIGNAL

        Recommended Actions
        CORRECT THEN RETRY

Failure Causes
SOFTWARE PROGRAM

        Recommended Actions
        RERUN THE APPLICATION PROGRAM
        IF PROBLEM PERSISTS THEN DO THE FOLLOWING
        CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data
SIGNAL NUMBER
          11
USER'S PROCESS ID:
              12779688
FILE SYSTEM SERIAL NUMBER
          13
INODE NUMBER
           0      116517
CORE FILE NAME
/oracle/db10g/log/zn_vlsdbs_01/racg/racgimon/core
PROGRAM NAME
racgimon
STACK EXECUTION DISABLED
           0
COME FROM ADDRESS REGISTER
sltsmna 14

PROCESSOR ID
  hw_fru_id: 1
  hw_cpu_id: 6

ADDITIONAL INFORMATION
clscugblm B88
clscugblm B24
crs_qstat 178
crs_qstat 40
clsrcqrya 24C
clsrcinst 1F58
clsrcacti 358
clsrd_do 1FC
_pthread_ F4
??

Symptom Data
REPORTABLE
1
INTERNAL ERROR
0
SYMPTOM CODE
PCSS/SPI2 FLDS/racgimon SIG/11 FLDS/clscugblm VALU/b88 FLDS/clsrcqrya