Folks,
IDS11.50 FC8, redHat Linux 6.
Recently(after reverted) , dbWorker3 session ( see below) could
dump/assert failed a couple of times daily.....
Not very familiar to dbWorker3. Can we manage it ? Is it controllable ?
Thanks,
Frank
[informix@stephanie tmp]$ more af.47b85224
01:11:00 Context ptr = 0x000000000178a480
01:11:00 Siginfo ptr = 0x000000000178a5b0
01:11:00 Reg 0(R8) 0x0000000000000000 Reg 1(R9)
0x0000000000000000
01:11:00 Reg 2(R10) 0x0000000000000008 Reg 3(R11)
0x0000000000000246
01:11:00 Reg 4(R12) 0x0000000045777570 Reg 5(R13)
0x00000000c6bae6f0
01:11:00 Reg 6(R14) 0x00000000c6bae6e0 Reg 7(R15)
0x00000000c6baec00
01:11:00 Reg 8(RDI) 0x00000000c6bae6e0 Reg 9(RSI)
0x00000000c6024028
01:11:00 Reg 10(RBP) 0x00000000c6bae6e0 Reg 11(RBX)
0x0000000000000001
01:11:00 Reg 12(RDX) 0x0000000000000000 Reg 13(RAX)
0x0000000000000000
01:11:00 Reg 14(RCX) 0x00007fd88445cba0 Reg 15(RSP)
0x00000000c6bae3e0
01:11:00 Reg 16(RIP) 0x000000000064bfe4 Reg 17(EFL)
0x0000000000010246
01:11:00 Reg 18(CSGSFS) 0x0000000000000033 Reg 19(ERR)
0x0000000000000004
01:11:00 Reg 20(TRAPNO) 0x000000000000000e Reg 21(OLDMASK)
0x0000000000000000
01:11:00 Reg 22(CR2) 0x0000000000000010
01:11:00
01:11:00 IBM Informix Dynamic Server Version 11.50.FC8 Software Serial
Number AAA#B000000
01:11:00 Assert Failed: Exception Caught. Type: MT_EX_OS, Context: mem
01:11:00 Who: Session(13188, informix@, 0, 0xc238c5b8)
Thread(17360, dbWorker3, c2376ae0, 1)
File: mtex.c Line: 417
01:11:00 Action: Please notify IBM Informix Technical Support.
01:11:00 Raw hex dump of stack located in /tmp/af.47b85224.rawstk
01:11:00 Stack for thread: 17360 dbWorker3
base: 0x00000000c6b90000
len: 135168
pc: 0x00000000010c91d7
tos: 0x0000000001789590
state: running
vp: 1
0x00000000010c91d7 (oninit) afstack
0x00000000010cb12c (oninit) afhandler
0x00000000010cb9e4 (oninit) affail_interface
0x00000000010cf7e9 (oninit) mt_ex_throw_sig
0x000000000108d895 (oninit) afsig_handler
0x00007fd8855b5500 (Linux) <signal frame>
0x000000000064bfe4 (oninit) sqapi_error_desc_first
0x000000000064e5c8 (oninit) sqapi_callback_list_call
0x000000000064eab8 (oninit) sqapi_call_callbacks
0x000000000064ca4c (oninit) sqapi_error_dispatch
0x000000000064d0e3 (oninit) sqapi_online_error
0x0000000000549e19 (oninit) mi_exec
0x0000000000d59383 (oninit) aus_refresh
0x0000000000968514 (oninit) udrlm_clang_execute_internal
0x000000000096a319 (oninit) udrlm_clang_execute
0x0000000000907846 (oninit) udrlm_exec_routine
0x000000000063e641 (oninit) udr_execute
0x0000000000559aa5 (oninit) mi_routine_exec
0x0000000000d507d9 (oninit) ph_exec_func
0x0000000000d55511 (oninit) ph_execute
0x0000000000d55be8 (oninit) do_task
0x0000000000d568a8 (oninit) db_sch_worker
0x000000000096861e (oninit) udrlm_clang_execute_internal
0x000000000096a319 (oninit) udrlm_clang_execute
0x0000000000907846 (oninit) udrlm_exec_routine
0x000000000063e641 (oninit) udr_execute
0x0000000000d4dda5 (oninit) dbsched_start_udr
0x000000000109a9bb (oninit) startup
01:11:00 See Also: /tmp/af.47b85224, shmem.47b85224.0
---------------------------------
Begin System Alarm Program Output
---------------------------------
Assertion Failure Type: FAILURE
Host Name: stephanie
Database Server Name: nsofpal_tcp
Time of failure: Sun Nov 24 01:11:00 UTC 2013
AF file: /tmp/af.47b85224
Shared memory file: None
System Blocking: OFF
===========------------- - - - - - -
tail -100 /dbbackup/online_log/online_pal.log:
01:00:09 Checkpoint Completed: duration was 0 seconds.
01:00:09 Sun Nov 24 - loguniq 89958, logpos 0x961d018, timestamp:
0x3027f49f Interval: 278316
01:00:09 Maximum server connections 93
01:00:09 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked
0, Plog used 80, Llog used 21
3
01:00:10 Level 0 Archive started on rootdbs, dblog1, dblog2, dberdata00,
dbdata09_1, dbdata09_2, dbdata0
0, dbdata01, dbdata02, dbdata03, dbdata04, dbdata05, dbdata06, dbdata07,
dbdata08, dbdata09, dbdata10, db
data11, dbdw00, dbdw01, dbdw02, dbdw03, dbdw04, dbdw05, dbdw06, dbdw07,
dbdw08, dbdata20, dbphylog, dbdat
a12, dbdata13, dbdata21, dbersbdata00, dbersbdata01, dbsbdata00
01:05:23 Checkpoint Completed: duration was 0 seconds.
01:05:23 Sun Nov 24 - loguniq 89958, logpos 0x963b018, timestamp:
0x30281701 Interval: 278317
01:05:23 Maximum server connections 93
01:05:23 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked
0, Plog used 7, Llog used 30
01:07:52 listener-thread: err = -25582: oserr = 0: errstr = : Network
connection is broken.
01:10:23 Checkpoint Completed: duration was 0 seconds.
01:10:23 Sun Nov 24 - loguniq 89958, logpos 0x968d018, timestamp:
0x302844be Interval: 278318
01:10:23 Maximum server connections 93
01:10:23 Checkpoint Statistics - Avg. Txn Block Time 0.000, # Txns blocked
0, Plog used 7, Llog used 82
01:11:00 Assert Failed: Exception Caught. Type: MT_EX_OS, Context: mem
01:11:00 IBM Informix Dynamic Server Version 11.50.FC8
01:11:00 Who: Session(13188, informix@, 0, 0xc238c5b8)
Thread(17360, dbWorker3, c2376ae0, 1)
File: mtex.c Line: 417
01:11:00 Action: Please notify IBM Informix Technical Support.
01:11:00 stack trace for pid 22737 written to /tmp/af.47b85224
01:11:00 See Also: /tmp/af.47b85224, shmem.47b85224.0
===========------------- - - - - - -
/usr/informix/bin/onstat -g ath:
IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 3 days
09:23:53 -- 2870104 Kbytes
Threads:
tid tcb rstcb prty status
vp-class name
2 c2e7f338 0 1 IO Idle
8lio* lio vp 0
3 c2ec22a8 0 1 IO Idle
9pio* pio vp 0
4 c2ee22a8 0 1 IO Idle
10aio* aio vp 0
5 c2f012a8 0 1 IO Idle
11msc* msc vp 0
6 c2f302a8 0 1 IO Idle
12fifo* fifo vp 0
7 c2f502a8 0 1 IO Idle
13aio* aio vp 1
8 c2f6f548 c2342028 3 sleeping secs: 1
1cpu main_loop()
9 c2ee2860 0 1 running
14soc* soctcppoll
10 c2f015c0 0 1 running
15soc* soctcppoll
11 c2f01970 0 1 sleeping forever
1cpu* sm_poll
12 c314c178 0 2 sleeping forever
1cpu* soctcplst
13 c31b1898 0 2 sleeping forever
1cpu sm_listen
14 c32a1028 0 1 sleeping secs: 1
1cpu sm_discon
15 c32f7220 0 2 sleeping forever
7cpu* soctcplst
16 c32f7a60 c2342880 1 sleeping secs: 1
1cpu flush_sub(0)
17 c32f7d78 c23430d8 1 sleeping secs: 1
1cpu flush_sub(1)
18 c33300e0 c2343930 1 sleeping secs: 1
1cpu flush_sub(2)
19 c33303f8 c2344188 1 sleeping secs: 1
1cpu flush_sub(3)
20 c3330710 c23449e0 1 sleeping secs: 1
1cpu flush_sub(4)
21 c3330a28 c2345238 1 sleeping secs: 1
1cpu flush_sub(5)
22 c3330d40 c2345a90 1 sleeping secs: 1
1cpu flush_sub(6)
23 c33c20e0 c23462e8 1 sleeping secs: 1
1cpu flush_sub(7)
24 c33c23f8 c2346b40 1 sleeping secs: 1
1cpu flush_sub(8)
25 c33c2710 c2347398 1 sleeping secs: 1
1cpu flush_sub(9)
26 c33c2a28 c2347bf0 1 sleeping secs: 1
1cpu flush_sub(10)
27 c33c2d40 c2348448 1 sleeping secs: 1
1cpu flush_sub(11)
28 c345e0e0 c2348ca0 1 sleeping secs: 1
1cpu flush_sub(12)
29 c345e3f8 c23494f8 1 sleeping secs: 1
1cpu flush_sub(13)
30 c345e710 c2349d50 1 sleeping secs: 1
1cpu flush_sub(14)
31 c345ea28 c234a5a8 1 sleeping secs: 1
1cpu flush_sub(15)
32 c345ed40 0 3 IO Idle
1cpu* kaio
33 c35f2d18 c234ae00 2 sleeping secs: 1
1cpu aslogflush
34 c3646ab8 c234b658 1 sleeping secs: 24
7cpu btscanner_0
36 c3777c70 0 3 IO Idle
7cpu* kaio
51 c37c0670 c234cf60 3 sleeping secs: 1
1cpu* onmode_mon
52 c37c08d0 c234c708 3 sleeping secs: 1
7cpu periodic
60 c3a3c418 c234e010 1 sleeping secs: 300
7cpu* dbScheduler
62 c390f760 c234d7b8 1 sleeping forever
7cpu dbWorker1
63 c3acd220 c234e868 1 sleeping forever
1cpu dbWorker2
75 c48035c8 c234beb0 1 cond wait bp_cond
1cpu bf_priosweep()
7385 c491cfc0 c235f368 1 cond wait netnorm
1cpu sqlexec
7386 c498fdc8 c2350170 1 cond wait netnorm
1cpu sqlexec
7387 c4965ce0 c236bba8 1 cond wait netnorm
7cpu sqlexec
7388 c495fca0 c23614c8 1 cond wait netnorm
7cpu sqlexec
7389 c49349d8 c23509c8 1 cond wait netnorm
7cpu sqlexec
7390 c4938fc0 c234f918 1 cond wait netnorm
1cpu sqlexec
7391 c4929180 c2354430 1 cond wait netnorm
7cpu sqlexec
7394 c492f6c0 c23522d0 1 cond wait netnorm
1cpu sqlexec
7395 c48a03a8 c2353380 1 cond wait netnorm
7cpu sqlexec
7396 c4910fc0 c234f0c0 1 cond wait netnorm
1cpu sqlexec
7397 c37876d8 c2368998 1 cond wait netnorm
1cpu sqlexec
7579 c3c2a448 c2362578 1 cond wait netnorm
1cpu sqlexec
7587 c48a2c68 c236b350 1 cond wait netnorm
1cpu sqlexec
8649 c3bee988 c2359ff8 1 cond wait netnorm
7cpu sqlexec
11387 c4907340 c236c400 1 cond wait netnorm
1cpu sqlexec
11860 c3786aa8 c2361d20 1 cond wait netnorm
7cpu sqlexec
17292 c4932500 c236aaf8 1 ready
7cpu ontape
17294 c4874868 c2360c70 1 IO Wait
7cpu arcbackup1
17295 c3bef568 c2363e80 2 sleeping secs: 1
1cpu arcbackup2
17321 c489b568 c2356de8 1 cond wait netnorm
1cpu sqlexec
17322 c4876728 c2354c88 1 cond wait netnorm
7cpu sqlexec
17323 c489ac68 c235e2b8 1 cond wait netnorm
7cpu sqlexec
17324 c3c081d8 c2351220 1 cond wait netnorm
1cpu sqlexec
17325 c4911d20 c2353bd8 1 cond wait netnorm
1cpu sqlexec
17326 c49236a8 c237b5f8 1 cond wait netnorm
7cpu sqlexec
17327 c494d8b0 c2358f48 1 cond wait netnorm
1cpu sqlexec
17328 c3c854f8 c2356590 1 cond wait netnorm
7cpu sqlexec
17329 c492bfc0 c2355d38 1 cond wait netnorm
7cpu sqlexec
17330 c49396c0 c235eb10 1 cond wait netnorm
1cpu sqlexec
17331 c4913fc0 c235c9b0 1 cond wait netnorm
1cpu sqlexec
17332 c49296c0 c236e560 1 cond wait netnorm
7cpu sqlexec
17333 c49142c8 c237ada0 1 cond wait netnorm
1cpu sqlexec
17334 c49409a0 c2351a78 1 cond wait netnorm
1cpu sqlexec
17335 c497ddc8 c23597a0 1 cond wait netnorm
7cpu sqlexec
17336 c494dcd0 c2357640 1 cond wait netnorm
1cpu sqlexec
17337 c492e110 c235b900 1 cond wait netnorm
7cpu sqlexec
17338 c3be6338 c235a850 1 cond wait netnorm
7cpu sqlexec
17339 c489ba40 c2357e98 1 cond wait netnorm
7cpu sqlexec
17340 c48b89d0 c235da60 1 cond wait netnorm
7cpu sqlexec
17341 c3c92178 c235c158 1 cond wait netnorm
7cpu sqlexec
*17360 c48741e8 c2376ae0 1 running
1cpu* dbWorker3
===========------------- - - - - - -
/usr/informix/bin/onstat -g stk 17360 light:
IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 3 days
09:23:53 -- 2870104 Kbytes
Stack for thread: 17360 dbWorker3
base: 0x00000000c6b90000
len: 135168
pc: 0x000000000107fdcd
tos: 0x00000000017898e0
state: running
vp: 1
Changing data structure forced command termination.
===========------------- - - - - - -
/usr/informix/bin/onstat -u:
IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 3 days
09:23:54 -- 2870104 Kbytes
Userthreads
address flags sessid user tty wait tout
locks nreads nwrites
c2342028 ---P--D 1 informix - 0 0
0 1446 34823
c2342880 ---P--F 0 informix - 0 0
0 0 536707
c23430d8 ---P--F 0 informix - 0 0
0 0 1016757
c2343930 ---P--F 0 informix - 0 0
0 0 1986746
c2344188 ---P--F 0 informix - 0 0
0 0 490505
c23449e0 ---P--F 0 informix - 0 0
0 0 1232776
c2345238 ---P--F 0 informix - 0 0
0 0 689221
c2345a90 ---P--F 0 informix - 0 0
0 0 1322229
c23462e8 ---P--F 0 informix - 0 0
0 0 703281
c2346b40 ---P--F 0 informix - 0 0
0 0 786083
c2347398 ---P--F 0 informix - 0 0
0 0 967357
c2347bf0 ---P--F 0 informix - 0 0
0 0 416551
c2348448 ---P--F 0 informix - 0 0
0 0 634078
c2348ca0 ---P--F 0 informix - 0 0
0 0 355771
c23494f8 ---P--F 0 informix - 0 0
0 0 392212
c2349d50 ---P--F 0 informix - 0 0
0 0 396608
c234a5a8 ---P--F 0 informix - 0 0
0 0 480264
c234ae00 ---P--- 12 informix - 0 0
0 0 1138
c234b658 ---P--B 13 informix - 0 0
0 691600 13175
c234beb0 Y--P--D 34 informix - 4586c970 0
0 0 0
c234c708 ---P--D 17 informix - 0 0
0 33 0
c234cf60 ---P--D 16 informix - 0 0
0 0 0
c234d7b8 ---P--- 27 informix - 0 0
1 21173 5902
c234e010 ---P--- 25 informix - 0 0
1 903 4957
c234e868 ---P--- 28 informix - 0 0
1 7011 8587
c234f0c0 Y--P--- 3711 saapmgr - c3c08b20 0
1 0 0
c234f918 Y--P--- 3705 saapmgr - c3bf2ab0 0
1 24 0
c2350170 Y--P--- 3701 saapmgr - c47734a8 0
1 3 0
c23509c8 Y--P--- 3704 saapmgr - c46ffe38 0
1 0 0
c2351220 Y--P--- 13151 saapmgr - c4773658 0
1 0 2
c2351a78 Y--P--- 13161 saapmgr - c3c276b0 0
1 0 0
c23522d0 Y--P--- 3709 saapmgr - c45925c8 0
1 3 0
c2353380 Y--P--- 3710 saapmgr - c45481d8 0
1 11 0
c2353bd8 Y--P--- 13152 saapmgr - c3c27aa8 0
1 0 2
c2354430 Y--P--- 3706 saapmgr - c4759388 0
1 0 0
c2354c88 Y--P--- 13149 saapmgr - c4909b60 0
1 0 15
c2355d38 Y--P--- 13156 saapmgr - c4974c88 0
1 0 2
c2356590 Y--P--- 13155 saapmgr - c46eeda8 0
1 0 1
c2356de8 Y--P--- 13148 saapmgr - c378bca0 0
1 0 2
c2357640 Y--P--- 13163 saapmgr - c46ff268 0
1 0 0
c2357e98 Y--P--- 13166 saapmgr - c492c918 0
1 0 1
c2358f48 Y--P--- 13154 saapmgr - c491d340 0
1 0 2
c23597a0 Y--P--- 13162 saapmgr - c3c05438 0
1 0 0
c2359ff8 Y--P--- 4686 saapmgr - c48bed30 0
1 1307389 5344
c235a850 Y--P--- 13165 saapmgr - c4773b68 0
1 0 0
c235b900 Y--P--- 13164 saapmgr - c3be3e30 0
1 0 0
c235c158 Y--P--- 13168 saapmgr - c3bf64d0 0
1 0 1
c235c9b0 Y--P--- 13158 saapmgr - c4044778 0
1 0 2
c235da60 Y--P--- 13167 saapmgr - c7595f80 0
1 0 0
c235e2b8 Y--P--- 13150 saapmgr - c49b0210 0
1 0 2
c235eb10 Y--P--- 13157 saapmgr - c3bfbd28 0
1 0 2
c235f368 Y--P--- 3700 saapmgr - c45924a8 0
1 28 0
c2360c70 ------- 13119 informix - 0 0
0 32084517 0
c23614c8 Y--P--- 3703 saapmgr - c4773538 0
1 1 0
c2361d20 Y--P--- 7815 wwwmgr - c4759928 0
1 116 56
c2362578 Y--P--- 3838 wwwmgr - c4737778 0
1 7 0
c2363e80 ------- 13119 informix - 0 0
0 0 0
c2368998 Y--P--- 3712 saapmgr - c40446e8 0
1 0 0
c236aaf8 Y-AP--M 13119 informix - c47592f8 0
0 2740 0
c236b350 Y--P--- 3846 wwwmgr - c46ee4a8 0
1 0 0
c236bba8 Y--P--- 3702 saapmgr - c3c08a00 0
1 3 0
c236c400 Y--P--- 7360 wwwmgr - c47821d8 0
1 997 0
c236e560 Y--P--- 13159 saapmgr - c3789f88 0
1 0 2
c2376ae0 *---P--- 13188 informix - 0 0
1 18 1
c237ada0 Y--P--- 13160 saapmgr - c4378538 0
1 0 2
c237b5f8 Y--P--- 13153 saapmgr - c3beef88 0
1 0 2
66 active, 128 total, 120 maximum concurrent
===========------------- - - - - - -
/usr/informix/bin/onstat -g ses 13188:
IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 3 days
09:23:54 -- 2870104 Kbytes
session effective #RSAM total
used dynamic
id user user tty pid hostname threads memory
memory explain
13188 informix - - 0 - 1 253952
234768 off
tid name rstcb flags curstk status
17360 dbWorker c2376ae0 ---P--- -985499872 running-
Memory pools count 2
name class addr totalsize freesize #allocfrag
#freefrag
13188 V c3cc3040 249856 18376 277 14
13188*O0 V c5a28040 4096 808 1 1
name free used name free used
overhead 0 6576 mtmisc 0 72
resident 0 72 scb 0 216
opentable 0 10336 filetable 0 2632
ru 0 600 log 0 16536
temprec 0 36000 partn 0 208
keys 0 928 ralloc 0 107768
gentcb 0 1584 ostcb 0 2920
sqscb 0 28136 sql 0 144
rdahead 0 1120 hashfiletab 0 552
sqtcb 0 10016 fragman 0 4240
sapi 0 208 SQL error 0 472
SAPI callback 0 904 udr 0 2040
sqlj 0 240
sqscb info
scb sqscb optofc pdqpriority optcompind
directives
c7364028 c6024028 0 0 0 1
Sess SQL Current Iso Lock SQL ISAM F.E.
Id Stmt type Database Lvl Mode ERR ERR Vers
Explain
13188 - sysadmin DR Wait 5 0 0 -
Off
===========------------- - - - - - -
/usr/informix/bin/onstat -g env 13188:
IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 3 days
09:23:54 -- 2870104 Kbytes
Environment for session 13188: (WARNING: No specific environment.)
Variable Value [values-list]
DBCENTURY C
DBDATE Y4MD/
DBDELIMITER |
DBPATH .
DBPRINT lp -s
DBTEMP /tmp
IFX_DEF_TABLE_LOCK ROW
IFX_LARGE_PAGES 0
INFORMIXDIR /usr/informix
[/usr/informix]
[/usr/informix]
[/usr/informix]
INFORMIXSERVER nsofpal_tcp
[nsofpal_tcp]
[nsofpal_tcp]
INFORMIXSQLHOSTS /usr/informix/etc/sqlhosts
INFORMIXTERM terminfo
KAIOON 10000
LANG en_US.UTF-8
LC_COLLATE en_US.UTF-8
LC_CTYPE en_US.UTF-8
LC_MONETARY en_US.UTF-8
LC_NUMERIC en_US.UTF-8
LC_TIME en_US.UTF-8
LKNOTIFY yes
LOCKDOWN no
NODEFDAC no
ONCONFIG onconfig.nsofpal
[onconfig.nsofpal]
[onconfig.nsofpal]
PATH
/usr/informix/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/s
bin:/usr/sbin:/sbin:/etc/opt/emcpower/bin:/home/informix/b
in:/usr/informix/bin:/usr/sbin:.:/home/shared/fqu/informix
/monitor:/dbbackup/bin
SERVER_LOCALE en_US.819
SHELL /bin/bash
TERM xterm
[xterm]
[dumb]
TERMCAP /usr/informix/etc/termcap
[/usr/informix/etc/termcap]
[/etc/termcap]
===========------------- - - - - - -
/usr/informix/bin/onstat -g sql 13188:
IBM Informix Dynamic Server Version 11.50.FC8 -- On-Line -- Up 3 days
09:23:54 -- 2870104 Kbytes
Sess SQL Current Iso Lock SQL ISAM F.E.
Id Stmt type Database Lvl Mode ERR ERR Vers
Explain
13188 - sysadmin DR Wait 5 0 0 -
Off
===========------------- - - - - - -
--485b397dd20b770f3804ebe29a82
