IDS Forum

HDR: ASF Echo-Thread Server error

Posted By: VIKAS HIVARKAR
Date: Friday, 26 August 2011, at 4:13 a.m.

Hello All,

Primary - IDS 11.50.FC8W2 on HP-UX 11.23, 12 cores, IDS memory - 48G
Secondry - IDS 11.50.FC8W2 on HP-UX 11.23, 04 cores, IDS memory - 40G
2 SDS nodes(updatable)

DRAUTO 0
DRINTERVAL 15
DRTIMEOUT 90

Things were working fine until two days back we started getting following messages in online.log:
AFAIK, No config/network/OS/appln changes made.

13:02:13 DR: Sending log 173640, size 150000 pages, 93.40 percent used
13:09:42 DR: Send error
13:09:42 ASF Echo-Thread Server: asfcode = -25580: oserr = 4: errstr = : System error occurred in network function.
System error = 4.
13:09:42 DR_ERR set to -3
13:09:42 DR: Failure recovery error (14)
13:09:42 DR: Turned off on primary server
13:09:42 DR: Cannot connect to secondary server
13:09:52 DR: Primary server connected
13:09:53 DR: Secondary server needs failure recovery

Checking with the network team and the bandwidth utilization looks consistence and fine.

Some chunks on Secondry(including root chunk) are in PI- status BUT all good on Primary.
c0000007fa4ef1d0 1 1 0 250000 202365 PI-B- /dev/prod/rootdbs
c00000081725ad70 7 16 0 3750000 3124947 PI-B- /dev/prod1/dd01_data

During the error I see that the log applying is almost stoped on secondry:
Following o/p interver ~2 sec

c0000008172326c0 94 U---C-L 173641 3:1650053 150000 135680 90.45
$ onstat -l |grep U---C-L
c0000008172326c0 94 U---C-L 173641 3:1650053 150000 135680 90.45
$ onstat -l |grep U---C-L
c0000008172326c0 94 U---C-L 173641 3:1650053 150000 135680 90.45
$ onstat -l |grep U---C-L
c0000008172326c0 94 U---C-L 173641 3:1650053 150000 135680 90.45
$ onstat -l |grep U---C-L
c0000008172326c0 94 U---C-L 173641 3:1650053 150000 135680 90.45
$ onstat -l |grep U---C-L
c0000008172326c0 94 U---C-L 173641 3:1650053 150000 135680 90.45
$ onstat -l |grep U---C-L

Does these needs to be checked for consistency? using oncheck on HDR an option?

On Pri: "netstat -s |grep drop" (pasting only with count > 0):
11064159 connections closed (including 31881 drops)
16169 embryonic connections dropped
48 connections dropped by rexmit timeout
25 connections dropped by keepalive
6130 connect requests dropped due to no listener
1812106 ICMP messages dropped

On Sec:
330443 connections closed (including 62512 drops)
34457 embryonic connections dropped
107 connections dropped by rexmit timeout
72 connections dropped by keepalive
1094 connect requests dropped due to no listener

heavy loading in night time(we always do) for last two days the logs on HDR are falling back as compared to Primary and we have to stop loading until both are sync with current log.

Any pointers or suggestions what all areas I need to investigate to deal with this errors?

Regards,
Vikas

Messages In This Thread

• HDR: ASF Echo-Thread Server error
  VIKAS HIVARKAR -- Friday, 26 August 2011, at 4:13 a.m.
  • Re: HDR: ASF Echo-Thread Server error
    Art Kagel -- Friday, 26 August 2011, at 6:11 a.m.

IDS Forum is maintained by Administrator with WebBBS 5.12.