Wednesday, July 2, 2014

Exchange 2007 SP3 Rollup 13 "Microsoft Information store" Crashes

Issue
=================
In Exchange 2007 environment, Single box holds Hub, CAS, MBX roles is installed with SP3 Rollup 13

OS: Microsoft Windows Server 2003 R2 Enterprise Edition x64, Service pack 2

Microsoft Information store is crashing within 2-3 Minutes and on checking the Application log shows the following Event id 



Event Type:       Error
Event Source:       MSExchangeIS
Event Category:       General
Event ID:       9659
Date:             7/1/2014
Time:             12:15:50 PM
User:             N/A
Computer:       computername
Description:
The Microsoft Exchange Information Store encountered an unexpected exception 0xC0000005 at address 00000001403B8C4B while processing a request for user NETWORK SERVICE.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 45 00 63 00 44 00 6f 00   E.c.D.o.
0008: 52 00 70 00 63 00 45 00   R.p.c.E.
0010: 78 00 74 00 32 00 00 00   x.t.2...
0018: 5b 44 49 41 47 5f 43 54   [DIAG_CT
0020: 58 5d 00 00 0e 02 00 00   X]......
0028: ff 4c 0d 00 00 00 00 00   ÿL......
0030: 00 03 00 02 00 00 b9 8e   ......¹Ž
0038: 00 10 0f 01 04 80 d3 15   .....€Ó.
0040: 40 10 0f 01 04 80 a9 fe   @....€©þ
0048: 00 10 ec 03 00 00 71 5d   ..ì...q]
0050: 40 10 ec 03 00 00 d2 55   @.ì...ÒU
0058: 60 20 0f 01 04 80 00 80   ` ...€.€
0060: 6f 67 0f 01 04 80 f1 5e   og...€ñ^
0068: 40 10 0f 01 04 80 1c 41   @....€.A
0070: 00 10 0f 01 04 80 49 9a   .....€Iš
0078: 00 10 0f 01 04 80 49 da   .....€IÚ
0080: 40 10 0f 01 04 80 1c 6c   @....€.l
0088: 00 10 94 00 00 00 1c 5c   ..”....\
0090: 00 10 0f 01 04 80 b9 8e   .....€¹Ž
0098: 00 10 0f 01 04 80 d3 15   .....€Ó.
00a0: 40 10 0f 01 04 80 a9 fe   @....€©þ
00a8: 00 10 0f 01 04 80 d2 55   .....€ÒU
00b0: 60 20 0f 01 04 80 40 00   ` ...€@.
00b8: 8f 66 0f 01 04 80 d2 55   f...€ÒU
00c0: 60 20 0f 01 04 80 02 01   ` ...€..
00c8: 80 38 0f 01 04 80 ac 1a   €8...€¬.
00d0: 40 10 0f 01 04 80 d2 55   @....€ÒU
00d8: 60 20 0f 01 04 80 02 01   ` ...€..
00e0: 80 38 0f 01 04 80 ac 1a   €8...€¬.
00e8: 40 10 0f 01 04 80 1c 41   @....€.A
00f0: 00 10 0f 01 04 80 1c 41   .....€.A
00f8: 00 10 0f 01 04 80 49 9a   .....€Iš
0100: 00 10 0f 01 04 80 49 da   .....€IÚ
0108: 40 10 0f 01 04 80 1c 6c   @....€.l
0110: 00 10 0f 01 04 80 1c 5c   .....€.\
0118: 00 10 0f 01 04 80 b9 8e   .....€¹Ž
0120: 00 10 0f 01 04 80 d3 15   .....€Ó.
0128: 40 10 0f 01 04 80 a9 fe   @....€©þ
0130: 00 10 0f 01 04 80 1c 79   .....€.y
0138: 00 10 0f 01 04 80 49 9a   .....€Iš
0140: 00 10 0f 01 04 80 49 da   .....€IÚ
0148: 40 10 0f 01 04 80 1c 6c   @....€.l
0150: 00 10 0f 01 04 80 1c 5c   .....€.\
0158: 00 10 0f 01 04 80 b9 8e   .....€¹Ž
0160: 00 10 0f 01 04 80 d3 15   .....€Ó.
0168: 40 10 0f 01 04 80 a9 fe   @....€©þ
0170: 00 10 0f 01 04 80 1c 41   .....€.A
0178: 00 10 0f 01 04 80 49 9a   .....€Iš
0180: 00 10 0f 01 04 80 49 da   .....€IÚ
0188: 40 10 0f 01 04 80 1c 6c   @....€.l
0190: 00 10 0f 01 04 80 1c 5c   .....€.\
0198: 00 10 0f 01 04 80 b9 8e   .....€¹Ž
01a0: 00 10 0f 01 04 80 d3 15   .....€Ó.
01a8: 40 10 0f 01 04 80 a9 fe   @....€©þ
01b0: 00 10 0f 01 04 80 b4 34   .....€´4
01b8: 40 10 bd f9 ff ff b4 34   @.½ùÿÿ´4
01c0: 40 10 bd f9 ff ff c0 1c   @.½ùÿÿÀ.
01c8: 40 10 0f 01 04 80 c8 10   @....€È.
01d0: 40 10 0f 01 04 80 c1 1d   @....€Á.
01d8: 40 10 0f 01 04 80 48 16   @....€H.
01e0: 40 10 0f 01 04 80 1c 41   @....€.A
01e8: 00 10 0f 01 04 80 49 9a   .....€Iš
01f0: 00 10 0f 01 04 80 49 da   .....€IÚ
01f8: 40 10 0f 01 04 80 1c 6c   @....€.l
0200: 00 10 0f 01 04 80 1c 5c   .....€.\
0208: 00 10 0f 01 04 80 b9 8e   .....€¹Ž
0210: 00 10 0f 01 04 80 d3 15   .....€Ó.
0218: 40 10 0f 01 04 80 a9 fe   @....€©þ
0220: 00 10 0f 01 04 80 d2 55   .....€ÒU
0228: 60 20 0f 01 04 80 40 00   ` ...€@.
0230: 8f 66 0f 01 04 80         f...€  



Event Type:       Error
Event Source:       MSExchange Common
Event Category:       General
Event ID:       4999
Date:             7/1/2014
Time:             12:15:50 PM
User:             N/A
Computer:       
computername

Description:
Watson report about to be sent to dw20.exe for process id: 5236, with parameters: E12N, c-rtl-AMD64, 08.03.0348.001, store.exe, store.exe, 3b8c4b, c0000005, b5d1, 08.03.0348.001.  ErrorReportingEnabled: True

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.




Event Type:       Error
Event Source:       Microsoft Exchange Server
Event Category:       None
Event ID:       5000
Date:             7/1/2014
Time:             12:15:50 PM
User:             N/A
Computer:       
computername

Description:
EventType e12n, P1 c-rtl-amd64, P2 08.03.0348.001, P3 store.exe, P4 store.exe, P5 3b8c4b, P6 c0000005, P7 b5d1, P8 08.03.0348.001, P9 NIL, P10 NIL.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 65 00 31 00 32 00 6e 00   e.1.2.n.
0008: 2c 00 20 00 63 00 2d 00   ,. .c.-.
0010: 72 00 74 00 6c 00 2d 00   r.t.l.-.
0018: 61 00 6d 00 64 00 36 00   a.m.d.6.
0020: 34 00 2c 00 20 00 30 00   4.,. .0.
0028: 38 00 2e 00 30 00 33 00   8...0.3.
0030: 2e 00 30 00 33 00 34 00   ..0.3.4.
0038: 38 00 2e 00 30 00 30 00   8...0.0.
0040: 31 00 2c 00 20 00 73 00   1.,. .s.
0048: 74 00 6f 00 72 00 65 00   t.o.r.e.
0050: 2e 00 65 00 78 00 65 00   ..e.x.e.
0058: 2c 00 20 00 73 00 74 00   ,. .s.t.
0060: 6f 00 72 00 65 00 2e 00   o.r.e...
0068: 65 00 78 00 65 00 2c 00   e.x.e.,.
0070: 20 00 33 00 62 00 38 00    .3.b.8.
0078: 63 00 34 00 62 00 2c 00   c.4.b.,.
0080: 20 00 63 00 30 00 30 00    .c.0.0.
0088: 30 00 30 00 30 00 30 00   0.0.0.0.
0090: 35 00 2c 00 20 00 62 00   5.,. .b.
0098: 35 00 64 00 31 00 2c 00   5.d.1.,.
00a0: 20 00 30 00 38 00 2e 00    .0.8...
00a8: 30 00 33 00 2e 00 30 00   0.3...0.
00b0: 33 00 34 00 38 00 2e 00   3.4.8...
00b8: 30 00 30 00 31 00 2c 00   0.0.1.,.
00c0: 20 00 4e 00 49 00 4c 00    .N.I.L.
00c8: 20 00 4e 00 49 00 4c 00    .N.I.L.
00d0: 0d 00 0a 00               ....    





Event Type:       Error
Event Source:       Application Error
Event Category:       (100)
Event ID:       1000
Date:             7/1/2014
Time:             12:15:56 PM
User:             N/A
Computer:       
computername
Description:
Faulting application store.exe, version 8.3.348.1, faulting module store.exe, version 8.3.348.1, fault address 0x00000000003b8c4b.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74   Applicat
0008: 69 6f 6e 20 46 61 69 6c   ion Fail
0010: 75 72 65 20 20 73 74 6f   ure  sto
0018: 72 65 2e 65 78 65 20 38   re.exe 8
0020: 2e 33 2e 33 34 38 2e 31   .3.348.1
0028: 20 69 6e 20 73 74 6f 72    in stor
0030: 65 2e 65 78 65 20 38 2e   e.exe 8.
0038: 33 2e 33 34 38 2e 31 20   3.348.1
0040: 61 74 20 6f 66 66 73 65   at offse
0048: 74 20 30 30 30 30 30 30   t 000000
0050: 30 30 30 30 33 62 38 63   00003b8c
0058: 34 62                     4b      




To resolve this issue

===============

researching on issue, suggests you install Latest update and since Windows Server is x64 bit so /4GB switch is also not an option to check. 


Found no Antivirus or any third party device except Storage driver and NIC driver


EXBPA is also not helping indicating towards any issues.


Increased the diagnostic logging on MsExchange Store to expert from EMC, alternatively you can also increased the diagnostic logging from Registry key or from EMS.


From Shell

Set-EventlogLevel "servername\MsExchange Store Driver" -Level Expert

I stopped all the Exchange related service on the server and restarted the Information store, It's not crashing so i have to find what could be causing the crash.

Since the only Exchange related service is running is "information store" so thought of starting Transport Service

Now, Started Transport service, MSExchange store Crashed, so changed the "Data" folder under Transport Roles, Location could be "Microsoft\Exchange server\Transport Roles\Data\Queue" so here i renamed the Data folder to "Old Data" and restart Transport service, no issues now, MsExchange Information Store is not crashing


Gradually started all the services and information store is not crashing.





3 comments:

  1. Thank you, thank you, thank you!! This just started on my server and I was going nuts trying to figure out what was causing it. Thank you again for this!!

    ReplyDelete