간헐적인 터미널 서비스 연결 실패, application hang 이 빈번하게 발생하여 사용자 컨트롤에 따른 시스템 반응이 느린 성능 문제로 추정되는 현상이 발생하고 있는 상황에서 관리자에 의해 키보드를 통한 강제 메모리 덤프를 수집하였습니다. 어떤 원인이 있었기에 이러한 현상이 발생한 것일까요?
[환경]
Windows Server 2003 SP2
[현상]
간헐적인 터미널 서비스 연결 실패, 성능 문제로 추정되는 어플리케이션 실행 속도 지연 및 hang 발생 상황에서 키보드를 통한 강제 메모리 덤프 수집
[원인]
커널 메모리 리소스 Paged Pool 최대값 270MB 중 268MB 를 사용하고 있습니다. 그 중 secuengine.sys 에서 사용하는 Pool Tag 인 BcMc 가 Paged Pool 커널 메모리가 가장 사용하고 있는 것을 확인할 수 있습니다. 자세한 원인 분석을 위해서는 해당 드라이버 공급사에 문의하여 추가 분석이 요구됩니다.
[해결방안]
secuengine.sys 드라이버 공급자에 문의하여 추가 확인이 필요합니다.
[분석결과]
memory.dmp 분석
2: kd> !vm 1
*** Virtual Memory Usage ***
Physical Memory: 522710 ( 2090840 Kb)
Page File: \??\C:\pagefile.sys
Current: 2095104 Kb Free Space: 1414196 Kb
Minimum: 2095104 Kb Maximum: 4190208 Kb
Page File: \??\D:\pagefile.sys
Current: 4177920 Kb Free Space: 3609764 Kb
Minimum: 4177920 Kb Maximum: 4194304 Kb
Available Pages: 4414 ( 17656 Kb)
ResAvail Pages: 18127 ( 72508 Kb)
Locked IO Pages: 141 ( 564 Kb)
Free System PTEs: 205957 ( 823828 Kb)
******* 52836 system cache map requests have failed ******
Free NP PTEs: 28710 ( 114840 Kb)
Free Special NP: 0 ( 0 Kb)
Modified Pages: 96 ( 384 Kb)
Modified PF Pages: 96 ( 384 Kb)
NonPagedPool Usage: 35923 ( 143692 Kb)
NonPagedPool Max: 65030 ( 260120 Kb)
PagedPool 0 Usage: 14998 ( 59992 Kb)
PagedPool 1 Usage: 13010 ( 52040 Kb)
PagedPool 2 Usage: 13049 ( 52196 Kb)
PagedPool 3 Usage: 12929 ( 51716 Kb)
PagedPool 4 Usage: 13157 ( 52628 Kb)
********** Excessive Paged Pool Usage *****
PagedPool Usage: 67143 ( 268572 Kb)
PagedPool Maximum: 67584 ( 270336 Kb)
********** 48293 pool allocations have failed **********
2: kd> !poolused /t 3 4
Sorting by Paged Pool Consumed
Pool Used:
NonPaged Paged
Tag Allocs Used Allocs Used
BcMc 9 576 183060 137299064 UNKNOWN pooltag 'BcMc', please update pooltag.txt
NtFI 0 0 184907 30259640 IndexSup.c , Binary: ntfs.sys
Toke 0 0 46526 27595432 Token objects , Binary: nt!se
TOTAL 553253 142290440 943521 269647176
2: kd> !for_each_module s-a @#Base @#End "BcMc"
809ba174 42 63 4d 63 60 67 35 f8-40 ac 2f f8 00 00 00 00 BcMc`g5.@./.....
b3cc5944 42 63 4d 63 8b 45 08 50-6a 01 ff 15 bc 0f d2 b3 BcMc.E.Pj.......
b3ccacb7 42 63 4d 63 6a 38 6a 00-ff 15 bc 0f d2 b3 89 45 BcMcj8j........E
Page a0 not present in the dump file. Type ".hh dbgerr004" for details
Page b8d20 too large to be in the dump file.
2: kd> u b3ccacb7
*** ERROR: Module load completed but symbols could not be loaded for secuengine.sys
secuengine+0x13ccb7:
b3ccacb7 42 inc edx
b3ccacb8 634d63 arpl word ptr [ebp+63h],cx
b3ccacbb 6a38 push 38h
b3ccacbd 6a00 push 0
b3ccacbf ff15bc0fd2b3 call dword ptr [secuengine+0x192fbc (b3d20fbc)]
b3ccacc5 8945fc mov dword ptr [ebp-4],eax
b3ccacc8 837dfc00 cmp dword ptr [ebp-4],0
b3ccaccc 7509 jne secuengine+0x13ccd7 (b3ccacd7)
2: kd> u b3cc5944
secuengine+0x137944:
b3cc5944 42 inc edx
b3cc5945 634d63 arpl word ptr [ebp+63h],cx
b3cc5948 8b4508 mov eax,dword ptr [ebp+8]
b3cc594b 50 push eax
b3cc594c 6a01 push 1
b3cc594e ff15bc0fd2b3 call dword ptr [secuengine+0x192fbc (b3d20fbc)]
b3cc5954 5d pop ebp
b3cc5955 c20400 ret 4
*** Virtual Memory Usage ***
Physical Memory: 522710 ( 2090840 Kb)
Page File: \??\C:\pagefile.sys
Current: 2095104 Kb Free Space: 1414196 Kb
Minimum: 2095104 Kb Maximum: 4190208 Kb
Page File: \??\D:\pagefile.sys
Current: 4177920 Kb Free Space: 3609764 Kb
Minimum: 4177920 Kb Maximum: 4194304 Kb
Available Pages: 4414 ( 17656 Kb)
ResAvail Pages: 18127 ( 72508 Kb)
Locked IO Pages: 141 ( 564 Kb)
Free System PTEs: 205957 ( 823828 Kb)
******* 52836 system cache map requests have failed ******
Free NP PTEs: 28710 ( 114840 Kb)
Free Special NP: 0 ( 0 Kb)
Modified Pages: 96 ( 384 Kb)
Modified PF Pages: 96 ( 384 Kb)
NonPagedPool Usage: 35923 ( 143692 Kb)
NonPagedPool Max: 65030 ( 260120 Kb)
PagedPool 0 Usage: 14998 ( 59992 Kb)
PagedPool 1 Usage: 13010 ( 52040 Kb)
PagedPool 2 Usage: 13049 ( 52196 Kb)
PagedPool 3 Usage: 12929 ( 51716 Kb)
PagedPool 4 Usage: 13157 ( 52628 Kb)
********** Excessive Paged Pool Usage *****
PagedPool Usage: 67143 ( 268572 Kb)
PagedPool Maximum: 67584 ( 270336 Kb)
********** 48293 pool allocations have failed **********
2: kd> !poolused /t 3 4
Sorting by Paged Pool Consumed
Pool Used:
NonPaged Paged
Tag Allocs Used Allocs Used
BcMc 9 576 183060 137299064 UNKNOWN pooltag 'BcMc', please update pooltag.txt
NtFI 0 0 184907 30259640 IndexSup.c , Binary: ntfs.sys
Toke 0 0 46526 27595432 Token objects , Binary: nt!se
TOTAL 553253 142290440 943521 269647176
2: kd> !for_each_module s-a @#Base @#End "BcMc"
809ba174 42 63 4d 63 60 67 35 f8-40 ac 2f f8 00 00 00 00 BcMc`g5.@./.....
b3cc5944 42 63 4d 63 8b 45 08 50-6a 01 ff 15 bc 0f d2 b3 BcMc.E.Pj.......
b3ccacb7 42 63 4d 63 6a 38 6a 00-ff 15 bc 0f d2 b3 89 45 BcMcj8j........E
Page a0 not present in the dump file. Type ".hh dbgerr004" for details
Page b8d20 too large to be in the dump file.
2: kd> u b3ccacb7
*** ERROR: Module load completed but symbols could not be loaded for secuengine.sys
secuengine+0x13ccb7:
b3ccacb7 42 inc edx
b3ccacb8 634d63 arpl word ptr [ebp+63h],cx
b3ccacbb 6a38 push 38h
b3ccacbd 6a00 push 0
b3ccacbf ff15bc0fd2b3 call dword ptr [secuengine+0x192fbc (b3d20fbc)]
b3ccacc5 8945fc mov dword ptr [ebp-4],eax
b3ccacc8 837dfc00 cmp dword ptr [ebp-4],0
b3ccaccc 7509 jne secuengine+0x13ccd7 (b3ccacd7)
2: kd> u b3cc5944
secuengine+0x137944:
b3cc5944 42 inc edx
b3cc5945 634d63 arpl word ptr [ebp+63h],cx
b3cc5948 8b4508 mov eax,dword ptr [ebp+8]
b3cc594b 50 push eax
b3cc594c 6a01 push 1
b3cc594e ff15bc0fd2b3 call dword ptr [secuengine+0x192fbc (b3d20fbc)]
b3cc5954 5d pop ebp
b3cc5955 c20400 ret 4
작성자 : Lai Go / 작성일자 : 2010.08.30
