If you have SP4(+) installed, and all kernel memory is being consumed, it may be due to the default value of MaxForwardBufferMemory, which is 0xFFFFFFFF.
This entry allows the forward buffer that TCP/IP uses to route packets between interfaces to grow to use all kernel memory.
Use Regedt32 to navigate to:
On the Edit menu, Add Value name MaxForwardBufferMemory as a type REG_DWORD. In the DWORD Editor, set the data, using the Decimal Radix, to 74240. The Range is 0 - 0xFFFFFFFF bytes, and must be a multiple of 256 bytes. The number must be GEQ (Greater Than or EQual To) ForwardBufferMemory, which defaults to 74,240 (hex 0x12200).
ForwardBufferMemory, a type REG_DWORD entry, specifies the initial size of the buffer to store routed packets. If the IP router is not enabled, or no buffers are allocated, this value is ignored. The 74,240 default can hold fifty packets. The size specified must be a multiple of 256. When the buffer exceeds MaxForwardBufferMemory, packets are discarded at random. Header packets are stored sperately.
NumForwardPackets, a type REG_DWORD, defaults to 50, and determines the initial number of headers in the router packet que. The value should be no larger than ForwardBufferMemory / 256 and at least ForwardBufferMemory / Max IP Data Size. The optimal number is somewhere between these two numbers. When the number of headers exceed MaxNumForwardPackets, they are discarded at random.
MaxNumForwardPackets, a type REG_DWORD, defaults to 0xFFFFFFFF. It must be GEQ NumForwardPackets.
While you can increase these numbers, they will allocate more system resources and impact system performance.