By default, Windows NT calculates the optimum memory assigned to each NTVDM (Windows NT Virtual DOS Machine). If this algorithm is published, I haven't found it.

You can set the amount of memory allocated for each NTVDM by using Regedt32 to navigate to:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\WOW

The size value name in the right hand pane is a string value (REG_SZ) that defaults to 0.

You can set it to the number of megabytes you want allocated to each NTVDM.