By default, Windows NT calculates the optimum memory assigned to each NTVDM (Windows NT Virtual DOS Machine). If this algorithm is published, I haven't found it.
You can set the amount of memory allocated for each NTVDM by using Regedt32 to navigate to:
The size value name in the right hand pane is a string value (REG_SZ) that defaults to 0.
You can set it to the number of megabytes you want allocated to each NTVDM.