The installation for GpsTools SDK and CoordTrans is using an older version of InstallShield which needs to run as a 16-bit application. You get the error because you are missing some resources on you computer. Note that this is a limitation in the installation program only and not in GpsTools (or CoordTrans) itself.
You can read more about it here in the Microsoft homepage:
And a direct qoute from microsoft.com
Start Quote -->
When the 16-bit Windows or Windows on Windows (WOW) subsystem starts, the following error message is displayed:
NOTE: The WOW subsystem may be started automatically when you start Windows NT or the first time you run a 16-bit Windows-based application.
The Win16 Subsystem was unable to enter Protected Mode, DOSX.EXE must be in your AUTOEXEC.NT and present in your PATH.
One or more of the following conditions exists in Microsoft Windows NT or Windows NT Advanced Server:
• HIMEM.SYS or DOSX.EXE is missing or corrupt in the SYSTEM32 directory.
• HIMEM.SYS is not being loaded in the CONFIG.NT file.
• DOSX.EXE is not being loaded in the AUTOEXEC.NT file.
Make sure that the line, "device=%SystemRoot%\system32\himem.sys," exists in the CONFIG.NT file.
Make sure that the line, "lh %SystemRoot%\system32\dosx.exe," exists in the AUTOEXEC.NT file.
Expand HIMEM.SYS, DOSX.EXE, or both from the Windows NT setup disks or CD- ROM disc if either or both are missing from the SYSTEM32 directory.
For more information on DOSX.EXE, query on the following keywords in the Microsoft Knowledge Base:
<-- End Quote