QPI: The QEMM-386 Programming Interface --------------------------------------- Introduction by Andrew Schulman: I had some reservations about this month's Undocumented Corner by Ralf Brown on the private programming interface provided by Quarterdeck's 386 memory manager, QEMM. While Ralf has done an admirable job of tracking down and describing this undocumented interface, I am unsure of its longevity. The future is uncertain for third-party memory managers such as Quarterdeck's QEMM and Qualitas's 386MAX. Quarterdeck appears to have squandered valuable resources on DesqView/X, and Qualitas appears to have avoided Windows for too long. On the other hand, Helix Software (makers of NetRoom) does seem with its Cloaking API to have technology of lasting value. But whatever the future of third-party memory managers such as QEMM, Ralf's description of the QEMM programming interface remains fascinating. For example, take a look at the output from Ralf's QEMMINFO program, shown in Figure 3. The three maps displayed by this program show the arrangement of the first megabyte (plus a smidgeon) of memory. Even though 386 machines, and with them Virtual-8086 (V86) mode, have been around for years, it's a sad fact that, even today, many PC programmers would be surprised to hear that the first megabyte of memory on a PC even has an "arrangement." If you're sitting at the DOS C:\> prompt, the first megabyte is the first megabyte, right? No, not if (like most users today) you're using a 386 memory manager such as QEMM or 386MAX, and/or are running in a DOS box under Windows Enhanced mode. For example, in Figure 3 bits of the first megabyte of linear memory actually belong to the fourth megabyte of physical memory. Of course, 386 memory managers try to make V86 mode as invisible as possible. However, it's often necessary (or at least helpful) for programmers to view the V86 mode reality behind the "it looks like real-mode DOS" facade. Ralf's article shows how to do this for QEMM. Some of these QEMM APIs are also partially implemented (or "spoofed," as Ralf puts it) by other memory managers, so this information is more widely applicable than might at first appear. Interest and space permitting, future articles will discuss the 386MAX API. An upcoming article by taQ {{get full name}} will reveal the Global EMM Import interface that Windows uses to dig beneath the surface of 386 memory managers. Besides describing how to use the QEMM interface, Ralf also presents some fascinating background information. For example, he shows how Compaq's original 386 memory manager, CEMM, is the basis for today's EMM386 and QEMM. As another interesting point, Ralf touches briefly on how QEMM _patches_ Windows. I also enjoyed his explanation of how V86 managers can hook interrupts and establish interfaces (by hooking I/O ports, for example) in ways that are likely to surprise those programmers who still think of DOS as a real-mode operating system. Make no mistake, when something like QEMM is loaded, DOS isn't running in real mode, so anything is possible. Other Undocumented Corners in the pipeline include a much-awaited article by Troy Folger on undocumented OS/2 (particularly the DosQProcStatus function), and one by Roger Alley on the inner workings of Windows listbox controls. Please send your comments and suggestions via the Undocumented Corner area in the Dr. Dobb's forum (GO DDJFORUM) on CompuServe; my CIS ID is 76320,302. ;---------------------------------------------------------------------- Ralf Brown maintains the MS-DOS Interrupt List, an enormous free collection of information about interrupt calls. He coauthored _Undocumented DOS_, as well as _PC Interrupts_ and _Network Interrupts_ (both based on the interrupt list), and is currently a Postdoctoral Fellow at Carnegie Mellon University's Center for Machine Translation. His electronic alter ego is ralf@telerama.lm.com. ;---------------------------------------------------------------------- by Ralf Brown Memory managers support a variety of industry-standard interfaces which have been developed over the years, such as EMS, XMS, VCPI, DPMI, and VDS. But any programmer who has used the utility programs included with memory managers such as Quarterdeck's QEMM-386 or Qualitas' 386MAX knows that there must be a way to retrieve information from and control these managers which goes beyond what is available through the standard interfaces such as EMS and XMS. How else, for example, could QEMM.COM or Manifest determine how much memory QEMM is using for its own code or mapped ROM? The answer is that QEMM, 386MAX, and Helix Software Netroom's RM386 all support extensive private APIs; Compaq's CEMM and Microsoft's EMM386 also have smaller private APIs. These five memory managers use three quite different methods of invoking their private functions. RM386 provides direct interrupt-based calls; QEMM, CEMM, and EMM386 have a FAR CALL entry point whose address may be determined in a number of ways, including interrupt calls; and 386MAX uses the 386's ability to trap access to a "magic" I/O port and transfer control to the V86 mode supervisor, 386MAX.SYS. Given the sizes of the APIs involved, and the fact that the Netroom API is almost entirely documented (rather a rarity these days), this column will focus on QEMM. Through a FAR CALL entry point, the QPI provides functions to change QEMM's state, provide statistics, control memory mapping and video virtualization, and support coexistence with Microsoft Windows. Various pieces of the QEMM private API have been unofficially known for years, in large part due to the author's efforts. In June 1993, Quarterdeck finally released some official documentation on what it calls the QEMM-386 Programming Interface (QPI); however, the great majority of this interface is still undocumented. Finding the QPI Entry Point --------------------------- Since QPI is based on a FAR CALL entry point, the first step is to determine the entry point. There are at least four methods for determining QEMM's private entry point in recent versions, two of which were officially (though obscurely) documented in two June 1993 Quarterdeck files called QDMEM.DOC and QPI.DOC (see Further Reading, below). The four known methods are scanning for a signature string, using INT 67h AH=3Fh, using an IOCTL call, and using Quarterdeck's RPCI (Resident Program Communication Interface); the last two have been documented. These four methods have accumulated over the numerous revisions of QEMM, and it is likely that the ones which were not documented in Quarterdeck's QPI files are present primarily for backwards compatibility. Scanning for a signature involves looking for the string "QUARTERDECK EXPANDED MEMORY MANAGER 386", which is located at offset 14h in the EMMXXXX0 (expanded memory manager) device driver's segment and preceded by a WORD containing the entry point's offset in the driver's segment. Prior to QEMM 7.0, this device driver code -- and thus the signature string -- was always located in low memory. Beginning with 7.00, this code can be relocated into upper memory, though in 7.01, copies of the EMMXXXX0 driver code are present in both low memory and an upper memory block under some circumstances (the low-memory copy was apparently completely unused and simply not freed at startup). This method of retrieving the entry point is probably used only to verify the entry point returned by the next method, since software scanning memory for the signature would probably not have considered the possibility that it would wind up in high memory with the release of QEMM 7, and would thus have been "broken" by the new version. An important point about the signature method is that both Compaq's CEMM and Microsoft's EMM386 also support the same method of getting their own private entry points, using the signatures "COMPAQ EXPANDED MEMORY MANAGER 386" and "MICROSOFT EXPANDED MEMORY MANAGER 386". The great similarity between the CEMM and EMM386 private APIs indicates that EMM386 is most likely a direct descendant of CEMM. There are also sufficient similarities between CEMM and QEMM to hint that QEMM is derived from Compaq's memory manager as well. All three memory managers have identical functions 00h and 01h in their private APIs. The second method is the simplest, but has also been (incompletely and not entirely correctly) copied by at least two other memory managers. To check for QEMM's presence and simultaneously retrieve the entry point, load AH with 3Fh, CX with 5145h ('QE'), and DX with 4D4Dh ('MM'), then invoke INT 67h. On return, AH will be 00h if the call was successful (QEMM or one of the "spoofing" managers is installed), and ES:DI will contain the address of the entry point. However, both Micronic's MICEMM and 386MAX provide only a few of the functions on this entry point that QEMM does. One way to distinguish between the real QEMM and these other memory managers is to test for the signature string at offset 14h in the entry point's segment; however, MICEMM will provide this same signature if it has been given the "DV" commandline switch. Beginning with version 5.0, Quarterdeck introduced a new interface shared by a number of its resident programs, including QEMM, QRAM, VIDRAM, and Manifest in resident mode. The Resident Program Communication Interface (RPCI) uses INT 2Fh with a dynamically-set function number between C0h and FFh, defaulting to D2h. All RPCI programs will share this same multiplex number. To find the RPCI multiplex number, scan AH values from D2h to FFh and then C0h to D1h, calling INT 2Fh with AX=XX00h, BX=5144h ('QD'), CX=4D45h ('ME'), and DX=4D30h ('M0'). On return, AL will be FFh if the multiplex number is in use, and (if it is the RPCI rather than some other program) BX=4D45h ('ME'), CX=4D44h ('MD'), and DX=5652h ('VR'). Armed with the multiplex number, check for QEMM by calling INT 2Fh with AH=multiplex number, AL=01h, BX=5145h ('QE'), CX=4D4Dh ('MM'), and DX=3432h ('42'); if QEMM is present, it returns BX=4F4Bh ('OK') and sets ES:DI to the address of the entry point. The final detection method was also added in version 5.0, though Quarterdeck's QPI.DOC claims that it is only available starting in 6.0. Open the character device QEMM386$ with INT 21h AX=3D00h and perform an IOCTL INPUT (INT 21h AX=4402h) of four bytes. The four returned bytes are a FAR pointer to the QPI entry point. Since both the RPCI and IOCTL methods have been officially (though admittedly obscurely) documented, they should be the preferred methods for retrieving the QPI entry point, rather than the older and still undocumented signature and INT 67h methods. The QPI Functions ----------------- Having found the private entry point, QEMM's private functions may be invoked by loading the function number into AH, the subfunction (if any) into AL, setting any other required registers, and calling the far entry point. On return, CF indicates if function was successful. Interestingly, using a debugger to examine the actual entry point for QEMM prior to 7.0 reveals nothing more than an INT 2Ch instruction followed by an IRET (in versions 7.x, there is some additional indirection before the INT 2Ch). But looking at INT 2Ch reveals nothing of QEMM -- by default, MS-DOS points it at an IRET instruction. How then could this possibly by QEMM's interface? Quite simple: on any hardware or software interrupt (such as this INT 2Ch) in V86 mode, the CPU switches to protected mode and calls a protected-mode interrupt handler rather than the handler pointed at by the real-mode interrupt vector table at 0000:0000. So QEMM gets to see all interrupts before applications do, and can filter out those meant for it instead of passing them down to the real-mode handler. INT 2Ch and most of the other interrupts in the range 22h to 30h, if issued from the QEMM386$ driver's segment, are meant for QEMM and will provide various functions to QEMM's real-mode stub; if issued from any other segment, they are passed back down to the "real-mode" (actually V86 mode) handler. (Thus, you can't call QPI simply by putting an INT 2Ch in your own code.) INT 2Ch happens to be the QPI provider and has remained stable over many versions of QEMM; others have varied, particularly in the actual interrupt number used for the particular service, but include EMS, XMS, VDS, and INT 15h AH=87h services, as well as some internal calls. Figure 1 provides an overview of the QPI functions, with the few calls that are officially documented marked as such. Except for Function 1Dh, each new version of QEMM has supported all of the functions supported by all previous versions since 4.23 (the earliest about which information is available). The QPI functions may be roughly classified as follows: * Changing QEMM's state: 00h, 01h, 04h, and 05h * Statistics: 11h, 16h, and 17h * Memory mapping: 06h to 0Bh, 0Fh, 18h, and 1Fh * Stealth: 1Dh, 1Eh, 21h, 24h * Video virtualization: 0Dh, 0Eh, 13h's subfunctions * Coexistence with MS Windows: 1Bh's subfunctions * DESQview support: 1306h, 14h, 1Ch, and 22h * VCPI functionality: 0Ch and 10h's subfunctions * Miscellaneous: 02h, 03h, 12h, 15h, 1Ah, 20h Although the ordering is different, Function 0Ch and the various subfunctions of Function 10h provide exactly the same calls as the Virtual Control Program Interface (VCPI), and are thus clearly the precursor of the public specification. In fact, QEMM implements both the VCPI calls on INT 67h AH=DEh and the QPI functions 10xxh with calls to the same underlying subroutines. This situation is analogous to the development of the DPMI specification, which in many ways is merely a description of preexisting functionality in the Windows VMM. Function 1Ah provides access to I/O ports which bypasses any protections or virtualization QEMM may have imposed on the I/O port the program wishes to use. Other functions, such as some of the 13xxh calls and function 15h, can affect which I/O ports are virtualized by QEMM. It has not yet been determined how (if at all) one can get access to QEMM's I/O permission bitmap to determine which ports are being virtualized, but ports 60h, 64h, 92h, and various VGA ports are normally virtualized by QEMM. Functions 1Dh, 1Eh, 21h, and 24h support QEMM's Stealth feature, which lets it provide more upper memory by hiding the system's ROMs. Stealth remaps memory so that the ROMs appear in the first megabyte only when they are actually required, namely during an interrupt call which reaches a handler in ROM. When QEMM starts up with Stealth enabled, it hooks all interrupts which point into ROM in order to intercept calls just before they are chained to a ROM. This technique allows Stealth to work with the existing ROMs, in contrast to Helix Software's Cloaking or Novell's DPMS (DOS Protected Mode Services). The latter require software written specifically to their interface (which involves a small stub in the first megabyte and the true handler running in protected mode), but in exchange offer the ability to move arbitrary resident programs out of the "real-mode" first megabyte. As noted earlier, QEMM provides some functions specifically for use by DESQview (in combination with which it creates the DESQview/386 multitasker). The close interaction between QEMM and DESQview can be seen especially well in Function 14h, which supports DESQview's "protection level" feature. A non-zero protection level for a program enables additional checks which can catch many errant programs before they can cause a system-wide crash. These functions are not usable by other applications because QEMM makes various DESQview API calls (INT 15h AH=10h-12h) when non-zero protection levels are in effect; in particular, QEMM assumes that it can pop up a DESQview error message window when it detects a protection violation. Naturally, the conventional-memory stub of QEMM386.SYS also uses QPI calls. QEMM uses functions 00h and 01h to either temporarily or permanently change its state, such as turning itself off when a laptop goes into sleep mode and then returning to its former state when the laptop is restarted. The code supporting the DISKBUF (DB) switch uses function 18h to determine whether there is any need to copy the data being transferred to or from the disk through a fixed buffer allocated by QEMM; if the logical address is identical to the physical address for every byte in the buffer being used by the application, there is no need for the temporary buffer. QEMM v6.0x used functions 1D00h and 1D01h in supporting the suspend/resume interrupt feature of many laptops. All seven of the 1Bxxh functions are used in one way or another while operating with MS Windows: * Function 1B00h returns the address of the Global EMM Import Structure (see below) * Functions 1B01h and 1B02h implement the Windows V86-mode enable/disable callback provided through INT 2Fh AX=1605h * Functions 1B03h and 1B04h are used by QEMM's conventional-memory stub to notify QEMM's protected-mode code that Windows is starting or terminating * Functions 1B05h and 1B06h are used in patching some of Windows' drivers as they are loaded into memory QEMM's patching allows QEMM to run Windows in all three modes: real, standard, and enhanced. In particular, QEMM versions 6 and 7 patch all *.DRV files loaded by Windows 3.0 standard mode before GDI.EXE is loaded, looking for an SMSW instruction followed by a TEST instruction on the destination of the SMSW (used to determine whether the machine is already in protected or V86 mode). They patch this sequence into a MOV from CS followed by a VERW, both using the same destination as the original instruction sequence, which has the effect that the protected-mode check will trigger only in true protected mode and not in virtual-86 mode (as it would with the SMSW/TEST sequence). The previous section mentioned that two other memory managers provide the INT 67h AH=3Fh call to get the QPI entry point, but provide only a subset of the QPI functions. MICEMM provides only functions 00h, 02h, and 03h; 386MAX 6 provides only function 0Ch and the various subfunctions of function 10h (interestingly, these are precisely the functions which later became the VCPI specification on INT 67h AH=DEh). The problem with 386MAX's implementation is that the few supported functions use a nonzero return value in AH instead of the carry flag to signal an error or unsupported function. Other Undocumented Functions ---------------------------- The QPI just described is not the full extent of QEMM's private API. QEMM provides an additional (documented) RPCI function beyond the two already shown. Similar to QPI function 12h, calling INT 2Fh with AH=multiplex number, AL=01h, BX=4849h ('HI'), CX=5241h ('RA'), and DX=4D30h ('M0') will return BX=4F4Bh ('OK') if high memory is present and will set both CX and DX. CX contains the segment of the first memory control block in the high-memory chain, and DX contains the segment of the owner of any locked-out memory blocks (i.e., video or ROMs between the regions of upper memory). In existing versions of QEMM, the value in DX is always the segment of the QEMM386$ device driver code. Unlike QPI function 12h, this call is also supported by Quarterdeck's QRAM, a memory manager for sub-386 machines which can use shadow RAM as upper memory blocks. Quarterdeck's high memory chain is identical to the DOS 4+ memory chain in low memory, with the owner field in the memory control block set to the string "UMB" for XMS upper memory blocks and the program name for programs and their environments loaded with LOADHI. Just as with DOS's memory chain, the first byte of each memory control block but the last is 4Dh ('M'), while the last one is 5Ah ('Z'). Another function is provided for MS Windows compatibility (and is thus also supported by EMM386, CEMM, and probably other memory managers). This is an IOCTL call on the character device EMMXXXX0, the actual EMS driver. EMM386 and CEMM support multiple subfunctions, but QEMM only supports the one needed to coexist with Windows: subfunction 01h, "Get EMM Import Structure Address." To use this function, Windows opens the device "EMMXXXX0" to get a file handle, then calls INT 21h with AX=4402h, BX=file handle, CX=0006h, and DS:DX pointing at a six-byte buffer whose first byte has been set to 01h. On return, CF will be clear if the call was successful, and the buffer will have been filled as shown in Figure 2. Bugs ---- Various versions of QEMM contain errors in range checks on function numbers which cause attempts to call some unimplemented functions to jump to random locations, generally causing a system crash. Versions 5.11 and 6.00, for instance, will accept INT 4Bh Virtual DMA Specification (VDS) calls with AX=810Dh even though the highest supported subfunction is 0Ch. Some Useful Undocumented Functions ---------------------------------- Not surprisingly, the functions which have been officially documented are those which are most critical for proper coexistence with QEMM's advanced features such as Stealth. Even so, there are a number of other functions which can come in handy. Function 18h (already mentioned in the context of the DISKBUF switch), for example, can tell a program whether it is safe to use DMA directly to a particular buffer. If this function indicates that the specified region of the program's address space is entirely in conventional memory, then the physical addresses needed for DMA are the same as the logical linear addresses the program sees, and the DMA controller can be used without going through VDS to allocate a buffer and copy data to and from that buffer. The memory allocated to an EMS handle may be made visible in the program's address space using Functions 0Bh and 0Fh. A program might thus make 128K of EMS visible at a time, with the limitation that no single EMS handle can be allocated more memory than the size of the address range into which the memory is mapped. This is possibly how DESQview virtualizes CGA graphics: allocate some EMS and map it into the video memory space. These two functions, or Functions 0Ah or 1F01h (both of which change the mapping for a single 4K page), could also map in the bulk of the memory required by a TSR, allowing a very small stub in the one-megabyte 8086 address space which maps in the remainder of the TSR as needed. The main TSR code would then be physically located in extended memory which can be made visible anywhere it is desired, for example on top of video memory. Some care is necessary, however, to properly preserve the prior page mappings; this is a particular problem when using Functions 0Bh and 0Fh, since Function 0Fh will undo any mappings that might have existed in the affected area before Function 0Bh was used. A Sample Program ---------------- To show how to use QPI, I have written an information-reporting utility. Like Quarterdeck's own QEMM.COM and Manifest, QEMMINFO displays maps of the memory types and which pages of memory have been accessed, plus other information. The QEMMINFO display (see Figure 3) consists of four columns. The first contains a map of the memory type for each 4K page in the first megabyte. This map is the same as the one generated by QEMM.COM or Manifest, and indicates which pages are conventional memory, mappable, high RAM, video memory, excluded, etc. The second column displays which pages have been accessed or modified. This map is an extension of the one displayed by QEMM.COM or Manifest, since it also shows the access status of the sixteen pages making up the High Memory Area (HMA). The display of the first megabyte uses the QPI function 1600h provided for that purpose, but the HMA display extracts the access bits from the page table entries for the HMA pages. The third QEMMINFO column dislays a map which neither QEMM.COM nor Manifest can generate -- the translations between the V86 mode addresses and physical memory addresses. For each 4K page in the first 1088K (one megabyte plus HMA) of linear address space, QEMMINFO shows which megabyte of physical memory actually appears in that page. This display is created by reading the page number for each of the first 272 pages (1088K) in the current V86 address space and converting the page number into a multiple of one megabyte. A value of zero indicates that the page shows memory from the first megabyte, and is thus conventional memory (except in very unusual cases, the physical address is the same as the logical address). Figure 3 was generated from a 512K DOS window under DESQview, and clearly shows how segments 0400h to 87FFh have been mapped to a block of EMS memory in megabytes three and four, while the 96K from 8800h to 9FFFh, which are not part of the DOS window, have not been remapped. The author's machine has 384K of "top" memory just below the 16M mark (as do many Compaq systems), and this memory is used to provide UMBs and shadow RAM, appearing as 'F' in the QEMMINFO display. Below the three maps, a few additional bits of information are shown. For example, the VHDIRQ setting affects background disk accesses by many advanced disk caches with delayed writes; this item will typically report that the setting is ignored when Stealth is disabled and respected when Stealth is active. QEMMINFO's memory-mapping display can be used to show that QEMM doesn't actually enable or disable the A20 line, but merely remaps memory to simulate the address wrapping due to A20. When A20 is open (which it will always be the case when DOS=HIGH), QEMMINFO will show that the HMA is mapped to megabyte 1, while when it is closed, QEMMINFO shows that the HMA is mapped to megabyte 0. One of QEMMINFO's options is to clear the memory-access flags, just like QEMM RESET. The QEMMINFO RESET option will also reset the access flags for the HMA, which QEMM RESET won't do. QEMMINFO also allows you to selectively clear either read or write flags as well as both flags for each page; QEMM RESET always clears both flags. The memory-access flags are only of informational interest in this environment, since QEMM does not perform demand paging, which the flags were designed to support (pages marked as written need to be stored on disk before being replaced, and unaccessed pages would normally be replaced first). Listing 1 shows QPICALL.ASM, which forms the core of QEMMINFO. This module exports the C-callable function QPIcall, which invokes QEMM's private API in the same way the int86 function permits C code to call software interrupts. QEMM.C builds more than three dozen "glue" functions around QPIcall to provide access to most of the QPI, and QEMMINFO.C in turn builds upon the functions provided by QEMM.C. The combination of QPICALL.ASM and QEMM.C can be used as a generic function library for calling QEMM functions, and is independent of the sample program QEMMINFO. The full listings are available electronically (see page xxx). The complete calling information known for the private API functions is provided in QEMMINTS.LST, also available electronically. Wrapping Up ----------- Though the makers of memory managers have tried hard to make the V86 mode under which programs operate while the memory manager is active behave just like true real mode, it is not possible (and in many cases not practical) to operate exactly as in real mode. For example, VDS was created to deal with the problem that logical and physical addresses are no longer the same when running under a memory manager. Although the memory manager could virtualize the DMA controller, that would not help bus-mastering cards such as many SCSI host adapters; a set of services which allow aware software to interact with the memory manager is far superior because it can be applied to any hardware, not just that to which the memory manager's programmers have ready access. QEMM's private calls similarly allow QEMM-aware programs to accomplish things that would be possible in real mode but are not otherwise possible in V86 mode under QEMM, such as access to I/O ports without the interference of virtualization. Further Reading --------------- Ralf Brown, ed. INTER40x.ZIP, "MS-DOS Interrupt List", Release 40, 4/3/94. Ralf Brown and Jim Kyle, _PC Interrupts_, Second edition. Addison-Wesley 1994. ISBN 0-201-62485-0. Quarterdeck Office Systems, Technical Note QDMEM.DOC, "Quarterdeck Memory Driver Interface", and QPI.DOC, "QEMM-386 Programming Interface", June 15, 1993. Available on the Quarterdeck BBS at 310-314-3227 in QPI.ZIP. {{Key for Figure 3: . mappable RAM M mapped ROM H high RAM X excluded memory V video R ROM A adapter / split ROM (2K ROM,2K RAM) f page frame r rammable C conventional }}