Caustic Moment

Emulating the database and geometry facilities of AutoCAD allows removing the IGES translator to an ADS application without extensive modifications. This development may herald the end of the growth of the AutoCAD core.

by John Walker
January 1st, 1990

Ten years ago today, I was celebrating not just the dawn of the 1980's, which I hoped would be an improvement over the Souring Seventies, but also the first successful multi-user test, achieved minutes before midnight, of NOS/MT, the Unix-like operating system that seemed to be the future of Marinchip, my company at the time. What was to eventually become AutoCAD was, on that night, less than 2000 lines of undocumented code in an obscure language for an unpopular computer.

Eight years ago, I was beginning to sketch out the plans for starting a software company from the bones of Marinchip, wondering what products might carry it to success in the marketplace. Developing the CAD product, which was at that time Marinchip's flagship, was high on the list. By the summer of 1982, development of AutoCAD was underway in earnest. The very rudimentary facilities of the original system were being extended in all directions, leading to the introduction of AutoCAD in November of 1982. At that time, the program had grown to about 12,000 lines of C.

In the ensuing years, as AutoCAD has continued to evolve to meet the needs of its community of users, the AutoCAD core has grown, seemingly without bound. As of development release Z.0.64, the core source plus include files totals 239,153 lines. AutoLisp adds 22,354 lines to the pile, and ADS contributes another 17,338. MS-DOS drivers account for 107,588 lines, and non-DOS drivers 18,048 lines. AutoCAD's special C library is 13,128 lines. So, the total source code contributing to a present-day release of AutoCAD is on the order of 417,609 lines—and that doesn't count tools, regression tests, the Macintosh tugboat code, the ADI kit, the OS/2 Presentation Manager interface and DDE library, and all the other stuff I undoubtedly forgot to count.

However you add it up, this is one big, hulkin' program. Further, all our efforts in removing drivers, then devolving them upon vendors through the ADI program, and providing user-customisability through menus, scripts, and AutoLisp have merely reduced the rate of growth of the AutoCAD core, which still accounts for roughly half the source lines in the program and continues to grow with every passing day. The core is a treacherous place, ruled by strange customs and conventions, including wild constraints on memory usage, files that aren't files (unless they be ASCII), C library routines that behave in unusual ways, and the ever-nagging question “what if this causes my overlay to be reloaded?”. The lore of AutoCAD core code craftsman is unwritten, continually evolving, and very hard for an apprentice to master. All of this makes the process of extending AutoCAD to maintain and expand its market leadership increasingly difficult, painful, expensive, and time-consuming. What can be done about this?

Sharp the strife, violent the collision,
Difficult 'twill be to reach a decision.

With the advent of ADS in 1989, it at last became practical to implement extensions to AutoCAD outside the core code without incurring unacceptable penalties in efficiency, user interface, and portability. The Eagle Project, launched in July of 1989 and scheduled to ship in mid-1990, demonstrated the feasibility of this approach and moved it into the mainstream of Autodesk's plans for AutoCAD. Additional projects such as the Applied Geometry NURBS interface and the Xanadu/AutoCAD front-end are also taking this approach. None of these efforts have been easy; each one has encountered shortcomings in the ADS interface that must be remedied by core code modifications before the project can succeed. Nonetheless, it is unarguable that each of these projects will be accomplished in less time, with less effort, suffering, and destabilisation of AutoCAD than had they been integrated into the AutoCAD core in the conventional manner.

Now that ADS has matured, tempered by the demands of these projects, it's time to consider the next logical step: not just freezing the AutoCAD core as-is and concentrating development resources outside it, but actually beginning to shrink the core by removing features from it which, had ADS existed when they were originally implemented, would scarcely have been considered as candidates for core code. This hopeful season is one of dismantling empires, of devolving structures that have proven inefficient and unworkable. What better time to put ADS to its most demanding test to date: the migration of IGES from AutoCAD into an ADS application.

A Brief History Of IGES

I never wanted IGES to be in the AutoCAD core. When I began the IGES project in July of 1985, I intended that IGES would be an extra-cost application, selling for say $500, rather than being included with every copy of AutoCAD. Not only did I hope by this to raise some extra revenue to fund IGES development and support, but, recognising from the start that IGES would be a difficult and messy chore, albeit an essential one, my intention was to limit the market for IGES to those serious customers who would use it for practical purposes and generate useful requests for change, rather than frivolous users writing bug reports every time an IGESOUT/IGESIN “lost something” a DXFOUT/DXFIN preserved.

In 1985, the only alternatives to implementing IGES within the core were as an AutoLisp application and as an external program that processed databases in DXF form. AutoLisp could be ruled out easily, as the restrictions on program size and execution speed were intolerable for a task as complicated as IGES translation. In addition, access to the entity database was not added to AutoLisp until Release 2.6 in June of 1986 (having been coded during the Week of Rest at the end of 1985), and access to AutoCAD's symbol tables, essential for IGES to obtain block definitions, line types, text fonts, etc., was not added until even later. A DXF-based translator would have certainly been feasible (indeed, several have been marketed by third parties), but it would have been far more cumbersome to use and difficult to implement. IGES translation, which already involves processing a huge ASCII file, would require an intermediate DXF file, also huge and slow to write (binary DXF not having been implemented until Release 10). In addition, many operations in an IGES translator require flexible and efficient random access to the AutoCAD database and symbol tables, and the support of a wide variety of geometry service functions. These routines are available within the AutoCAD core, but an external DXF-based application would be forced to create its own AutoCAD-like database from the DXF file and replicate all the processing facilities so readily at hand within AutoCAD.

Regarding this soberly, and remembering that IGES was initiated at the very end of the development cycle of what was eventually called Version 2.5, I concluded that the best way to get IGES done and out the door was by going ahead and implementing it within the core. After all, overlaying would prevent IGES from increasing the memory requirements to run AutoCAD, and we'd already decided to abandon support of floppy disc only machines as of Version 2.5, so growth of the on-disc size was not a major consideration. Integration of IGES in the core would not preclude marketing it as an extra-cost option; after all, at that time AutoCAD was offered in base, ADE-1, ADE-2, and ADE-3 trim levels, differing primarily in which overlay files were supplied on the release disc (actually, it wasn't that simple, but you probably don't want to know any more than that). With a little ingenuity and native cunning, we could figure out a way to sell an IGES package consisting of overlay files you copied into your AutoCAD directory to enable the IGESIN and IGESOUT commands. This would be packaged with the IGES document, and, if done in a sufficiently elegant (read sneaky) fashion, might even permit updates to IGES without the need to release a new AutoCAD.

In the words of Dennis the Menace, “You can't tell how deep a puddle is from the top.” Even though we walked into IGES knowing it was going to be difficult, IGES has never failed to surprise us with the amount of work needed to usably interchange drawings with other CAD systems, themselves only marginally compliant with the ill-defined and mutable IGES standard. With the camel's nose of IGES within the AutoCAD tent, IGES became a part of the mainstream of an AutoCAD release, at least in the sense that IGES bugs had to be closed out before an AutoCAD release could be shipped, even though the overwhelming majority of AutoCAD users never used IGES. However, IGES never became fully integrated in the minds of the developers, so that new features would be added to AutoCAD without implementing them in IGES. This would lead to crises in the final development phase of AutoCAD releases, when furious efforts had to be mounted to implement new IGES code, delaying the AutoCAD shipment date.

To compound the pain, what with everything else we had to do between mid-1985 and the shipment of Version 2.5 in 1986 (this was the first year after our initial public stock offering and was the period in which we were struggling to get the hardware lock ready for introduction with Release 2.5), we never managed to separate IGES from the AutoCAD product, foregoing the additional revenue this would have generated and making it impossible to distinguish an IGES user from any other AutoCAD customer. The following years have not seen IGES and AutoCAD coming to co-exist on any better terms. Virtually every major AutoCAD release has been delayed by an “IGES crisis” of one form or another, resolution of IGES bugs critical to obtaining business remains rigidly coupled to the AutoCAD release cycle, and the memory constraints imposed by the AutoCAD core have required the IGES code to become increasingly fragmented and convoluted and consequently harder to maintain.

Our decision to support IGES in 1985 was correct; IGES has become an essential part of a CAD system today, and with the growing focus on CALS, of which IGES is a part, will continue to be the price of admission to sales to government agencies, their contractors and subcontractors, and, increasingly, large industrial customers. We can only expect our commitment of resources to develop, maintain, and support IGES to increase as our business expands in those sectors. Any steps we can take to reduce the difficulty of IGES development will further the achievement of our goals. The whole smelly, dirty camel's in the tent, and his tail's flickin' at the 640K tentpole that holds the whole mess up. It's time to see if there's a better way to go about this.

Goals and Strategy

There's little doubt that IGES can be implemented as an ADS application. ADS in Let It Be, with the advent of ads_entmake(), provides essentially the same access to the AutoCAD database as exists within the core. The only major disadvantage an ADS program faces is the inability to create AutoCAD symbol table entries (for blocks, layers, line types, etc.) with a mechanism like ads_entmake(), but carefully chosen commands submitted with ads_command() can circumvent this shortcoming.

However, code written as an ADS application looks nothing like AutoCAD core code. The means used to access the database, to look up information from symbol tables, to inquire system variable settings, and to create entities are entirely different. Consequently, extracting a chunk of AutoCAD and turning it into an ADS application consists not just of the usual operation of severing connections and tying off bleeders, but virtually reimplementing all the portions of the code that access the database. In the case of IGES, that's about all there is. Although dwarfed alongside the AutoCAD core, IGES is a huge program in its own right; as of Z.0.64 the IGES complex totals 14,103 lines of code. This is larger than all of AutoCAD 1.0: core, drivers, the works! Another purportedly arcane part of the core, the hidden line complex, is about a third the size of IGES.

Whatever the benefits of extracting IGES from the core, the prospect of what amounts to re-implementing 14,000 lines of code from scratch is not one that excites this programmer. After my earlier experiences with ADS suggested that ADS would support IGES, I began to investigate ways that IGES could be piecewise migrated from the present implementation within the core to a totally separate ADS version without disrupting the product development and release cycle, never requiring a total rewrite with attendant hiatus in IGES releases, one which would deliver the benefits of an ADS IGES as soon as possible with the minimum effort. This project is the fruit of that investigation; what I have at this point is very much a work in progress which will require substantial additional labours before the cutover from the current built-in IGES. However, the nature of the implementation allows this work to be done in parallel with the close-out of Let It Be, without disrupting that process in any way, and should permit shipment of an ADS extended IGES product well before the shipment of Release 12.

Implementation Notes

If you feel your audience uninitiate,
Unable profundities to penetrate,
Rest easy; out of fashion is naïveté.

Using IGES Modules

Since I envision Let It Be shipping with the current, within-the-core implementation of IGES, and the transition to ADS IGES occurring thereafter, and since substantial work remains to be done on IGES before Let It Be ships, I assigned a high priority to avoiding divergence between the built-in and ADS streams of IGES development until the built-in stream is frozen in anticipation of its discontinuation, presumably in Release 12. As long as divergence is avoided, the built-in IGES will continue to be the main development stream until Let It Be ships. Changes to this stream can be integrated into the ADS IGES stream simply by recompiling the updated files, resolving whatever new conflicts may have appeared by changes to the emulation libraries and, as a last resort, by #ifdef code submitted back into the AutoCAD development stream.

As an indication of the closeness between the core and ADS implementations of IGES, long DIFFs of all changes in IGES modules (both .h and .c files and including a couple of IGES bugs I stumbled over and fixed along the way), between the Z.0.64 distribution and my current ADS IGES development directory total only 150 lines. Although some additional changes will be required to bring the ADS IGES up to full functionality, I do not anticipate the need for additional major code modifications in these modules.

Any conditional code required in the IGES modules is controlled by a new compile-time variable named OUTBOARD. This variable, never defined within the AutoCAD core, is intended to be used by code that exists both within the core and outside. I recommend that we define a variable, INBOARD, within AutoCAD, so that conditional compilation statements can be written in the most straightforward manner. My emulation library does not, of course, define INBOARD.

Extracting Core Routines

The AutoCAD core provides a rich set of tools for linear algebra, geometry, storage and list management, file I/O, and manipulation of objects used within AutoCAD. Any code removed from the core must either be provided with versions of these functions or converted to get along without them. Fortunately, most of the heavily-used service functions are reasonably self-contained and easily extracted from their homes in the core. As I worked to get IGES running outside the core, I simply made a list of the functions it used, and for all true support functions (i.e. those not used to access the database or other state-containing parts of AutoCAD), collected the required functions into a module I named autocore.c performing, as I went, transitive closure to pull in functions used by those I included.

addvec alloc angle angle2 arbaxis aschand atwt cfclose cfunlink cndfre cross distan distan2 distan3 distsq distsq2 dot dotp eqv fabsv fcirceq fcirceq2 fixangle flineq fmax fmin fuzzeq g2pilc ga2p gd2p gdpl gdsq2p gl2p handasc hashandle hmabort identv intconic is1bcode is2bcode makerot matxmat mfv napln nulvec pconvt rswap scal setflags setpos sqabsv strsave subvec sumvec ucase unimat unisub univec vconvt vecxmat wtat

Function prototypes, data type declarations, and other definitions required by these functions are in autocad.h, which is included in the compilation of every module extracted from the AutoCAD core. In addition to the routines collected into autocore.c, the following AutoCAD core modules are used, essentially unchanged: dxf.h, erstruc.h, erstruc.c, lstsubs.c.

The AutoCAD Database Emulator

Veteran campaigners of many a fray,
The spectators come well-girt:
Each has book in hand, each has wits alert.

The functions in the file autosim.c emulate the facilities used by AutoCAD core code to access the database, symbol tables, and system variables. As with the functions in autocore.c, these were chosen simply because they were needed by the IGES complex; if this package is used to export other components from the AutoCAD core (such as DXF, for example), some additional functions will almost certainly have to be emulated. Currently emulated functions are listed below; functions marked with an asterisk are currently dummies, not yet emulated but unneeded by the portions of IGES exercised so far into the project.

In addition to these emulated functions, autosim.c contains the ADS initialisation and linkage logic used to invoke the IGES commands when loaded as an ADS application.

Loading A Database

Let's take a deeper look at how the AutoCAD core environment is emulated. When one of the IGES commands is entered, its command implementing function first calls dbload(), implemented in autosim.c. This function builds a simulated AutoCAD database in three steps.

System variables.

First, ads_getvar() is used to capture the current settings of all AutoCAD system variables referred to by IGES. These variables are stored in static data defined in the file autodata.h, each named the same as its incarnation within the AutoCAD core. Other required information accessible through ads_getvar(), including the current drawing name and the AutoCAD version, are obtained as well.

Database index file.

Since ADS provides all the facilities we need to read and write the AutoCAD database, there's no need to prepare a local copy of it. AutoCAD core code, however, addresses the database by means of long handles instead of the ads_name objects used by ADS. To avoid modifying the many places database addresses are assumed to be longs, dbload() makes a pass over the AutoCAD database and prepares a temporary file named dbmap which, addressed by an entity index, returns the ADS name for that entity. All entity references are translated through the file and the global symbols entadr and entcnt contain, respectively, an index into the file and its current length.

Symbol table files.

Finally, the entire contents of each of AutoCAD's symbol tables are retrieved with the ads_tblnext() function. Each symbol table item is decoded into the format used within AutoCAD and stored in a temporary file named after the symbol table. Its file handle is kept in the local copy of the AutoCAD symbol table descriptor. The symbol tables are read and the copies prepared by symtgrab(), called from dbload().

As the block table is copied, the entities that define the block must be stepped through and their names added to the database index file. The index of the block definition entity is stored in the local copy of the block symbol table item. This is not presently implemented, and hence IGESOUT of drawings containing blocks will fail.

Using The Database

Once the local copy of the database is built, the various emulation routines access it straightforwardly. Since the local database is only used for the duration of one command, there's no need for it to be written in a portable format, so regular I/O calls can be used rather than the complicated mechanisms needed to maintain portability within AutoCAD. The dbname() function is called to map a long entity address into the ADS entity name used to retrieve or update it. Whenever an entity is read, it is obtained from AutoCAD with ads_entget(), then “scattered” into the fields of the E structure used within the core by the scatlist() function. The inverse process of collecting the values from the E fields for an entity and building a result buffer chain for ads_entmake() or ads_entmod() is performed by function apmodr(). When entities are added to the database, entries for them are added to the database index file and entcnt incremented.

Symbol tables are accessed by routines identical to those in AutoCAD's smio.c. Since local copies of symbol tables are kept, there is no need to call AutoCAD every time an entry is read. When a putsm() call is made to add an item to a symbol table, special code is required to submit the corresponding commands with ads_command() to build the table item. A local copy must then be added to the symbol table emulation file. The current version of the IGES application does this only for layers; the final version will have to implement all symbol table objects created or updated by IGESIN.

Closing The Database

When an IGES command is done, it calls dbclose(), which releases all the temporary files used to simulate the database. Since changes to the database are passed to AutoCAD as they occur, there's no need to pass any information in the temporary files back to AutoCAD at this point.

Changes to system variables made by IGESIN should be passed back to AutoCAD at this time. This is not presently done.

Development Environment

The ADS IGES application is developed in a self-contained directory, intended to be added to the AutoCAD standard build environment as a directory named iges, at a peer level to ads, lisp, and the others. There's a Unix Makefile in the directory which obtains its ADS libraries from the ads directory in the build tree. This code was built on the Sun, but there's no Earthly reason it shouldn't run on any machine that supports ADS, once the appropriate build procedures are defined.

At the moment, for convenience in developing and debugging, a composite application called iges is built that implements both IGESIN and IGESOUT. It probably makes more sense for the final customer version to supply separate igesin and igesout applications, as one should rarely need both loaded at the same time. The code in autosim.c and the Makefile supports this partitioning into separate applications, although at the moment doing so requires manually changing some #defines at the top of autosim.c.

Additional ADS Requirements

These clever men, how diligent!
Here's another brand-new portent,
Beyond the ordinary man's capacity;
If told I'd doubt the teller's veracity
And scorn the simpleton's naïveté.

One benefit of developing ADS applications in-house is that the process helps to identify and remedy shortcomings of the ADS mechanisms before third party developers encounter them. IGES is very demanding, yet so far only one new ADS facility appears to be required. (I say “appears” because, as should be apparent from the many parts of this project cited in the text as still undone, more may lurk as consequences of emulation functions not yet written.)

The one unambiguous extension needed by IGES is an ads_textbox() function. This would parallel the txtbox() function within AutoCAD; given a text string, its height, style name, X stretch factor, obliquing, and mode flags, it would calculate and return (perhaps as two ads_points with Z co-ordinates presently zero) the extents of the box enclosing that text string. IGES expresses text positions in that form, and I can't think of any way to obtain it other than reimplementing all of AutoCAD's text generation pipeline within the IGES application.

A “would be nice” feature that's not essential and may be provided for already in the ongoing development of ADS would be a signal to an ADS application at the first opportunity within each drawing editor session (including that which loaded it, if not by acad.ads) when AutoCAD was quiescent and the application could submit commands. ADS programs such as GRAVITY and Eagle get around this with a little piece of code that's triggered when the first command they implement is entered, and that suffices for matters such as creating layers, blocks, and suchlike needed by the application. What I'd like to do with IGES is have the application, if loaded, UNDEFINE the standard IGESIN and IGESOUT commands, replacing them with those it implements (the built-in commands would remain accessible through their “true names”). I expect the ability for applications to transparently replace AutoCAD core commands will be a much-requested feature after ADS is shipped.

Also convenient, as additional AutoCAD commands are removed to ADS applications, would be a facility, perhaps implemented through the ACAD.PGP file, which could trigger the automatic xloading of an ADS application when one of the commands it implements was entered. Any automatically-loaded applications would be automatically unloaded at the end of the drawing editor session that caused them to be loaded.

User Notes

Oh yes, how do you use this thing? First of all, very carefully. Getting the fundamentals of the simulation library working took much longer than I expected, so numerous loose ends remain to be cleaned up. Just because my demo works beautifully doesn't mean you can use it; a sufficiently advanced technology is indistinguishable from a rigged demo!

To load an IGES file, bring up AutoCAD on a new drawing, then enter the command:

Other than some diagnostic messages generated as the local copy of the database is prepared, the text screen output is identical to the built-in IGES commands (at least until it crashes).

Summary and Future Directions

Although the time available for this work permitted only a limited amount of the functionality of the built-in IGES commands to be debugged and demonstrated, the approach of transitioning IGES from the AutoCAD core into a standalone ADS application by initially emulating the AutoCAD core environment, permitting the existing well-debugged IGES code to serve as the base for development during the transition, has been tested and appears sound. This suggests a strategy for the future development of IGES something like what follows.

Let It Be continues development toward shipment without any change in direction. IGES changes required by that release are integrated into the core IGES modules. In parallel, the IGES subdirectory is added to the build environment and the support and simulation routines are placed there. The changes to the core IGES modules, made conditional on INBOARD and OUTBOARD will be submitted as changes to the Z stream and carefully code reviewed to establish that they change nothing when built with just INBOARD defined.

It will then be possible to quickly and easily build an ADS version of IGES by copying the IGES files from the coresrc and include directories into the iges directory and invoking make. At this point it will be easy to continue development of the emulation libraries, aiming for identical operation of built-in and ADS-implemented IGES commands by the time Let It Be ships.

Accomplishing this will allow us to freeze the built-in IGES translator as of the Release 11 shipment date and, if we wish, ship interim updates as ADS programs. Indeed, if we continue to include IGES with AutoCAD at no additional charge, we could simply post the latest ADS executables for IGES for each machine on CompuServe and allow our dealers and customers to download and distribute it at will—it will hardly be of use without a copy of AutoCAD.

The working ADS IGES application would, then, serve as the base for our future ambitious plans for IGES support. Starting with a working, albeit limited, version of IGES will allow piecewise extension and replacement with new, CALS-compliant, code. The new code will be written as ADS-native code, bypassing the emulation library since there would be no need for the new code to be compatible with the conventions of the AutoCAD core. This code will probably use SGLIB or an equivalent package for linear algebra rather than the more baroque constructs within AutoCAD—there's no need to make the sacrifices of comprehensibility the AutoCAD routines make for code that's not in the generation pipeline. In time, most of the original IGES code from AutoCAD will have been replaced, and the pieces of the emulation library can be retired as they are no longer referenced.

With Release 12 we could discontinue the built-in IGES commands and support IGES only through the ADS implementation. Alternatively, we could offer Release 11 level IGES support built-in and offer much more comprehensive IGES support through the ADS version as an extra cost option.

Conclusion

“Caustic” has two meanings. In mathematical optics, it refers to the figure formed by rays of light reflected or refracted into a singularity—a mathematical point where the trend reverses, when what was converging diverges, and vice versa. In chemistry, it's a corrosive substance, prone to etch away even substantial structures. As the 1980's pass into the 1990's we seem to stand at a caustic moment in both senses of the word: we're passing through a time when linear projections based on the past provide little guidance for the future; a time when the structures, institutions, and dynamics we've assumed were fixed for many years seem to be disappearing into dust to be replaced by…what?

AutoCAD, as well, is poised at a caustic moment in its history. Having grown comfortably in a fairly linear manner for the last seven years, it is now faced with user requirements and market challenges that dwarf any in its history. AutoCAD will need to grow into something much larger than perhaps any of us can today envision, and must do so on an accelerating time scale. We can achieve these goals: goals I believe essential if Autodesk and AutoCAD are to remain dominant in the market, only if we decouple the evolution of AutoCAD from the necessity of modifying the AutoCAD core code. The AutoCAD core must not just cease to grow—it must shrink in the future. By removing IGES, which in an ideal world would never have been in the core to start with, we can commence this process.

In several years, AutoCAD can come to resemble, not the intimidating monolith of code it now is, but a community of independent and cooperating applications, each developed and maintained by a small team expert in that aspect of the overall task and able to grasp the totality of a program that, by division of labour, has been reduced to a scale the human mind can comprehend. The AutoCAD user will see increased capability, better user interfaces, and improved reliability. If we do our job correctly, the seams won't show, any more than you wonder if the mitochondria in your cells are really you. This caustic moment, while marking the high point in the growth of the AutoCAD core, may also be a key milestone on the long road toward AutoCAD's ultimate destiny.