Yadex $VERSION ($SOURCE_DATE)

Hacker's guide

Blah

Foreword

I apologize for the poor quality of Yadex's code but it seemed to me it was better to release something imperfect now than something clean two years from now. If you want to improve it, be my guest.

Introduction

Original platform

The DOS version has been developped with Borland C++ 4.0 on a PC 486 DX4/75 with MS-DOS 6.22.

Yadex should be compilable on all reasonable Unix-and-X11 platforms provided that

• you have GNU make,
• you have a C compiler and a C++ compiler,
• $c short is 16-bit long,
• $c long is 32-bit long.

Historic background

DEU 5.21 was written by Raphaël Quinet, Brendon Wyber and others and released on 1994-05-21. As you probably already know, DEU was a real-mode DOS program, in C, compiled with Borland C++ 4.0 (I think) and using BGI for its output.

In the mists of time (that is probably 1996), I began to hack DEU for my own use. In 1997, other people began to use my hack and I gave it a name : "Yade" (which meant Yet Another Doom Editor). It was still a real-mode DOS program.

In june 1998, tired of rebooting to DOS every time I wanted to do some Doom level editing, I started porting Yade to Linux. As there already was a Unix program called "Yade" (Yet Another Diagram Editor), I changed the name of my baby to "Yadex". At the same time, I began to use C++ in places so that's why Yadex is such an ugly mixture of languages.

Development cycle

Compiling, installing, testing

OK. So I want to hack Yadex ; what do I do now ? The obvious development cycle is

1. modify the files in src/,
2. type make,
3. type su -c 'make install',
4. run Yadex and test.

However, there are a few things to know that can make you more efficient.

To compile with debugging information, you don't have to modifiy the makefile ; just use the appropriate target (dyadex). The resulting executable is put in dobj/0/ instead of obj/0/.

You don't have to install to test : you can test in place with the special targets test and dtest (test is for the regular executable, dtest is for the debugging executable and is my favourite target by far). When testing through make, you can't pass arguments to Yadex directly : you have to pass them through the A macro. For example :

  $ make dtest A='-g heretic foo.wad'

You can also run your hack through the debugger with the dg (GDB) and dd (DDD) targets. With those targets, you can't use the A= convention : you have to type "set args blah blah blah" at the debugger prompt.

  $ make dg
  (gdb) set args -g heretic foo.wad
  (gdb) run

Even if you want to install your hack, you may want to keep the original Yadex around, for reference or for a good laugh. To do that, edit the VERSION file before compiling your hack. If your hack bears a different version number, it will not overwrite the original version. You don't have to change the version number much : you can just change "3.8.0" into "3.8.0a" or "3.8.0.0" for example.

The programming environment

Memory allocation

$c GetMemory() and friends manage more things for you. They include an anti-fragmentation system, they try to swap things out when memory is tight (this is an only an issue for the 16-bit DOS version) and if they fail, they call $c fatal_error() so you don't need to check their return value.

The reason for $c GetFarMemory() is that, for the 16-bit DOS version, it can allocate more than 64 kB ($c GetMemory() cannot). I must say that I don't use $c GetFarMemory() a lot myself because I don't like the idea of having to use two different memory allocation routines depending on the size I expect to allocate. I modified $c GetMemory() so that it accepts an unsigned long but checks that the passed value fits in $c size_t. In other words, if you call $c GetMemory() with a size of 65,536 the 16-bit DOS version will trigger a fatal error immediately instead of silently allocating 1 byte and letting you search afterwards why the program behaves strangely. A better fix would be to make $c GetMemory() call $c GetFarMemory() when the block is too large for $c malloc(). Any volunteers ?

Memory allocated with $c GetMemory() is guaranteed to be freeable with $c free(). On the other hand, memory allocated with $c GetFarMemory() must be freeed with $c FreeFarMemory().

Endianness

On the other hand, Yadex keeps all its in-core integer in the platform's native endianness, i.e. in little-endian format on little-endian machines and in big-endian format on big-endian machines.

The wad endianness <-> native endianness conversion is done automagically by $c wad_read_i16() and $c wad_read_i32().

To maintain compatibility with big-endian platforms, all I/O of multibyte integers should be done with those functions.

The directory

Principle

As you might expect, if the same lump is present in more than one wad, it has only one directory entry, pointing to the last occurence.

When you need to load a lump by name, you call FindMasterDir(). It returns you a pointer on its entry in the directory, which in turn contains everything you need to read it.

Managing the directory

The directory is kept in memory at all times. It is only modified when you initially load the iwad and when you load or unload a pwad. And of course when you delete the directory. Simple, isn't it ?

Ha-ha, gotcha, this is not how the actual API works. The actual set of operations is somewhat different in that it does not include unloading a pwad. Instead, there is a function to close all unused pwads, that is pwads that are not referenced by a single entry in the directory. I suppose that the reason why it was done this way is that unloading a pwad is a bit more complicated, since it involves finding the previous "provider" of all lumps that the wad to unload used to "provide". I guess the only way to do that would be rebuilding the directory from scratch. This "feature" is discussed in the Comments section below. Anyway, here is the API.

OpenMainWad()

Create the directory and fill it with the contents of the directory of the iwad.

OpenPatchWad()

Add a pwad to the list of the open wads and to the directory.

CloseUnusedWadFiles()

Close all wads that are not used in the directory anymore.

CloseWadFiles()

Close all open wads and delete the list of open wads and the directory.

Implementation

struct WadFileInfo
   {
   WadPtr next;                 // Next file in linked list
   char *filename;              // Name of the wad file
   FILE *fd;                    // C file stream information
   char type[4];                // Type of wad file ("IWAD" or "PWAD")
   i32  dirsize;                // Directory size of wad
   i32  dirstart;               // Offset to start of directory
   DirPtr directory;            // Array of directory information
   };
typedef struct WadFileInfo *WadPtr;
extern WadPtr WadFileList;      // List of wad files

The directory itself is a linked list of the MasterDirectory structure. The global variable MasterDir points to the first element of the directory. The type MDirPtr is an alias for "struct MasterDirectory *".

struct Directory
   {
   i32  start;                  // Offset to start of data
   i32  size;                   // Byte size of data
   char name[WAD_NAME];         // Name of data block
   };
struct MasterDirectory
   {
   MDirPtr next;                // Next in list
   WadPtr wadfile;              // File of origin
   struct Directory dir;        // Directory data (name, offset, size)
   };
typedef struct MasterDirectory *MDirPtr;
extern MDirPtr MasterDir;       // The master directory

Sample code

Comments

The decision to use a directory is arguable. It is convenient for the programmer when he/she is looking for the effective instance of a lump, which is the case most of the time. But it also prevents the user from editing a resource (notably, a level), if it has been overridden in another wad. It's not a big deal but I don't like it.

The fact that the directory managing operations don't include removing a pwad from the directory means that there is no way for the user to "unload" a pwad. The "read" command has no inverse. The only way to do it is to restart Yadex.

Another somewhat non obvious design decision is that, in most places where the directory is updated, CloseUnusedWadFiles() is called too. This means that you can't load two pwads that have exactly the same lumps. As the second pwad is loaded, the first one is automatically (and silently) unloaded. Not a big deal either but, as a user, I don't like the programs I use to behave like that. I'll illustrate my point with the following scenario, which assumes the "unload pwad" function exists :

1. Load pwad A (MAP01)
2. Load pwad B (MAP01)
3. Unload pwad B
4. Edit MAP01

I don't like the directory management API very much because it's unexpectedly asymmetric and therefore neither intuitive nor orthogonal. The way Yadex plays games with the directory is really disgusting and confusing to me.

I should look into replacing the iwad "on the fly", so that the user is able to change the game parameter dynamically, without restarting Yadex. In fact, the ultimate goal is to remove the game parameter completely or, more precisely, to make it local to a Level object, automatically adjusting and dynamically modifiable by the user.

The wad data

The level data

Structure

int   NumThings;         /* number of things */
TPtr  Things;            /* things data */
int   NumLineDefs;       /* number of linedefs */
LDPtr LineDefs;          /* linedefs data */
int   NumSideDefs;       /* number of sidedefs */
SDPtr SideDefs;          /* sidedefs data */
int   NumVertices;       /* number of vertices */
VPtr  Vertices;          /* vertices data */
int   NumSectors;        /* number of sectors */
SPtr  Sectors;           /* sectors data */

Scope and lifetime

I think that the level data class should be separate from the editing window class because it might be useful to open several editing windows on the same level. Separate class should also make the design of the read level and write level routines cleaner and simpler.

Maintenance

• the $c Num* variables must remain accurate,
• vertex references in linedefs must be either $c OBJ_NO_NONE or the number of an existing vertex,
• sidedef references in linedefs must be either $c OBJ_NO_NONE or the number of an existing sidedef,
• sector references in sidedefs must be either $c OBJ_NO_NONE or the number of an existing sector.

Loading

Since the endianness of the wad files is fixed (little endian) and thus not necessarily identical to the endianness of the CPU, reading 2- and 4-byte integers from the file is done through special endianness-independant routines.

Saving

Since the endianness of the wad files is fixed (little endian) and thus not necessarily identical to the endianness of the CPU, writing 2- and 4-byte integers to the file is done through special endianness-independant routines.

Editing windows, or the lack of it

See "_edit.h".

The editor loop

The $c EditorLoop() is an endless loop (okay, not really endless) which, for each iteration, first, refreshes the display, second, waits for an event, third, processes that event. I could have put things in a different order but I liked the idea of displaying something before waiting for user input.

Because the event input and the graphical output are complex and not-quite-synchronous processes, I've tried to separate them. $c EditorLoop() gets input events and processes them and calls another function, $c edisplay_c::refresh(), to take care of everything display related. If you replaced $c edisplay_c::refresh() by a stub (and did the same with a couple of functions in $c gfx.c and $c input.c), you could perfectly well, if blindly, run Yadex without a connection to an X server. While you may object that this would be a pointless exercise (to which I agree), it still proves the modularity of the design.

The $c edisplay_c::refresh() function is also a very important one to understand, at least if you work on graphical output. It is discussed in another section but, just to settle ideas, I thought I'd give you here a bird's eye view of the whole thing. If there is a single paragraph in this document that you need to read, it's probably this one :

• $c EditorLoop()
  • Call $c edisplay_c::refresh(),
    • Do some basic geometry management,
    • setup widgets according to editing session,
    • call the $c need_to_clear() method of the widgets to determine whether we need to redisplay everything from scratch,
    • if so, call the $c clear() method of the widgets and call $c ClearScreen() to clear the pixmap (if available) and make sure all subsequent graphical output will be sent to the pixmap (if available),
    • else, make sure all subsequent graphical ouput will be sent to the window and call the $c undraw() method of the widgets from the top down,
    • call the $c draw() method of the widgets from the bottom up,
    • call $c update_display() to refresh the physical display (if we're using the window, it's a no-op, if we're using the pixmap, call $c XCopyArea()).
  • call $c get_input_status() to get the next event,
  • process that event.

The display

Logical and physical display : widgets

There's obviously more to say on this...

The pixmap

As long as we do incremental changes to the display (E.G. "undisplaying" the highlight on a vertex or redisplaying the pointer coordinates), we do it directly on the window.

But, if we have to redraw everything from scratch, we have to clear the window first which generates an annoying "flashing" of the screen. To avoid this, we instead clear a pixmap, do our display thing on it and then put the pixmap onto the window, with $c XCopyArea(). The result is a flicker-less refresh.

The graphical routines from $c gfx.c switch automatically to the pixmap if $c ClearScreen() was called. Thanks to this, that window vs. pixmap thing is nearly transparent to the application functions. $c edisplay_c::refresh() just forces widgets that can undraw themselves to use the window, not the pixmap.

But a pixmap is large. For a 800x600 window in 64k colours, 937 kB. And copying it to the window is obviously long. So, on machines with little memory or a slow CPU, the user might prefer to do without it. That's what $c no_pixmap is for.

The selection

Introduction

From the programmer's point of view, the selection is a singly linked list of objects of this type :

typedef struct SelectionList *SelPtr;
struct SelectionList
   {
   SelPtr next;                 /* next in list */
   int objnum;                  /* object number */
   };

Note that the $c SelectionList structure has no $c objtype field ; the type of the object (THING, vertex...) is implicit from the current mode (the $c obj_type field from the $c edit_t structure). As a consequence, the selection cannot contain objects of different types.

The selection manipulation functions are supposed to be defined in $c selectn.c and declared in $c selectn.h. Here they are :

void SelectObject (SelPtr *s, int n)

Adds object $c n at the beginning of list $c *s. $c *s can be $c NULL ; it means the list is empty. Warning : does not check that object $c n is not already in the list.

void UnSelectObject (SelPtr *s, int n)

Removes from list $c *s all occurrences of object $c n. If all objects are removed, sets $c *s to $c NULL.

void select_unselect_obj (SelPtr *s, int n)

If the object $c n is already in the list $c *n, remove it. If it's not, insert it at the beginning. $c *s can be $c NULL ; it means the list is empty.

Bool IsSelected (SelPtr s, int n)

Tells whether object $c n is in selection $c s. $c s can be $c NULL ; it means the list is empty.

void ForgetSelection (SelPtr *s)

Frees all memory allocated to list $c *s and sets $c *s to $c NULL.

void DumpSelection (SelPtr s)

Debugging function ; prints the contents of the selection to $c stdout.

SelPtr cur;
for (cur = list; cur; cur = cur->next)
   do_something_with_object (cur->objnum);

Selecting in/out

That's when $c select_unselect_obj() is used.

The highlight

Colours

$c set_colour()

Set the current colour to a new value.

$c push_colour()

Save the current colour on the colour stack and set the current colour to a new value.

$c pop_colour()

Set the current colour to the value it had at the moment of the last call to $c push_colour().

Menus and pop-up windows

Game graphics

Viewing graphics

Textures are made by pasting one or more patches onto a rectangular buffer. The resulting image can be of course transparent.

For all formats, a pixel is a byte. The value of the byte (between 0 and 255 inclusive) is an index into one of the palettes contained in the PLAYPAL lump. To be able to know the actual RGB colour of a pixel, it is necessary to look up its value in the PLAYPAL (normally, only the first palette is used).

As is the case with saving graphics to a file, viewing graphics would involve a lot of coding to do it in an optimized fashion. Therefore it's done in a somewhat wasteful way :

1. the image is put into a buffer
2. the buffer is pasted onto the window or screen

The first step is done with DisplayFloorTexture() (for flats) and LoadPicture() (for other graphics). For textures, LoadPicture() is called repeatedly on the same game image buffer, once for each flat.

The second step is done with display_game_image(). It takes care of converting Doom pixel values into actual pixel values needed by the graphical API (which depend on a lot of factors like the video mode, type and depth of visual, byte order of X server, etc.) and getting the graphical API to display it on the window or screen.

display_pic() combines fetching the image into the game image buffer and displaying it.

spectrify_game_image() can be used on a game image buffer to make it look more or less like a spectre.

Saving graphics to file (screenshots etc.)

1. get the image to save in an Rgbbmp buffer (with window_to_rgbbmp()),
2. save the Rgbbmp buffer to a file (with rgbbmp_to_ppm()).

The way to retrieve the image from the window or screen is very dependent on the platform or video mode or type of visual and the fashion in which the image is read is very dependent on the graphic format. Therefore, if you want to use a one-step algorithm and you have n platforms or video modes or types of visuals and m graphic formats, you have to write n × m functions. With a two-step algorithm, you have only n + m functions to write. As saving graphics to file is not a critical function, I'd rather keep the code short and simple, unless the result proves unacceptable.

Saving

One very important constraint in the saving code is that it should never fail unless there is no reasonable alternative. That's because, for users, saving is a passage obligé to quit the program without losing their work. Therefore, the saving code must be particularly robust and able to recover from errors.

This implies that the saving code must not need to allocate new memory. To understand why, consider the following scenario : you load a level and edit it until Yadex uses all the available memory save a few kB. Then you want to save it and Yadex says "nope, I don't have enough memory to do that". From a user point of view, that sucks.

Unfortunately, at present, many benign errors are considered fatal when saving. For example, trying to save to a file on which you don't have the necessary rights will make Yadex abort even though this error is perfectly recoverable. This should be fixed since it could cause users to lose data.

Compile-time variables (defines)

$c AYM_MOUSE_HACKS

Some experimental code by me to try to understand why, under DOS, the mouse pointer moves 8 pixels at a time (seems to depend on the mouse driver ?).

$c CIRRUS_PATCH

Dates back to DEU 5.21. Apparently, some code specific to Cirrus VGA boards. Does nothing unless $c Y_BGI is defined.

$c DEBUG

The obvious.

$c DIALOG

Experimental code by me to test the dialog box function that Jim Flynn wrote for Deth in the beginning of 1998.

$c NO_CIRCLES

If your BGI driver does not support drawing circles, define this and Yadex will draw squares instead.

$c OLD

Misc. obsolete stuff I didn't want to delete at the time. Never define it or you'll break Yadex ! Code under #ifdef OLD should probably be removed.

$c OLD_METHOD

My cruft. Code thus #ifdef'd should probably be removed.

$c OLD_MESSAGE

My cruft. Code thus #ifdef'd should probably be removed.

$c ROUND_THINGS

Draw THINGS as circles (like DEU did), not as squares.

$c SWAP_TO_XMS

Comes from DEU : related to code supposed to use XMS as "swap space". Apparently, was never used ?

$c Y_BGI

Use BGI for graphics output and BIOS for keyboard and mouse input. Makes senses only for DOS + Borland C++ 4.0. Exactly one of ($c Y_BGI, $c Y_X11) must be defined.

$c Y_DOS

Compile for DOS (with Borland C++ 4.0). Allows, among others, the $c huge and $c far pointer modifiers. Exactly one of ($c Y_DOS, $c Y_UNIX) must be defined.

$c Y_UNIX

Compile for Unix (with GCC 2.7.2). Causes, among other things, "huge" and "far" to be #define'd to "". Exactly one of ($c Y_DOS, $c Y_UNIX) must be defined.

$c Y_X11

Use X11 (Xlib) for graphics output, keyboard and mouse input and other events. Exactly one of ($c Y_BGI, $c Y_X11) must be defined.

Coding standards

Indent style

• tab stops every 8 characters,
• indent width is 3 spaces,
• line width limited to 80 characters,
• the braces have the same indentation as the text they contain,
• one space between the name of the function/statement and the "(",
• no space between the "(" and the first argument,
• no space between the end of the argument and the "," or ";",
• one space between the "," or ";" and the next argument,
• no space between the end of the argument and the ")",
• no space between the ")" and the ";",
• the body of a function is not indented.

Example :

static const char *foo (int n)
{
for (stuff; stuff; stuff)
   {
   if (thingie || gadget ())
      call_this_one (with, three, exactly[arguments]);
   else
      dont ();
   call_that_one ();
   }
return NULL;
}

Identifiers

I consistently use the English spelling (E.G. "colour", not "color").

General style

/* This is a long comment. A long comment is a comment
   that spans over several lines. See how I "typeset" it. */
func         (object, mutter    );
another_func (object, mumble, 46);
yet_another  (object, grunt     );  // Short comment.

Code

• Use $c const for all arguments passed by reference unless they're modified.
• Use $c static for all functions and variables unless they're extern.
• Be careful with $c int, on some platforms, it's equivalent to $c short, on others, it's equivalent to $c long.
• Whenever you need a fixed number of bytes, use $c u16, $c i32, etc.
• Use prototypes.
• Prototype functions that take no args with "foo (void);", not "foo ();" (author's note : this is C-ish !).
• Try to avoid buffer overruns by using $c snprintf() instead of $c sprintf(), $c al_scps() instead of $c strcpy(), $c al_saps() instead of $c strcat(), etc.

Includes

If the need arises to protect a Yadex header against multiple inclusion, use this :

#ifndef YH_FOO
#define YH_FOO
(put the contents of the header here)
#endif

File format

Notes

1

It's important to set the $c line_width to 1 and not 0 for GCs that have the $c GXxor function, for the following reason.

Those GCs are used for objects that should be "undrawn" by drawing them again, E.G. the highlight. Now, imagine the following scenario : you highlight a linedef and then press, say, page up to make the window scroll. This is not an incremental change to the display so everything is redrawn from scratch onto the pixmap. The pixmap is XCopyArea()'d onto the window, the linedef still highlighted. Then, Yadex realizes that the map coordinates of the pointer have changed so the linedef is not highlighted anymore. It dutifully unhighlights the linedef. But $c XDrawLine() on a window does not use the same algorithm as on a pixmap (attempts to use the blitter on the video card). So the first line and the second don't coincide exactly and the result is a trail of yellow pixels where the highlight used to be.

That's why I use a $c line_width sure that the same algorithm is used both for pixmap output and for window output.