1. Introduction

This document describes the game debug environment for the Nintendo
Ultra64 system. It briefly explains the hardware and software
environments, illustrates recommended programming model, gives you an
idea of how to get started with the debug environment, and introduces
you to the most commonly used debugger features.

2. Hardware Environment

The debugger is intended to work as part of a hardware environment
based on the Ultra64 development board and an SGI INDY workstation. The
development board is an extension of the game system itself. The
Ultra64 game system consists of a MIPS R4300 RISC microprocessor, a
graphics and audio coprocessor (the RCP), 2 to 4 Megabytes of dynamic
RAM, and some peripheral devices. The game itself is contained in ROM
on a game cartridge. For the development system, the ROM on the game
cartridge is replaced by RAM on the development board; in this
document, we refer to it as "virtual ROM". This allows the game
developer to load the game program into memory, control its execution,
and observe the effects of modifying the game without having to rebuild
from source.

The development board plugs into the GIO bus of the workstation. Audio
and video output connections are provided. Communication facilities
between the workstation (referred to as the host in the rest of this
document) and the development board (called the target) are via the RAM
devices that emulate the cartridge ROM and several registers provided
for handshaking and synchronization.

3. Software Environment

The software debug environment consists of a number of software modules
that must be present to support debugging. Some of these will also be
present in the final game system, but many will not. A good
understanding of the software architecture will enable the game
developer to deal with unexpected situations that arise during a
debugging session.

The target software is the set of modules that will run in the game,
plus those target resident modules that support debugging only.
Clearly, the most important of these is the game application itself.
The game will also make extensive use of run time library functions to
do its work. These include not only graphics and audio routines, but a
collection of operating system primitives that support the
multithreaded design of the Ultra64 system. While using the operating
system in a game is not absolutely required, its use is strongly
recommended. It is designed to be simple, yet powerful, and is both
fast and relatively unobtrusive at run time. For debugging, its use is
required, since the debugger is designed to run under the OS.

At the highest level, the debugger consists of two major parts. On the
development host, a graphically-oriented source level debugger called
tvd is provided. In the target system, a small in-circuit debug monitor
called rmon acts as the agent for tvd. The operator of the debugger
sees only tvd, but requests are actually fulfilled by rmon. That is,
the user may open a window on the host for the purpose of looking at
memory contents. The host cannot access such memory directly, but it
can ask rmon to fetch the memory contents from the target so that the
host can display them. Rmon runs as three threads under the OS, but
these threads spend most of their time either blocked awaiting a host
request or stopped. Thus, they do not interfere with the operation of
the game (other than taking up some memory) unless they are processing
debugging commands under operator control.

Like the OS and other library routines, rmon is included in a build
only if the game developer specifically asks for it. This is done by
creating a thread with rmonMain specified as the function to be started
when that thread is run. The rmon program is part of libultra, the
Ultra64 run time library. The user does not need to have any special
files to include rmon in a build. Referencing rmonMain will
automatically include all code and data for all three of rmon's
threads.

On the host side, the main program seen by the user is tvd, the
debugger. However, there are a number of support programs that run in
conjunction with the debugger. Since tvd is designed to work in other
environments as well, it uses a separate program called dbgif (for
debugger interface) to communicate with the target environment. Only
dbgif knows the actual means of communication with the target system;
tvd is independent of such concerns. Since we wish to share the GIO
interface between the host and target with other programs (e.g.,
diagnostics), a third module is provided on the host. This program is
called driverd (for driver demon), and functions as the target
manager.  When any program (such as dbgif) wishes to communicate with
the target, it issues requests to driverd. In this way, it is possible
for two pairs of programs running on the host and target to communicate
through a single channel without interference.

4. rmon Theory of Operation

As mentioned in the previous section rmon consist of three threads that
run under the operating system, but these threads run very
infrequently. The rmon main thread consists of a command parser, a
command dispatcher, and a collection of service routines. In operation,
the debugger sends a request to the target. This request consists of a
number of 32-bit words that describe the work to be done - for example,
"read 40 words starting at address 0x10000000 in the address space of
thread 6." (Note: all threads run in the same address space in this
environment, but the debugger could support a more complex environment
where this was not the case. The debugger does consider the RCP to be a
separate address space internally.) This request gets passed through
dbgif to driverd. The host (through operation of the driverd process)
alerts the target that it wishes to send a message. A very small, high-
priority thread called the rmon IO thread responds to the interrupts
that occur when driverd writes to one of the GIO registers. Only one
access to the "virtual ROM" is allowed at a time, so the host must wait
until any DMA access in progress is completed.

When this has happened, the target notifies the host that it is now
possible to use the memory. At this point, the target system starts a
high priority system thread (the rmon spin thread) that keeps the game
from running and starting any more accesses to virtual ROM. Since the
game is not accessing this memory, the host is now free to load the
request packet into a predetermined location at the high end of
memory.  When the packet has been deposited in memory, the host
notifies the target that a request has arrived. This stops the rmon
spin thread. The rmon IO thread notifies the main rmon thread and waits
for the next interrupt.

The rmon main thread wakes up in response to the message from the rmon
IO thread. It fetches the incoming packet and dispatches a service
routine based on what service was requested. In our example,
rmonReadMem will be called. This function will examine the arguments,
read the memory, and deposit the contents in another section of virtual
ROM as part of a reply packet. It will then send an interrupt to the
host , alerting it to the arrival of the reply packet in memory. The
host responds to this interrupt by copying the reply packet out of
virtual ROM and sending another interrupt to the target. This provides
feedback to the target that the host has finished with the reply buffer
and the target may use it again.

Most transactions between the host and target follow this model, but
there are a few exceptions. It is likely that the target will
asynchronously send a packet to the host that is not a reply to a host
request. This occurs whenever a breakpoint has been encountered, for
example. Both host and target "sign on" when starting, and each has a
reply that it sends to the other when such a sign-on is received. The
debugger can also process notification that a thread has been created
and destroyed. While not presently used, these may be added in the
future.

Target-generated interrupts are received by the driverd process on the
host system and routed to other processes (e.g., dbgif) that have
registered that they would like to receive a given set of interrupts.
(Interrupts are associated with a six-bit value identifying which
interrupt occurred.) Thus, rmon sends a specific interrupt code to the
host. This code indicates that the message should be send to dbgif and
not some other process. Driverd does not read the communication buffers
except as an agent for dbgif.

5. Installation Requirements

Many game developers will use a "turn-key" system that came with all
development software installed. Early users are unlikely to have this
luxury, and will need to install software themselves. In this section,
we will describe what is required to run the debugger.

libultra - The libultra library is part of the development tools
distribution. It includes all operating system and rmon functions
needed to run the development environment, in addition to graphics and
audio functions. This is really all you need for target code, and
inclusion of the appropriate software modules is as simple as doing a
build of your application. The language development tools (compiler,
assembler, linker, makerom, etc.) will do all the work for you.

CaseVision - If you do not have CaseVision installed on your
workstation, install it. This should be installed FIRST.

WorkShop - If you do not have WorkShop installed on your workstation,
install this SECOND.

tvd - You must have a copy of tvd on your system, and it must be
installed AFTER CaseVision and WorkShop (it needs to overwrite some
files). Note that there are two programs called tvd; you need the one
specifically modified for the Ultra64 environment. We will likely
change the name of this program to avoid confusion. However, tvd is
really a collection of independent, communicating processes. Some of
these may have names that conflict with other versions of tvd. It is
recommended that only the Ultra64 version be installed on systems used
for game development. tvd is installed from a set of images, and
includes executables and on-line help. For internal users, these images
may be obtained by doing an inst from:

/hosts/sushila/usr/tmp/images_reality.

We will move these into our own location before release.

dbgif - You must have a copy of dbgif installed on your system. For
internal users, this is built in the source tree as
$ROOT/PR/debugger/dbgif.

driverd - You must also have a copy of driverd to run the system. For
internal users, this is also in the tree.

nld - You must have the latest version of nld. Unfortunately, all
versions of nld sign on as 6.02. For internal users, use the one that
has been checked into the tree.

Only tvd requires licenses. Developers will be provided with a license
locked to the ID of their development machines. For internal users, you
may obtain a temporary license via WWW. You should submit a request for
a permanent license, as well.

6. Programming model

While a game may use any programming style desired by its author(s),
there are certain restrictions imposed by the debugger. Those
developers who wish to use the debugger will have to conform to the
rules of the programming model (sorry guys) to obtain the benefits of
source level debugging. In this section, we will discuss what
restrictions apply.

The most obvious requirement is that you must use the OS, since the
debugger depends on it. It will not work under an OS of your own
design, since it is designed for the Ultra64 OS.

Use of the debugger also requires that you restrict thread priorities
to a specific range.  User threads (those that are part of the game)
are assigned the range 1..127, with 127 being the highest priority
thread. The OS does not prevent you from assigning thread priorities
higher than 127, but you will be unable to debug them. In fact, use of
priorities in this range make prevent the debugger from working at all.
While the OS does not impose any restrictions on the null thread (other
than the requirement that there be one), the debugger requires that the
null thread be assigned priority level zero. It is not sufficient that
it be the lowest priority thread in the system - it must be zero.
Otherwise, the debugger may attempt to suspend it, which will lock up
the system.  The rmon main thread should be set to priority 250.

The boot procedure for the system is described elsewhere, but some
parts of it are repeated here because a review is helpful.  Each
application has a boot function, which is called at startup (after
security checking, of course). The boot function initializes the
operating system, and then creates and starts the main thread. The boot
procedure may also do other things, such as hardware initialization if
desired. It can also create other threads, but starting a thread is
always the last thing the boot procedure does. The reason for this is
simple; once control is transferred to a thread, there is no way to get
back to the boot procedure. To enable as much debugging of your startup
code as possible, the boot procedure should be minimal - probably just
the three function calls that are required to start the main thread.

The main thread starts other threads within the system, including the
debugger thread.  There is more flexibility here, although the ability
to debug system startup is significantly better if the recommended
model is followed.  The recommended model is for the main thread to
create and all other threads in the system, start only the rmon
thread(s), and then lower its own priority and become the idle thread.
Again, you don't have to do this, but debugging will work much better
if you do. For internal users, see
$ROOT/PR/debugger/rmon/ermon/example.c for a file that has the
recommended structure for debugging. It is also an attachment to this
document.

Clearly, you can't debug any code that comes before starting the
debugger (rmon) thread. It is also the case that you can't really debug
code that has already executed by the time the debugger starts up. This
is not so much a function of time as it is of the traditional approach
used in debugging embedded systems like the Ultra64. That is, if you
want to watch the system start from inside the debugger, then you can't
really start running the application.  Since the debugger is just
another thread under the OS, it does not keep your application from
running off and executing the game application. Some debuggers may
"hold off" the application until the debugger is ready; this one
doesn't.

Of course, this does not mean that you can't debug the startup of your
application. It just means you must bring up your system in a stopped
state and start it running from within the debugger. To do this, your
code should start only two threads (although it can create as many as
it wants, since creating a thread does not cause it to run). The two
threads are the rmon thread, which we will consider to be only one
thread for now, and the idle thread. Comment out or conditionally
compile in the osStartThread calls for other threads so that they do
not run until told to do so. Running a thread from the debugger is
exactly like calling osStartThread.

What happens if you don't follow this procedure and you start all the
threads in your system? Unfortunately, in most cases the debugger will
not be able to start, since it needs a stopped thread to connect to.
The idle thread and the debugger threads will be running, but it is
likely that all your application threads will be blocked on some
event.  We have provided a dummy thread that is created by the debugger
but never started. The purpose of this dummy thread is to give the user
at least one stopped thread to which the debugger can attach. Be
careful never to run this thread!

7. Getting Started

Once you have all the required software installed on your system, you
can modify your application to include rmon. Since rmon is rather
passive, it does not require you to run the debugger. It just waits for
incoming requests and does not interfere with the game operation unless
requests arrive. Example source code is provided as an attachment to
this document. This illustrates how rmon is started and how an
application may be modified to use it. An include file, rmon.h, is
provided as part of the distribution.

Once you have built your application, you must download it following
the standard procedure.  Once the application has been loaded and is
running, you can start the dbgif process on the workstation. The
driverd process should already be running as a result of downloading.
The dbgif process will register with driverd and sign on with the
target system. A small script is provided to start dbgif with the
proper options. It is available as
$ROOT/PR/debugger/rmon/ermon/emutest, and is also an attachment to this
document.

You may now start tvd itself. For the Ultra64, it is required that tvd
be started with the name of your executable (the boot executable if
there is more than one) on the command line. For example, if you
executable is names sample, you would enter:

tvd sample &

The debugger will then start. It makes no attempt to contact the target
system yet.

You should have a source window and a small status window (which may be
minimized if desired). At this point, you must establish a link to the
target. This is done by selecting the Admin pull down and clicking on
Change Thread. You will be prompted for the id of the thread to which
you wish to connect.  Under the OS, threads do not really have small
integer ID's; instead, they are referenced by the address of their
thread control blocks.  However, this is quite inconvenient to type, so
rmon maintains a table mapping thread IDs to thread control blocks. For
historical reasons, the first user thread is thread five. Thread ID's
are assigned in ascending order based on the order of creation. If you
wish to connect to the thread created by the first call to
osCreateThread, specify that you wish to attach to thread five. These
numbers apply to all threads in the range 1..127; the idle thread is
not included, nor are any threads with priorities grater than 127.

You may only attach to a thread that is in stopped state. If you start
the application with all threads stopped as recommended above, you will
not have any problems attaching. Unfortunately, you cannot attach to a
thread that is running, and you cannot stop any threads until you have
attached at least once. Thus, there must be at least one thread stopped
when bringing up the debugger.

Once you have successfully attached, the host and target will
communicate to pass information about the system state back and forth.
This will take a few seconds, or even longer if you have many threads.
Once completed, you may bring up other views as appropriate to your
debug session. Views are opened by selecting the Views pull down and
then clicking on the view you wish to see. The most-used of these are:

register view - This is where you may examine or modify the contents of
all R4300 registers (except for some system control registers). Note
that these registers apply to the thread to which you are currently
attached. Switching threads with this view open will refresh it with
the register contents for the new thread. You can only examine and
modify the registers of a thread that is stopped.  memory view - As you
would expect, this is where you examine and modify memory contents. You
may specify the window origin by address or symbol. This window has two
modes. In single-word mode, it displays and modifies exactly one memory
word without touching any other locations.  This is the mode you would
use for dealing with memory mapped registers. In block mode, it
displays a block of memory from the specified starting address. The
size of the block is mostly determined by the size of the window on
your screen. Stretching the window gets you more memory to look at.
Shrinking it gets you less. You may specify the base in which you wish
memory to be displayed.

disassembly view - This view shows you memory contents as disassembled
code based on the current PC value, or else disassembled from some
address you specify.  The source line corresponding to the disassembled
memory is also displayed. There are a number of configuration options
for this window that let you customize it to the display that you find
most useful.  trap manager - this view shows you all breakpoints that
are set.  Breakpoints also show up in the source and disassembly
windows as pink lines. The current PC shows up as a green line.

The source view, which is the main view of tvd, consists of a set of
control buttons for running and stopping the selected thread, plus two
other windows. The source window (the middle portion of the view)
displays the source at the current PC (by default), and tracks the
program counter to keep it on screen whenever possible. You may set
breakpoints here by clicking in the margin to the left of the line at
which you wish to set the breakpoint.

The bottom of the source view is a small command line window where you
may enter commands and see the results. The mouse cursor must be in
this window to use it. This window is mostly used to examine data
objects like structures. For example, if you wish to look at a message
queue called audioMQ, you can enter "print audioMQ" and the contents of
the structure (including all its members) will be printed. Since the
compiler and debugger were designed to work together, the debugger has
quite good type information for displaying complex structures like
this. If you plan to use this window much, it is probably a good idea
to move the debugger higher on the screen and stretch the bottom down
to enlarge the command portion of the view. The default size is a bit
small. This window accepts most dbx commands, for those of you familiar
with this popular UNIX debugger.

The command window is also useful for setting breakpoints in functions
that are not on- screen because they are in a different source file.
While you can always change source files and set a breakpoint, it is
more convenient (providing you wish to stop at the start of a function)
to use the "stop in" command. If you know that you are trying to
isolate a problem in a function called sendDisplayList, then it is
probably best to type "stop in sendDisplayList" in the command window,
then click on Continue. This will run your application until any thread
enters the specified function. Note that encountering a breakpoint will
stop all threads with priorities in the user range (1..127). However,
the OS does not permit stopping a thread that is blocked on a message
queue.  These threads will remain in waiting state. They could run
again only if another thread sent them a message or if an interrupt
occurred. In general, interrupts are blocked while rmon is running,
except for the GIO interrupt which is used by rmon. Since no other
thread is running when a break has been hit, waiting tasks will not get
to run.

The Admin pull down also contains a few other useful items. First, this
is how you exit the debugger. You may also change to a different
executable here, but you should then do another Change Thread command.
There is a multithread view in this menu, which is useful to have
opened if you use more than one thread. It allows you to start and stop
threads as a group, and indicates whether a given thread is running or
stopped. If stopped, it shows you which function it was executing.

You may invoke the coprocessor view from the admin menu as well. The
coprocessor view brings up a second, smaller copy of the debugger. This
copy shows the source and resources of the RCP, while the main debugger
shows the R4300 state. There are fewer choices on the RCP Views menu.
The register view shows the scalar and vector registers of the RCP,
plus some system registers. The disassembly view shows IMEM contents as
disassembled RCP instructions. The source view is less useful for the
RCP, since it is programmed exclusively in its own assembly language.
Processor control (running , stopping, stepping) is best done from the
disassembly view. Memory and trap manager views are available here as
well.

Most users will find tvd to be fairly intuitive, especially if you have
used other source level debuggers. The on-line help should answer most
questions that arise in debugger operation.


Attachments:

=========
example.c
=========
#include <os.h>
#if defined( EHS )
#include "ehs.h"
#endif
#if defined( _EMULATOR )
#define printf emPrintf
#endif

#include "rmon.h"

#define VERBOSE 1

OSThread t1, t2;
OSMesgQueue mq1, mq2;
OSMesg msgs1[1], msgs2[1];

double appstack1[0x200], appstack2[0x200], mainThreadStack[0x200];

char * errorText = "Bad blurfl at";

void myHandler( int errno, int argc, char * errstring, int errval )
{
	printf( "My handler caught error %d, %s %x\n", errno, errstring, errval );
}

void fptest1( void * args )
{
	double d1, d2;
	int iteration = 0;
	OSMesg m;

	printf( "Starting FP thread 1\n" );
	d1 = 1.0;
	d2 = 2.0;
	for ( ;; ++iteration )
	{
		d1 += 0.0000001;
		d2 += 0.0000001;
		osRecvMesg( &mq1, OS_MESG_BLOCK );
#if VERBOSE
		printf( "Sum1 (%d) = %08x,%08x\n", iteration, d1 + d2 );
#endif
		osSendMesg( &mq2, &m, OS_MESG_BLOCK );
	}
}

void fptest2( void * args )
{
	double d1, d2;
	int iteration = 0;
	OSMesg m;

	printf( "Starting FP thread 2\n" );
	d1 = 3.0;
	d2 = 4.0;
	for ( ;; ++iteration )
	{
		d1 -= 0.0000001;
		d2 -= 0.0000001;
		osRecvMesg( &mq2, OS_MESG_BLOCK );
#if VERBOSE
		printf( "Sum2 (%d) = %08x,%08x\n", iteration, d1 + d2 );
#endif
		osSendMesg( &mq1, &m, OS_MESG_BLOCK );
	}
}

void mainproc( void * dummy )
{
	OSMesg m;
	OSThread rmonThread;
#if defined( EHS )
	ERRFUNPTR oldHandler;
#endif

	printf( "Mainproc\n" );
	osCreateThread( &rmonThread, rmonMain, (void *)0, rmonInitialSP,
		OS_IM_CART, (OSPri) 250 );
	osCreateThread( &t1, fptest1, 0, &appstack1[0x200], 0xc00, 5 );
	osCreateThread( &t2, fptest2, 0, &appstack2[0x200], 0xc00, 5 );
	osCreateMesgQueue( &mq1, msgs1, 1 );
	osCreateMesgQueue( &mq2, msgs2, 1 );
	osSendMesg( &mq1, &m, OS_MESG_BLOCK );
	osStartThread( &rmonThread );
#if defined( EHS )
	ehsReport( 42, 2, errorText, mq1 );
	oldHandler = ehsSetHandler( (ERRFUNPTR) myHandler );
	ehsReport( 42, 2, errorText, mq1 );
	ehsSetHandler( oldHandler );
	ehsReport( 42, 2, errorText, mq1 );
#endif
	printf( "Idle...\n" );

	/* become the idle thread */

	osSetIntMask( OS_IM_CART );
	osSetThreadPri( 0, 0 );
	for ( ; ; )
		;
}

void bootproc( void )
{
	OSThread mainThread;

	osInitialize();
	printf( "Bootproc\n" );
	osCreateThread( &mainThread, mainproc, (void *)0,
		&mainThreadStack[0x200], OS_IM_NONE, (OSPri) 127 );
	osStartThread( &mainThread );
}


=======
emutest
=======
#! /bin/sh

$ROOT/usr/etc/driverd &
DRIVERD_PID=$!

$ROOT/usr/sbin/emulate -S -k $$ ermon.tvd rom  &
EMULATE_PID=$!

$ROOT/usr/tv/bin/dbgif -kcdbr -e $EMULATE_PID 

kill -9 $! EMULATE_PID DRIVERD_PID
ipcrm -M $$
exit 0