Development/Experimentation
From Soar Wiki
Often times researchers want to automate repeated runs of some Soar agent (stand alone or in the context of an environment), collecting data from each run for later analysis. Under Soar 8.5.x and earlier, a typical setup would work like this:
- Write a Tcl script to run Soar (and environment if necessary) in a loop. The script is specific to your particular agent/environment.
- Use Soar's log command or RHS functions to collect data.
- Post-process log using Tcl/Perl/C to produce a form suitable for loading in Excel.
- Use Excel to analyze data and create plots.
This page is primarily concerned with how to do the first two under Soar 8.6.x. The second two are up to you.
Contents |
Automating repeated Soar runs
Since Soar is no longer integrated with Tcl and has a new interface, old Tcl scripts you may have will not "just work" but it may be possible to convert them. We'll discuss that possibility below under Creating Mini Environments
Let's describe how you might do this from scratch. The general approach is to let some "program" external to Soar control the launching of the environment or debugger and the repetition, and for the environment or debugger to use a configuration file that tells it how to run. That is, the environment/debugger needs to support at least two capabilities:
- The ability to take a configuration file from the command line.
- The ability to automatically quit at the end of a run.
Launching and Looping
One of the easiest ways to repeatedly launch an environment is to use a batch file. This example runs "myenvironment" with the arg "-load config.file" 3 times:
FOR %%x IN (1 2 3) DO myenvironment -load config.file
In Tcl, the equivalent is:
for {set i 0} {$i<3} {incr i} { exec myenvironment -load config.file }
Running Stand Alone Agents
If you have a stand alone agent that you would just run in the debugger, you have at least two options: automate the launching of the debugger and running of your agent, or create a mini "environment" in Tcl/Java/C#/C++ to do the automation.
Automating Debugger Runs
The debugger can be instructed from the command line to source a file, like this:
java -jar SoarJavaDebugger.jar -source config.soar -quitonfinish
The -quitonfinish flag means the debugger will automatically quit when it finishes executing the contents of config.soar.
config.soar might have contents that look like this:
source myagent.soar watch 0 run
Thus, the debugger will start, load your agent, run, and then quit. To do this repeatedly you just need to use one of the looping techniques described above. For example, a batch file might look like this:
FOR %%x IN (1 2 3) DO java -jar SoarJavaDebugger.jar -source config.soar -quitonfinish
Creating Mini Environments
Using the debugger as described above might be too slow (the launching can take time) or not flexible enough (you might want to do something fancy). In that case, we recommend creating a mini "environment" that contains its own instance of Soar and can be controlled directly via the SML interface from any of the supported lanugages. The example given here is in Tcl. Those who want to convert old Tcl scripts can use this as an example.
#This script will run from SoarLibrary/bin (I use tclsh, started in that dir, to run)
#load the sml stuff
lappend auto_path .
package require tcl_sml_clientinterface
#Here is a print callback example.
#You probably won't need this, but you can do some fancy stuff with callbacks.
proc PrintCallback {id userData agent message} {
puts -nonewline $message
}
#this event loop periodically checks for new events from the kernel
# ensuring that they get executed on the client side
# (this is required for Tcl, but not for other languages that support threading properly)
proc CheckForEvents {k} {
$k CheckForIncomingCommands
after 50 CheckForEvents $k
}
#create an embedded kernel running in a the current thread
# (running in a new thread isn't supported in Tcl, but is what you probably want to do in other languages)
set kernel [Kernel_CreateKernelInCurrentThread SoarKernelSML]
#start event loop (again, not necessary in other languages if starting kernel in new thread)
CheckForEvents $kernel
#this is a loop that creates an agent, loads some productions, runs, and destroys the agent
# you'll know each time the agent goes through the loop because the "Success!" message will be written out
# (it gets printed because we register a print callback)
for {set i 0} {$i<3} {incr i} {
#create an agent named Soar1
set agent [$kernel CreateAgent Soar1]
#this demonstrates registering the print callback defined above (let's you see output from Soar)
set printCallbackId [$agent RegisterForPrintEvent $smlEVENT_PRINT PrintCallback ""]
#load productions
set result [$agent LoadProductions ../demos/towers-of-hanoi/towers-of-hanoi.soar]
#excise the monitor production
set result [$kernel ExecuteCommandLine "excise towers-of-hanoi*monitor*operator-execution*move-disk" Soar1]
#set the watch level to 0
set result [$kernel ExecuteCommandLine "watch 0" Soar1]
#run the agent. Note that this call will return when the agent stops, e.g. when it halts, even though it says forever
set result [$agent RunSelfForever]
#this is how you would init-soar
set result [$kernel ExecuteCommandLine "init-soar" Soar1]
#this is how you destroy an agent
$kernel DestroyAgent $agent
}
#this last part makes sure the script exits properly. Otherwise it may appear to hang.
#shutdown the kernel; this makes sure agents are deleted and events fire correctly
$kernel Shutdown
#delete kernel object itself
set result [$kernel -delete]
Running Agents in an Environment
As noted above, the current approach to controlling environments is to launch them externally using a batch file or other means, and to control the run using a configuration file. That is, your environment needs to support a configuration file. If it doesn't, then you'll need to come up with some other means of controlling it (e.g. direct integration with the environment, which was the old Soar way).
Let's discuss how one would control Java Eaters and Java TankSoar. Both of these environments support configuration files: for Eaters you can specifiy a .seaters file, and for TankSoar you can specify a .stank file. These files allow one to specify things like the starting map, the number of agents, their productions, and preferred agent starting locations. See the example seaters and stank files in the Applications/JavaEaters and Applications/JavaTankSoar directories for the details.
Specifically to facilitate repeated runs, Java Eaters and Java TankSoar were designed to be runnable without any graphics. To do this, use the -w off switch on the command line. Thus, one might run TankSoar like this:
java -jar tanksoar.jar -w off config.stank
You can do this repeatedly using one of the looping techniques described above. For example, using a batch file to run 3 times would look like this:
FOR %%x IN (1 2 3) DO java -jar tanksoar.jar -w off config.stank
SoarSim
Documentation for an experiment framework in the works is here.
Logging Data From Soar Runs
Logging is very application specific. That is, it is very difficult to make a general tool that will log everything that everyone will want to log. This is especially true when you consider that some people will want to log information related to custom kernel modifications or internal information that isn't generally available to clients.
Thus, our approach to logging is to provide some general logging support (such as has been available in previous versions of Soar), but also to provide examples of simple custom logging clients that users can modify to suit their needs (e.g. to log only exactly what they need). In general, the approach is to register for callbacks that provide the information you need. Thus, one would primarily register for XML trace event to get trace info, but one could also register for any arbitrary callback to get other information. Since XML is the primary method for getting information to log we will also provide some kernel-level convenience functions to output custom XML for those that want to log arbitrary things from the kernel (functions already exist for outputting XML from the kernel, but are not as easy to use as they could be).
Logging from the Debugger
In the debugger, right-click on the window you want to log the contents of and select "Log this window..." The generated log file will be plain-text, and everything at the given watch level will be logged.
Example Logging in C++
The LoggerWinC project (in the SML.sln) is a Windows-specific example logger. Logger.cpp shows how to log using an XML callback handler. It specifically logs states and operators. This is just a client that can be connected to a running Soar agent.
Example Logging in Java
This is the Java analog of the C++ logger.
