Cactus Code Thorn Spawner
Authors    : Gabrielle Allen, Tom Goodale, Gerd Lanfermann, Thomas Radke
CVS info   : $Header$
--------------------------------------------------------------------------

This thorn spawns scheduled routines to new resources by creating a 
datafile and parameter file designed to run the routine, and then
informing a server.

At the moment this thorn requires IOUtil, IOHDF5 and PUGH

Useful parameters
-----------------

spawner::test = "yes" 
	- spawns and runs routines in main simulation so that output
	  can be compared
spawner::info = "debug"
	- says more about what it is doing


What it does
------------

By default, Spawner will try and spawn routines which are in the
CCTK_ANALYSIS timebin, and create parameter files such that when 
the simulation is rerun the spawned routine will be executed, along 
with scheduled functions in the timebins:

CCTK_STARTUP (note that this bin isn't called by CCTK_CallFunction)

CCTK_BASEGRID

CCTK_RECOVER_VARIABLES

CCTK_TERMINATE

This behaviour can be customised, to spawn or not spawn specific
routines and timebins, using the following string parameters:

Parameters to decide what to spawn:
----------------------------------

do_spawn_timebins ..... defaults to CCTK_ANALYSIS

do_spawn_routines ..... additional routines to spawn along with any in 
 		        the do_spawn_timebins

donot_spawn_routines .. these routines are never spawned

Parameters to decide how to run spawned simulation:
--------------------------------------------------

donot_run_routines .... don't run these scheduled functions

do_run_timebins ....... defaults to CCTK_BASEGRID, CCTK_RECOVER_VARIABLES,
                        CCTK_TERMINATE (CCTK_STARTUP is always run).


Output files and parameter files
--------------------------------

The data files for spawning are written to the main outdir specified 
in IOUtil, there is one file for each variable, with the naming convention

SPAWN<GENERATION>_<ROUTINE>_<VARIABLE>_<ITERATION>_3d.h5

The spawning parameter files are written into the top level directory with 
the naming convention

SPAWN<GENERATION>_<ROUTINE>_<ITERATION>.par

The spawning parameter files set the default output directory to be

SPAWN<GENERATION>_<ROUTINE>_<ITERATION>

The additional parameters written to the end of the parameter file are
e.g.

# Additions/changes for spawning (Thorn Spawner)
Cactus::cctk_itlast = 0
Cactus::cctk_initial_time = 0.000000
Spawner::generation = 1
IO::recovery_dir = "isco_small"
IO::recover_ID_files = " SPAWN0_evaltrK_coarse_dx_0_3d etc..."
IO::recover_ID_vars = "all"
Spawner::do_run_routines = "evaltrK"
IO::outdir = "SPAWN0_evaltrK_0"


Things to think on:
------------------

1. At the moment it doesn't work, because of the ActiveThorns group 
   in the written parameter file which isn't self consistent. For now 
   I'm not writing out the ActiveThorns in CCTKi_ParameterFileWrite

2. Because of the way memory is switched on, we write files containing
   the variables which are going to be calculated in the spawned simulation

3. We also output all the other variables, when we only really want to 
   output the variables which are associated with the spawned thorn.
   Maybe we just need to output variables in the trigger groups or
   something like this?