Makeflow is Copyright (C) 2009 The University of Notre Dame. This software is distributed under the GNU General Public License. See the file COPYING for details.
You can run a Makeflow on your local machine to test it out. If you have a multi-core machine, then you can run multiple tasks simultaneously. If you have a Condor pool or a Sun Grid Engine batch system, then you can send your jobs there to run. If you don't already have a batch system, Makeflow comes with a system called Work Queue that will let you distribute the load across any collection of machines, large or small.
Makeflow is part of the Cooperating Computing Tools. You can download the CCTools from this web page, follow the installation instructions, and you are ready to go.
Makeflow attempts to generate all of the target files in a script. It examines all of the rules and determines which rules must run before others. Where possible, it runs commands in parallel to reduce the execution time.
Here is a Makeflow that uses the convert utility to make an animation. It downloads an image from the web, creates four variations of the image, and then combines them back together into an animation. The first and the last task are marked as LOCAL to force them to run on the controlling machine.
CURL=/usr/bin/curl CONVERT=/usr/bin/convert URL=http://www.cse.nd.edu/~ccl/images/capitol.jpg capitol.montage.gif: capitol.jpg capitol.90.jpg capitol.180.jpg capitol.270.jpg capitol.360.jpg LOCAL $CONVERT -delay 10 -loop 0 capitol.jpg capitol.90.jpg capitol.180.jpg capitol.270.jpg capitol.360.jpg capitol.270.jpg capitol.180.jpg capitol.90.jpg capitol.montage.gif capitol.90.jpg: capitol.jpg $CONVERT $CONVERT -swirl 90 capitol.jpg capitol.90.jpg capitol.180.jpg: capitol.jpg $CONVERT $CONVERT -swirl 180 capitol.jpg capitol.180.jpg capitol.270.jpg: capitol.jpg $CONVERT $CONVERT -swirl 270 capitol.jpg capitol.270.jpg capitol.360.jpg: capitol.jpg $CONVERT $CONVERT -swirl 360 capitol.jpg capitol.360.jpg capitol.jpg: $CURL LOCAL $CURL -o capitol.jpg $URLNote that Makeflow differs from Make in a few important ways. Read section 4 below to get all of the details.
% makeflow example.makeflowNote that if you run it a second time, nothing will happen, because all of the files are built:
% makeflow example.makeflow makeflow: nothing left to doUse the -c option to clean everything up before trying it again:
% makeflow -c example.makeflowIf you have access to a batch system running SGE, then you can direct Makeflow to run your jobs there:
% makeflow -T sge example.makeflowOr, if you have a Condor Pool, then you can direct Makeflow to run your jobs there:
% makeflow -T condor example.makeflowTo submit Makeflow as a Condor job that submits more Condor jobs:
% condor_submit_makeflow example.makeflowYou will notice that a workflow can run very slowly if you submit each batch job to SGE or Condor, because it typically takes 30 seconds or so to start each batch job running. To get around this limitation, we provide the Work Queue system. This allows Makeflow to function as a master process that quickly dispatches work to remote worker processes.
To begin, let's assume that you are logged into a machine named barney.nd.edu. start your Makeflow like this:
% makeflow -T wq example.makeflowThen, submit 10 worker processes to Condor like this:
% condor_submit_workers barney.nd.edu 9123 10 Submitting job(s).......... Logging submit event(s).......... 10 job(s) submitted to cluster 298.Or, submit 10 worker processes to SGE like this:
% sge_submit_workers barney.nd.edu 9123 10Or, you can start workers manually on any other machine you can log into:
% work_queue_worker barney.nd.edu 9123Once the workers begin running, Makeflow will dispatch multiple tasks to each one very quickly. If a worker should fail, Makeflow will retry the work elsewhere, so it is safe to submit many workers to an unreliable system.
When the Makeflow completes, your workers will still be available, so you can either run another Makeflow with the same workers, remove them from the batch system, or wait for them to expire. If you do nothing for 15 minutes, they will automatically exit.
Note that condor_submit_workers and sge_submit_workers are simple shell scripts, so you can edit them directly if you would like to change batch options or other details.
# This is an incorrect rule. output.txt: ./mysim.exe -c calib.data -o output.txtHowever, the following is correct, because the rule states all of the files needed to run the simulation. Makeflow will use this information to construct a batch job that consists of mysim.exe and calib.data and uses it to produce output.txt:
# This is a correct rule. output.txt: mysim.exe calib.data ./mysim.exe -c calib.data -o output.txtWhen a regular file is specified as an input file, it means the command relies on the contents of that file. When a directory is specified as an input file, however, it could mean one of two things. First, the command depends on the contents inside the directory. Second, the command relies on the existence of the directory (for example, you just want to add more things into the directory later, it does not matter what's already in it). Makeflow assumes that an input directory indicates that the command relies on the directory's existence.
When using Condor, this string will be added to each submit file. For example, if you want to add Requirements and Rank lines to your Condor submit files, add this to your Makeflow:
BATCH_OPTIONS = Requirements = (Memory>1024)
When using SGE, the string will be added to the qsub options. For example, to specify that jobs should be submitted to the devel queue:
BATCH_OPTIONS = -q devel
% makeflow -D example.makeflow | dot -T gif > example.gif
% makeflow -T wq -p 9567 exmaple.makeflow
% makeflow -T wq -a -N MyProj example.makeflowThe -N option gives the master a project name called 'MyProj'. The -a option enables the catalog mode of the master. Only in the catalog mode a master would advertise its information, such as the project name, running status, hostname and port number, to a catalog server. Then a worker could retrieve these information from the same catalog server. The above command uses the default catalog server at Notre Dame which runs 24/7. We will talk about how to set up your own catalog server later.
To start a worker that automatically finds MyProj's master via the default Notre Dame catalog server:
% work_queue_worker -a -N MyProjThe '-a' option enables the catalog mode on the worker, which tells the worker to contact a catalog server to find out a project's (specified by -N option) hostname and port.
You can also give multiple -N options to a worker. The worker will find out which ones of the specified projects are running from the catalog server and randomly select one to work for. When one project is done, the worker would repeat this process. Thus, the worker can work for a different master without being stopped and given the different master's hostname and port. An example of specifying multiple projects:
% work_queue_worker -a -N proj1 -N proj2 -N proj3
Now let's look at how to set up your own catalog server. Say you want to run your catalog server on a machine named barney.nd.edu (the default port that the catalog server will be listening on is 9097, you can change it via the '-p' option), do:
barney% catalog_serverNow you have a catalog server listening at barney.nd.edu:9097. To make your masters and workers contact this catalog server, simply add the '-C hostname:port' option to both of your master and worker:
% makeflow -T wq -C barney.nd.edu:9097 -N MyProj example.makeflow % work_queue_worker -C barney.nd.edu:9097 -a -N MyProj