% *======================================================================* % Cactus Thorn template for ThornGuide documentation % Author: Ian Kelley % Date: Sun Jun 02, 2002 % $Header$ % % Thorn documentation in the latex file doc/documentation.tex % will be included in ThornGuides built with the Cactus make system. % The scripts employed by the make system automatically include % pages about variables, parameters and scheduling parsed from the % relevant thorn CCL files. % % This template contains guidelines which help to assure that your % documentation will be correctly added to ThornGuides. More % information is available in the Cactus UsersGuide. % % Guidelines: % - Do not change anything before the line % % START CACTUS THORNGUIDE", % except for filling in the title, author, date, etc. fields. % - Each of these fields should only be on ONE line. % - Author names should be separated with a \\ or a comma. % - You can define your own macros, but they must appear after % the START CACTUS THORNGUIDE line, and must not redefine standard % latex commands. % - To avoid name clashes with other thorns, 'labels', 'citations', % 'references', and 'image' names should conform to the following % convention: % ARRANGEMENT_THORN_LABEL % For example, an image wave.eps in the arrangement CactusWave and % thorn WaveToyC should be renamed to CactusWave_WaveToyC_wave.eps % - Graphics should only be included using the graphicx package. % More specifically, with the "\includegraphics" command. Do % not specify any graphic file extensions in your .tex file. This % will allow us to create a PDF version of the ThornGuide % via pdflatex. % - References should be included with the latex "\bibitem" command. % - Use \begin{abstract}...\end{abstract} instead of \abstract{...} % - Do not use \appendix, instead include any appendices you need as % standard sections. % - For the benefit of our Perl scripts, and for future extensions, % please use simple latex. % % *======================================================================* % % Example of including a graphic image: % \begin{figure}[ht] % \begin{center} % \includegraphics[width=6cm]{MyArrangement_MyThorn_MyFigure} % \end{center} % \caption{Illustration of this and that} % \label{MyArrangement_MyThorn_MyLabel} % \end{figure} % % Example of using a label: % \label{MyArrangement_MyThorn_MyLabel} % % Example of a citation: % \cite{MyArrangement_MyThorn_Author99} % % Example of including a reference % \bibitem{MyArrangement_MyThorn_Author99} % {J. Author, {\em The Title of the Book, Journal, or periodical}, 1 (1999), % 1--16. {\tt http://www.nowhere.com/}} % % *======================================================================* % If you are using CVS use this line to give version information % $Header$ \documentclass{article} % Use the Cactus ThornGuide style file % (Automatically used from Cactus distribution, if you have a % thorn without the Cactus Flesh download this from the Cactus % homepage at www.cactuscode.org) \usepackage{../../../../doc/latex/cactus} \begin{document} % The author of the documentation \author{Erik Schnetter \textless eschnetter@perimeterinstitute.ca\textgreater} % The title of the document (not necessarily the name of the Thorn) \title{WaveToyOpenCL} % the date your document was last changed, if your document is in CVS, % please use: % \date{$ $Date: 2004-01-07 14:12:39 -0600 (Wed, 07 Jan 2004) $ $} \date{May 15, 2012} \maketitle % Do not delete next line % START CACTUS THORNGUIDE % Add all definitions used in this documentation here % \def\mydef etc % Add an abstract for this thorn's documentation \begin{abstract} This thorn implements WaveToy, solving the scalar wave equation (in a Euclidean, i.e.\ trivial geometry). The thorn is implemented in OpenCL, with some wrapper code in C++. \end{abstract} % The following sections are suggestive only. % Remove them or add your own. \section{Introduction} This thorn \texttt{WavetoyOpenCL} solves the scalar wave equation, the same equation solved in thorn \texttt{WaveToy} and its companions written in other languages. Its major purpose is to serve as high-level example of using OpenCL in Cactus. It is purposefully written to be simple and easy to understand; for example, there are no parameters to choose different types of initial or boundary conditions. \section{Thorn Structure} We assume the reader is familiar with the structure of a Cactus thorn written e.g.\ in C or C++. An OpenCL thorn is slightly more complex because it (1) has to describe when and what data are moved between host and device, and (2) Cactus does not (yet?) support calling OpenCL code directly; some boilerplate code is necessary. \subsection{Schedule Declarations} Thorn \texttt{WaveToyOpenCL} relies on thorn \texttt{Accelerator} to handle data movement between host and device. This does not need to be managed explicitly; instead, the file \texttt{schedule.ccl} describes which routines are executed where (host or device), and which variables or groups are read or written. The location where a scheduled routine is ultimately executed needs to be described in a \texttt{Device=} schedule tag. The set of variables that are read and/or written needs to be declared in \texttt{READS} and \texttt{WRITES} schedule statements. For example, this is the schedule item for the evolution routine of thorn \texttt{WaveToyOpenCL}: \begin{verbatim} SCHEDULE WaveToyOpenCL_Evol AT evol { LANG: C TAGS: Device=1 WRITES: WaveToyOpenCL::Scalar } "Evolve scalar wave" \end{verbatim} This indicates that this routine executes on the device, i.e.\ its kernel is implemented in OpenCL\@. Note that, in OpenCL, both CPU and GPU count as \emph{devices} (thus every routine written in OpenCL counts as executing on a device, even if the device happens to be the CPU\@). This also indicates that this routine writes (i.e.\ defines) the grid function group \texttt{Scalar}, without looking at (the current timelevel of) this group. \subsection{Schedule Routines} Executing OpenCL code requires some boilerplate: One needs to choose an OpenCL platform and device, needs to compile the code (from a C string), needs to pass in arguments, and finally needs to execute the actual kernel code. Thorn \texttt{OpenCLRunTime} provides a simple helper routine for these tasks that can be used e.g.\ as follows: \begin{verbatim} char const *const groups[] = { "WaveToyOpenCL::Scalar", NULL}; int const imin[] = {cctk_nghostzones[0], cctk_nghostzones[1], cctk_nghostzones[2]}; int const imax[] = {cctk_lsh[0] - cctk_nghostzones[0], cctk_lsh[1] - cctk_nghostzones[1], cctk_lsh[2] - cctk_nghostzones[2]}; static struct OpenCLKernel *kernel = NULL; char const *const sources[] = {"", OpenCL_source_WaveToyOpenCL_evol, NULL}; OpenCLRunTime_CallKernel(cctkGH, CCTK_THORNSTRING, "evol", sources, groups, NULL, NULL, NULL, -1, imin, imax, &kernel); \end{verbatim} The function \texttt{OpenCLRunTime\_CallKernel} performs the following steps: \begin{enumerate} \item Choose a platform and device, compile the kernel code, and memoise (remember) the kernel for the next call \item Pass a set of grid functions to the kernel routine \item Parallelise the kernel over a certain set of grid points (this is similar e.g.\ to an OpenMP parallelisation, except that OpenCL devices may offer much more parallelism) \item Call the kernel \end{enumerate} Consequently, one needs to define the set of grid functions to be passed to the OpenCL kernel (\texttt{groups}, a C array terminated by NULL), needs to define the iteration bounds (\texttt{imin} and \texttt{imax}), and needs to provide the actual source code (\texttt{sources}, a C array terminated by NULL\@). Note that the first element of \texttt{sources} contains declarations (it is empty here), while the second element contains the actual kernel code (see thorn \texttt{OpenCLRunTime}). See thorn OpenCL how the string \texttt{OpenCL\_source\_WaveToyOpenCL\_evol} is generated from \texttt{.cl} files). The actual kernel is contained in the file \texttt{evol.cl}, and should be readable with some C knowledge. \texttt{LC\_LOOP3} is a macro that parallelises a loop, similar to the macros provided by thorn \texttt{LoopControl}. % \begin{thebibliography}{9} % % \end{thebibliography} % Do not delete next line % END CACTUS THORNGUIDE \end{document}