ClearSilver Template Engine

ClearSilver is a fast, powerful, and language-neutral template system. It is designed to make it easy to create template driven static or dynamic websites.
This module implements the ClearSilver API in an easy-to-use BlitzMax style.

Features of ClearSilver :

• Fast, simple, language-neutral template system.
• Powerful template language supports variable substitution, template includes, conditionals, loops, macros, recursive macros, local variables.
• Full CGI-handling kit with support for CGI/1.1, RFC 2388, automatic gzip compression, and other CGI standards.
• "Pipelined" rendering model stores rendering data in an intermediate static dataset and makes debugging easier.
• Separate your web application from its appearance.
• Allow non-programmers access to edit parts of the site appearance.

Templating Guide

The following guide is based on the documentation available at http://www.clearsilver.net/docs/.

The dataset and HDF Files

It is often convenient to load elements of a dataset from disk. Normally data is loaded from a ClearSilver file format called HDF, although it could easily be loaded from XML or another format. HDF stands for Hierarchical Data Format (yes, another HDF). It supports two different syntax for representing the dataset hierarchy which can be intermixed arbitrarily. The first is a simple dotted path scheme:

Page.Name = My Index
Page.URL = /myindex.html
Page.Menu.0 = Home
Page.Menu.1 = Preferences
Page.Menu.2 = Help
Page.Menu.3 = Support

Page {
  Name = My Index
  URL = /myindex.html
  Menu {
    0 = Home
    1 = Preferences
    2 = Help
    3 = Support
  }
}

Page {
  Menu {
    0 {
      Name = Home
      URL = /
    }
    1 {
      Name = Preferences
      URL = /prefs
    } 
    2 {
      Name = Help
      URL = /help.html
    }
    3 {
      Name = Support
      URL = /feedback/
    }
  }
}

When a ClearSilver template is rendering, it can reference specific variables in the dataset or iterate over all of the elements at a specific point in the dataset hierarchy. For example, a CS template which was rendering the HDF dataset above might iterate over Page.Menu, rendering each menu item's .Name and .URL elements.

There are two more important parts of the HDF file syntax.

1. A name can copy the contents of another name by using a colon instead of an equals sign. For example,

   Page.Name : Page.Menu.0.Name

   This means that Page.Name is the same as Page.Menu.0.Name. Note that Page.Menu.0.Name has to be defined before you can copy it into Page.Name.
2. A name can be set to a multi-line string value. This uses a syntax similar to standard UNIX shell/Perl syntax. For example:

   Page.Name << EOM
   This is my multi-line page name.
   Isn't it spiffy?
   EOM
   

ClearSilver templates

The template commands are:
   Substitution: var, evar, lvar, include, linclude, set, name
   Flow Control: if, else, elif, alt
   Iteration: each, loop, with
   Macros: def, call

All Flow Control, Iteration, and the def command have a corresponding ending command, just like HTML. So if has a corresponding /if.

Most commands take one or more expresssions as arguments.

In addition, Clearsilver supports comments using the # (hash) character, ie: <?cs # this is a comment ?>

Substitution

Simple variable substitution is done with: <?cs var:Page.Name ?> Ie, the command is var and the parameter is the name of the var, ie Page.Name.

evar is like var, except that the value of the dataset variable is also parsed for ClearSilver commands. This evar parsing happens during the load and parse of the CS template. As a result, syntax errors will be reported before display begins, and evar can not be used to display elements within an each loop.

lvar is like evar, except that the value of the dataset variable is parsed at render time, not parse time. Syntax errors will be reported at render time. If you are not buffering your output, this will cause a partial page render. The CGI Kit always buffers its output.

name will substitute the name of the lowest part of the dataset, ie <?cs name:Page.Name ?> will result in Name. This is most useful during iteration or macro expansion, where the true name of the aliased variable could have specific meaning. (pointers to later examples). There is an equivalent built-in function which does the same thing, such that <?cs name:Page.Name ?> is equivalent to <?cs var:name(Page.Name) ?>. Note also that when used on a local variable, this will return the name of the variable the local variable is mapped to.

include will include (and parse) another CS file. The parameter to include can be either a string or a variable. A string is denoted by double quotes, ie: <?cs include:"header.cs" ?> The file is searched for in the HDF search path. Note: because this include happens at load/parse time, you can not use a local variable which is part of an each expression. This also means that using flow control (if/else) will have no affect on whether or not the file is loaded and parsed, it will always be loaded and parsed.

linclude is similar to include, but the loading/parsing occurs at render time. local variables can be used to specify the filename, and conditional variables can be used to prevent loading of the file. Local variables are not passed to the linclude file, however. Like lvar, errors will be reported at render time.

set will set a value in the dataset. The syntax for a set is fairly complicated. In general, the set command should be used only where necessary: most of the dataset should be specified either in static HDF files, or in the CGI via the HDF api. Typically, the set command is used for formatting, ie for splitting a set of elements into a number of columns, or for determining whether there were any matches during an iteration. (pointers to later examples)

A set command consists of the HDF value you are setting, and an expression that is evaluated to determine the value. For example: <?cs set:Page.Title = "The Title is " + Page.Menu.0.Name ?>

Flow Control

ClearSilver provides if/elif/else commands for flow control. The argument to if and elif is an expression, evaluated as a boolean. So, the following if command always evaluates to true:

<?cs if:#1 ?>
<?cs /if ?>

alt is short hand for an if var else. If the argument to alt is true, it is displayed, otherwise everthing to the closing /alt is displayed. These two are identical:

  <?cs alt:my_text ?>There is nothing to see here<?cs /alt ?>
  <?cs if:my_text ?><?cs var:my_text<?cs else ?>There is nothing to see here<?cs /if ?>

Iteration

As part of the "safety" of running rendering a CS macro, the only iteration supported is guaranteed to be finite. The iteration command is each. each iterates over all of the children of a node of the dataset. For example, consider the following dataset:

Page {
  Access = Public
  Content = myword.cs
  Menu {
    0 {
      Name = Home
      URL = /
    }
    1 {
      Name = Preferences
      URL = /prefs
    } 
    2 {
      Name = Help
      URL = /help.html
    }
    3 {
      Name = Support
      URL = /feedback/
    }
  }
}

<?cs each:item = Page.Menu ?>
  <?cs name:item ?> - <a href="<?cs var:item.URL ?>">
        <?cs var:item.Name ?></a><br>
<?cs /each ?>

  0 - <a href="/">Home</a><br>
  1 - <a href="/prefs">Preferences</a><br>
  2 - <a href="/help.html">Help</a><br>
  3 - <a href="/feedback/">Support</a><br>

Note also the use of the name command to access the name of the child that the local variable is pointing to.

with is similar to the Pascal with operand. It is similar to each, except instead of iterating over the variables, it allows access to that variable by a different variable name. Conceptually, you can also think of this as a scoped variable pointer. This is usually most convenient when you have longer variable names consisting of a complex syntax, for instance:

<?cs with:item = Page.Menu[Query.foo - #1] ?>
  <?cs name:item ?> - <a href="<?cs var:item.URL ?>">
        <?cs var:item.Name ?></a><br>
<?cs /with ?>

  2 - <a href="/help.html">Help</a><br>

loop is used to create a numeric loop. The arguments to loop are the starting number, the ending number, and the step. ClearSilver will always evaluate the loop expression to guarantee that it will finish. For instance, if you use a negative step value, but the ending number is greater than the starting number, ClearSilver will automatically switch the starting and ending numbers. The expressions are only evaluated once, so changing the variables in the loop will have no affect. Some quick examples:

<?cs loop:x = #1, #5, #2 ?><?cs var:x ?>, <?cs /loop ?>
1, 3, 5

<?cs loop:x = #1, #205, #2 ?><?cs var:x ?>, <?cs /loop ?>
1, 3, 5... 205

backwards
<?cs loop:x = #205, #1, "-2" ?><?cs var:x ?>, <?cs /loop ?>
205, 203, 201, ... 1

Template Expressions

Some example expressions:

  Page.Title
  Page["Title"]
  Page[varname]
  Page["Title"] == "Home"
  (#Page.Count > #1) || (?Page.Next)

• A number. A number is any string that can be converted to a number, or you can force a string to be a number using the # operator.
  Examples include: 103, 0x1a, -23, +14, #83
• A String. A string is any set of characters within either single quotes or double quotes. There is currently no escaping mechanism.
• A Variable. A variable is either a reference to a local variable (as created by call/def, each, with, loop) or a reference to the global HDF dataset. If your variable name is a number, you must use the $ operator to force variable evaluation.
• Numeric Variable. This is a variable value converted to a number.

ClearSilver expressions have four different evaluation types: variable, string, numeric and boolean. Which evaluation is used depends on the types involved and the operator. Operators are either boolean, numeric, or numeric/string. For numeric/string operators, if either argument is a number, then they are evaluated as a number. Otherwise, the string version of the operator is used. The only really different operator is +, which is numeric addition and string concatenation.

Evaluation as Boolean:

• Number: 0 is false, everything else is true
• String: Empty string is false. String "0" is false. Everything else is true.
• Variable: Non-existing variable is false. Existing variable is treated the same as a string.
• Numeric Variable: Non-existing is 0. Doesn't convert to a number is treated as a 0. Otherwise, converted to a number. Evaluated same as a Number.

• Number: as a number
• String: Converted to a number. If its not a valid number, than it will be converted to 0.
• Variable: If it doesn't exist, its 0. Otherwise, treated the same as a string.
• Numeric Variable: Same as Variable.

• Number: a number in base 10
• String: As String
• Variable: Non-existing variable is the empty string.
• Numeric Variable: Converted to number, than converted to string in base 10.

The list of operators, from low to high precedence:

Operator	Operation	Evaluation Type
,	C Comma Operator	n/a
||	Boolean OR	Boolean
&&	Boolean AND	Boolean
==	Equal	String/Numeric
!=	Not Equal	String/Numeric
>	String Greater Than	String/Numeric
>=	Greater Than/Equals	String/Numeric
<	Less Than	String/Numeric
<=	Less Than/Equals	String/Numeric
+	String Concat / Add	String/Numeric
-	Subtract	Numeric
*	Multiply	Numeric
/	Divide	Numeric
%	Modulo	Numeric
+	(unary) Positive	Numeric
-	(unary) Negative	Numeric
#	(unary) Force Numeric	Numeric
$	(unary) Force Variable	Variable
!	(unary) Boolean NOT	Boolean
?	(unary) Existance	Boolean
.	Descend Variable Name	Variable
[ ]	Expand Variable Name	Variable
( )	Function Call	n/a

Addition does not force numeric evaluation. During string evaluation, + represents string concatination.

Here is an example of rendering data into 2 columns in an HTML table:

<table>
  <tr>
    <th>Column 1</th><th>Column 2</th>
  </tr>
  <?cs set:count = #0 ?>
  <?cs each:item = Page.Items ?>
    <?cs if:count % #2 ?>
      <tr>
    <?cs /if ?>
      <td><?cs var:item.Number ?> - 
        <?cs var:item.Name ?></td>
    <?cs set:count = count + #1 ?>
    <?cs if:count % #2 ?>
      </tr>
    <?cs /if ?>
  <?cs /each ?>
</table>

Macros

ClearSilver also supports the concept of macros. There are two commands for this support, def which defines the macro, and call which issues the call. Here is an example macro definition. This macro takes the value of one variable, and walks a part of the dataset to expand that value into a string value.

<?cs def:map_val(val, map) ?>
  <?cs each:item = map ?>
    <?cs if:val == item ?>
      <?cs var:item.val ?>
    <?cs /if ?>
  <?cs /each ?>
<?cs /def ?>

Lang.Dates {
  Weekdays {
    0 = 0
    0.val = Sunday
    1 = 1
    1.val = Monday
    2 = 2
    2.val = Tuesday
    3 = 3
    3.val = Wednesday
    4 = 4
    4.val = Thursday
    5 = 5
    5.val = Friday
    6 = 6
    6.val = Saturday
  }
}

<?cs call:map_val(#6, Lang.Dates.Weekdays) ?>

<?cs def:map_val(val, map) ?><?cs each:item = map 
    ?><?cs if:val == item 
      ?><?cs var:item.val ?><?cs 
      /if ?><?cs 
    /each ?><?cs 
/def ?>

You can also see from this example, that macro arguments can be any of the four types of arguments. Currently, if you were to pass a string or numeric value as the item parameter, ie a parameter which is expected to be a part of the dataset, weird things will happen. In some cases, it will always evaluate to the value, ie if you pass item as "wow", and ask for item.foo, you'll get "wow". If you ask for the children of item, it will think you are crazy, and instead assume that you meant item as a global variable, and look it up in the dataset under that name.

Expression Functions

Function	Arguments	Description
subcount(var)	An HDF variable	Returns the number of child nodes for the HDF variable
name(local)	A local variable	Returns the HDF variable name for a local variable alias
first(local)	A local variable	Returns true iff the local variable is the first in a loop or each
last(local)	A local variable	Returns true iff the local variable is the last in a loop or each
abs(expr)	A numeric expression	Returns the absolute value of the numeric expressions
max(expr, expr)	Two numeric expressions	Returns the larger of two numeric expressions
min(expr, expr)	Two numeric expressions	Returns the smaller of two numeric expressions
string.slice(expr, start, end)	A string expression, and two numeric expressions	Returns the string slice starting at start and ending at end, similar to the Python slice operator
string.find(string, substr)	Two string expressions	Returns the numeric position of the substring in the string (if found), otherwise returns -1 similar to the Python string.find method
string.length(expr)	A string expression	Returns the length of the string expression
_(expr)	A string expression	Only available if compiled with gettext support, returns the translated version of the string expression as returned by gettext()

Template Filters

url_escape	This URL encodes the string. This converts characters such as ?, &, and = into their URL safe equivilants using the %hh syntax.
html_escape	This HTML escapes the string. This converts characters such as >, <, and & into their HTML safe equivilants such as &gt;, &lt;, and &amp;.
js_escape	This Javascript escapes the string so it will be valid data for placement into a Javascript string. This converts characters such as ", ', and \ into their Javascript string safe equivilants \", \', and \\.
text_html	This pretty-formats normal text into an HTML fragment, attempting to detect paragraph boundaries and allowing it to wrap reasonably.
html_strip	This removes all HTML tags and then converts any & based HTML escaped data into normal text. Combine this with html_escape() if you would like to strip the HTML tags from text and display the result in an HTML safe way.

These filters can be used anywhere in an expression. This makes them extremely useful for composing URLs or forcing data to be HTML safe. Here are some examples:

<?cs var:html_escape(Page.Title) ?>
<?cs set:url = "http://www.google.com/q=" + url_escape(Query.q) ?>

<IMG onclick="handleClick('<?cs var:js_escape(url)')" SRC="foo.gif">

<A HREF="/newurl?_done=<?cs var:url_escape(url) ?>">click here</A>

CGI Kit

The ClearSilver CGI kit provides an interface to common CGI interface which integrates with the ClearSilver template language. It preloads CGI information into your ClearSilver dataset, making that information available in a standard location in the HDF dataset.The ClearSilver CGI kit also supports RFC 2388 file uploads via HTTP, and it is one of the only interfaces which allows your CGI to retrieve upload status information via a callback.

1. Preloaded HDF Data

Several different types of data are preloaded into the HDF dataset by the Clearsilver CGI kit. This allows Clearsilver template code direct access to information such as query parmaters and Cookies. This also makes it easy for your application code to get this data from one place (the HDF context) with a single API. Because this is a terribly useful utility, environment which operate without the Clearsilver CGI kit may wish to emulate this behavior.

1.1. HTML Form Variables

The HTML form GET or POST variables are pre-populated into the HDF dataset at Query.*. If a paramater is specified multiple times, then the element will have subelements including the different values provided. The base variable will retain the last value provided. Here is an example of an HDF Query.* tree that might result from an HTML form:

Query.name = David
Query.email = david@neotonic.com
Query.multSelect = third value
Query.multSelect.0 = first value
Query.multSelect.1 = second value
Query.multSelect.2 = third value

1.2 Cookies

The HTTP Cookie data is provided, along with decoded versions of individual cookies. Here is an example of the HDF Cookie.* tree that might result:

Cookie = help_pos_x=461; help_pos_y=206; E=v=1&e=305,1,303; 
Cookie.help_pos_x = 461
Cookie.help_pos_y = 206
Cookie.E = v=1&e=305,1,303

1.3. HTTP headers

The following HTTP headers are mapped into HDF:

HTTP.Accept = image/gif, image/x-xbitmap, image/jpeg, image/pjpeg
HTTP.AcceptEncoding = gzip, deflate
HTTP.AcceptLanguage = en-us
HTTP.Cookie = help_pos_x=461; help_pos_y=206; E=v=1&e=305,1,303; 
HTTP.Host = jeske-dev.trakken.com
HTTP.UserAgent = Mozilla/4.0 (compatible; MSIE 6.0;)
HTTP.Referer = http://www.clearsilver.net/cgi-bin/test.py

1.4. Apache Environment

CGI.DocumentRoot = /home/jeske/htdocs
CGI.GatewayInterface = CGI/1.1
CGI.QueryString = done=1
CGI.RemoteAddress = 216.103.193.234
CGI.RemotePort = 1870
CGI.RequestMethod = GET
CGI.RequestURI = /cgi-bin/test.py?done=1
CGI.ScriptFilename = /home/jeske/htdocs/cgi-bin/test.py
CGI.ScriptName = /cgi-bin/test.py
CGI.ServerAddress = 64.167.132.245
CGI.ServerAdmin = webmaster@neotonic.com
CGI.ServerName = www.clearsilver.net
CGI.ServerPort = 80
CGI.ServerProtocol = HTTP/1.1
CGI.ServerSoftware = Apache/1.3.26 (Unix)

2. HTTP Upload

The CGI Kit also provides a facility for parsing of HTTP POST data, including multi-part file upload. It has a unique callback interface which allows you to retrieve constant status information about the progress of an ongoing upload. Handling the POST data occurs when you call parse(). You must have already setup the upload callback, either by calling setUploadCB() from a wrapper, or by setting the upload_cb value of the CGI structure from C/C++.

After parse has completed, the Query.* HDF value for your INPUT TYPE=FILE paramater will contain the filename of the file uploaded. You then provide the name of your form value to the fileHandle() method to get a FILE * for an uploaded file.

3. Utility Functions

Utility functions are provided to handle redirection, as well as setting cookies. Refer to the reference for information about how to use these functions.

The following HDF variables are used by the ClearSilver CGI Kit:

Changing the output

cgiout.Location, cgiout.Status, cgiout.ContentType, cgiout.other.*

This variable tree is used to send extra HTTP & CGI headers during display(). Location, Status and ContentType are the known values. These are mapped directly to the Location, Status and Content-Type headers. display() will also walk the cgiout.other.* vars, using the value as output. Ie, cgiout.other.encoding = Content-Encoding: gzip will add a Content-Encoding: gzip header to the output.

Config.NoCache

If Config.NoCache is set, then the following headers will be added during display():

  Pragma: no-cache
  Expires: Fri, 01 Jan 1990 00:00:00 GMT
  Cache-control: no-cache, must-revalidate, no-cache="Set-Cookie", private
  

These headers are designed to attempt to prevent all browsers and proxies from caching the page. This is generally a very evil thing to do, as it means any page render, including the user pressing the back button, for some browsers this will also include printing, viewing page source, etc.

Config.CompressionEnabled

If this is set to a positive integer, and the output is HTML, then display will gzip compress the output for browsers which support it.

Config.TimeFooter

By default, display will add an "execution time" html comment to the end of the rendered page if the page output type is text/html. The format of the comment is <!-- 0.5:1 --> where the first number is the total time between the call to cgi-init and the call to display, and the number after the colon is 1 if the output is compressd, and 0 if not.

Config.WhiteSpaceStrip,

This variable indicates the amount of whitespace stripping that is applied to html output in display. ClearSilver has a tendency to add lots of whitespace to your output unless you are very careful with the formatting of your clearsilver commands. This is because all whitespace is included in the output, which includes and space you use to indent your clearsilver commands, and any newlines you add. There are three levels available. 0 is no whitespace stripping. 1 is a somewhat "minimal" whitespace stripping that attempts to just remove extra lines but leave your formatting mostly the same. 2 is designed to strip all extra whitespace. The default is level 1. If you are compressing your output, you probably don't need to remove the whitespace.

Configuration

Config.Upload.TmpDir

Where the PUT handler and the RFC2388 multipart/form-data POST handler create temporary files that are uploaded to the server. The default value is /var/tmp.

Config.Upload.Unlink

By default, the PUT handler, and the RFC2388 multipart/form-data POST file handler unlink the files they create in parse as soon as they create them, but keep the file handle open to the files. This means that as soon as the files are closed, they will be removed. If the CGI prematurely exits, the files will automatically be removed. If this is set to 0, the files will not be unlinked immediately. This would allow you to access the files directly from your CGI, instead of only accessing them through the existing filehandle returned by filehandle. The files will be in Config.Upload.TmpDir and be named cgi_upload.XXXXXX as created by mkstemp(). The files will be removed in cgi_destroy(), but if the CGI exits prematurely, the files will not be removed. You will need to have a periodic job cleanup old files to prevent your filesystem from filling up.

CookieAuthority.*

A cookie authority is the domain for which a cookie is said to be valid. Ie, if your CGI is running on www.neotonic.com, and you want your cookie to be valid for all neotonic.com domains, the authority is .neotonic.com. If your CGI is responding to multiple domains, the easiest way to handle this is to have a list of domain authorities, and use the cookieAuthority call to determine what domain authority to use in the cookieSet call. The CookieAuthority.* variables each contain a single authority, ie:

    CookieAuthority.0 = .neotonic.com
    CookieAuthority.1 = .neotonic.co.uk
    CookieAuthority.1 = .neotonic.jp

So, the cookie will be valid on www.neotonic.com and www1.neotonic.com.

Config.TagStart

This variable is not specific to the CGI Kit. You can set this variable to change the clearsilver command. By default, all clearsilver commands are enclosed in <?cs ... ?>. You can replace the cs portion of this with another tag using the Config.TagStart variable.

Config.VarEscapeMode

This variable is not specific to the CGI Kit. This variable must be set before the call to cs_init() (which is done in display() if using the CGI Kit). This variable sets the default escape mode for all var: commands. Currently, possible values are none, script, url, and html.

Debug Variables

Query.debug, Config.DebugPassword, Config.DebugEnabled

If Config.DebugEnabled is set to 1, and if Query.debug is non-empty and matches Config.DebugPassword, then display with add debug output to the end of the rendered page. This output is subject to change, but currently includes the environment variables and the full HDF dump.

Query.debug, Config.DumpPassword, Config.DebugEnabled

If Config.DebugEnabled is set to 1, and if Query.debug is non-empty and matches Config.DumpPassword, then display will dump the HDF and the CS parse tree as text/plain instead of rendering the page.

Query.debug_pause, Config.DebugPassword, Config.DebugEnabled

If Config.DebugEnabled is set to 1, and if Query.debug_pause is non-empty and matches Config.DebugPassword, then cgi_init will call sleep(20). This will cause the CGI to pause for 20 seconds, allowing you to attach to the running CGI will a debugger, strace, truss, or other debugging aid.

Query.xdisplay, Config.Displays.*, Config.Debugger

Used during cgi_init(). If you specify a form value of xdisplay=yourmachine:0 to a CGI, cgi_init() check your display against the list of allowed displays in Config.Displays (ie, Config.Displays.0 = yourmachine:0, and if it matches, it will launch the debugger specified by Config.Debugger against the currently running CGI. This allows you to use an X based debugger on a currently running CGI. The default Config.Debugger is

/usr/local/bin/sudo /usr/local/bin/ddd -display %s %s %d

Where the first %s will get the value of Query.xdisplay, the second %s gets the argv[0] (ie, the path to the currently running program) and the %d is the PID of the running program.

Extra Variables

There are also a couple of useful extra variables you can use to change the rendering :

ClearSilver.DisplayDebug

If this variable is set to 1, then render will include a dump of the HDF contents at the end of the rendered page.

ClearSilver.WhiteSpaceStrip

This is used the same as the Config.WhiteSpaceStrip variable in the CGI Kit.

Functions

Types

Return nerrRaise(CGIParseNotHandled)

length = hdf.getIntValue("CGI.ContentLength")

Module Information