preprocess Ant Task

preprocess task

Each preprocess task copies a single directory (recursively) from input to output, transforming preprocessor comments as it goes.

preprocess attributes

var element

The var element sets the value of a preprocessor variable which can be tested in preprocessor comments in the input.

var attributes

filetype element

The filetype element extends the file types examined by the preprocessor.

filetype attributes

These two filetypes illustrate how to preprocess Java and XML files:

	<filetype name="Java" extensions="java" commentbegin="/*" commentend="*/"/>
	<filetype name="XML" extensions="xml" commentbegin="&lt;!--" commentend="-->"/>

The outextensions attribute allows you to use the preprocessor as a code generator. The output file will be written to a file with the same root name but a different extension. outextensions can specify one or more extensions. If the number of outextensions is greater than or equal to the number of extensions listed, each corresponding item in the outextensions list is used for the item in the extensions list; any extra outextensions are ignored. If there are more extensions than outextensions, the last extension in the outextensions list is used repeatedly.

Examples

Example 1: Split by multiple versions

<project name="com.objfac.prebop.split" default="split" basedir=".">
	<target name="properties">
		<property name="sourcedir" value="${basedir}"/>
		<property name="targetdir" value="${basedir}/../../versions"/>
		<property name="target2dir" value="${targetdir}/e2.1"/>
		<property name="target2prodir" value="${targetdir}/e2.1pro"/>
		<property name="target3dir" value="${targetdir}/e3.0"/>
		<property name="target3prodir" value="${targetdir}/e3.0pro"/>
		<property name="skip" value="xerces.jar,draw.jar,dtdparser.jar,genjava.jar,isorelax.jar,jaxen.jar,jing.jar,moved.jar,saxon.jar,saxpath.jar,trang.jar,trangutils.jar,util.jar,walker.jar,xml-apis.jar,xml-commons-resolver.jar,xml.jar,xmleditor.jar,xsdlib.jar"/>
	</target>
	<target name="split" depends="properties">
		<!-- A two-dimensional split by (Eclipse) version and pro
			 for a particular appversion -->
		<preprocess indir="${sourcedir}" outdir="${target2dir}" out="replace" 
			except="${skip}">
			<var name="version" value="2.1.2"/>
			<var name="pro" value="false"/>
			<var name="appversion" value="2.0.4"/>
			<filetype commentend="*/" commentbegin="/*" extensions="java"/>
			<filetype commentend="-->" commentbegin="<!--" extensions="xml"/>
		</preprocess>
		<preprocess indir="${sourcedir}" outdir="${target2prodir}" 
			out="replace" except="${skip}">
			<var name="version" value="2.1.2"/>
			<var name="pro" value="true"/>
			<var name="appversion" value="2.0.4"/>
			<filetype commentend="*/" commentbegin="/*" extensions="java"/>
			<filetype commentend="-->" commentbegin="<!--" extensions="xml"/>
		</preprocess>
		<preprocess indir="${sourcedir}" outdir="${target3dir}" out="replace" 
			except="${skip}">
			<var name="version" value="3.0.0"/>
			<var name="pro" value="false"/>
			<var name="appversion" value="2.0.4"/>
			<filetype commentend="*/" commentbegin="/*" extensions="java"/>
			<filetype commentend="-->" commentbegin="<!--" extensions="xml"/>
		</preprocess>
		<preprocess indir="${sourcedir}" outdir="${target3prodir}" 
			out="replace" except="${skip}">
			<var name="version" value="3.0.0"/>
			<var name="pro" value="true"/>
			<var name="appversion" value="2.0.4"/>
			<filetype commentend="*/" commentbegin="/*" extensions="java"/>
			<filetype commentend="-->" commentbegin="<!--" extensions="xml"/>
		</preprocess>
	</target>
</project>

In the example, the common source code is copied into four different output folders, each based on different values of the "version", "pro" and "appversion" preprocessor variables. This is often done as the first step in a multi-version scripted build.

Example 2: Preprocess in place

<project name="com.objfac.prebop.split" default="this" basedir=".">
	<target name="properties">
		<property name="sourcedir" value="${basedir}"/>
		<property name="targetdir" value="${basedir}"/>
	</target>
	<target name="this" depends="properties">
		<!-- Update in place for particular settings -->
		<preprocess indir="${sourcedir}" outdir="${targetdir}" out="merge" 
			except="${skip}">
			<var name="version" value="2.1.2"/>
			<var name="pro" value="true"/>
			<var name="appversion" value="2.1.0"/>
			<filetype commentend="*/" commentbegin="/*" extensions="java"/>
			<filetype commentend="-->" commentbegin="<!--" extensions="xml"/>
		</preprocess>
	</target>
</project>

Preprocessing in place allows you to change only the files affected by changes in preprocessor variables, e.g., to quickly test multiple configurations of the same source code.

Example 3: Generate files

<project name="com.objfac.prebop.split" default="this" basedir=".">
	<target name="properties">
		<property name="sourcedir" value="${basedir}"/>
		<property name="targetdir" value="${basedir}"/>
	</target>
	<target name="this" depends="properties">
		<preprocess indir="${sourcedir}" outdir="${targetdir}" out="merge" 
			except="${skip}">
			<var name="version" value="2.1.2"/>
			<filetype commentend="*/" commentbegin="/*" extensions="jtem" outextensions="java"/>
		</preprocess>
	</target>
</project>

To use the preprocessor as a file generator, specify a different extension as the value of the outextensions attribute. The preprocessed file will be written to a file with the same root name but a different file extension.

As with all code generators, you should check into source control either the template files or the generated files, but not both. Check in the generated files if the templates are to be used only once; otherwise, the best practice is to check in the template files and generate the other files each time you check out.

Using templates is slightly more complicated than using the preprocessor to modify files in place (merge), as if you want to change one of the generated files, you need to change the template file, instead. On the other hand, template files make it easier to avoid conflicts.

How the preprocessor works

The preprocess task copies files and folders from the input directory to the output directory, and allows conditional inclusion of source contents for, e.g., Java (.java extension) and XML (.xml extension) source files.

The preprocessor works by examining comments within source files. If it finds a comment that begins with the characters "$if" it enters replacement mode. In replacement mode:

• Every comment or non-comment line that begins with the characters "$if", "$elseif", "$else" or "$endif" is interpreted as a preprocessor statement.
• Preprocessor statements must be written entirely on one line with nothing but optional whitespace or opening comment brackets to the left of the statement
• Preprocessor statements use the comment brackets /* */ (for Java) and <!-- --> (for XML). Comments beginning with // are not treated specially.

The formats of the preprocessor statements are as follows.

	[BEGIN] $if condition $ [END]
	[BEGIN] $elseif condition $ [END]
	[BEGIN] $else $ [END]
	[BEGIN] $endif $ [END]

In the above, BEGIN stands for, e.g., /* in Java or <!-- in XML. END stands for, e.g., */ in Java or --> in XML. [] enclose optional parts. The BEGIN at the start of the first preprocessor $if in a comment group and the END at the end of the last preprocessor $endif in a comment group are required. All others are optional.

Every preprocessor $if must be followed by a balancing $endif. $else must follow all occurrences of $elseif at the same level.

The basic operation of the preprocessor is to add or remove BEGIN and END brackets as needed to honor the conditions. A simple example makes this obvious:

	/* $if version >= 3.0.0 $ */
	import org.eclipse.ide.*;
	/* $endif$ */

The above would be the result if the value of the version preprocessor variable was, e.g., "3.0.0". On the other hand, if the value of the version variable were "2.1.2", the result would be:

	/* $if version >= 3.0.0 $
	import org.eclipse.ide.*;
	$endif$ */

Because the condition evaluates to false, the contents between the $if and $endif are "commented out" by removing the appropriate BEGIN and END brackets. Note that the $ characters in a preprocessor statement and everything between them are always preserved, as is the leading whitespace on the line.

Here is a more complicated example, assuming that the variables verson and lite have the values "2.1.2" and "false", respectively.

	/* $if version < 3.0.0 $
	  $if lite$
	  foo();
	  $else$ */
	  bar(); // this line is included
	  /* $endif$
	$else$
	  $if lite$
	  foo3();
	  $else$
	  bar3();
	  $endif$
	$endif$ */

Nested conditionals like this are allowed but are difficult for humans to read. (In most editors, the above would be much easier to read than it is here, because only one line would be colored like Java code, while the rest would be colored like a Java multi-line comment.)

You can sometimes make conditionals easier to read by using more complex expressions and removing levels of nesting. For example, here is the same example rewritten with compound conditionals:

	/* $if version < 3.0.0 && lite$
	foo();
	$elseif version < 3.0.0 && !lite$ */
	bar(); // this line is included
	/* $elseif version >= 3.0.0 && lite$
	foo3();
	$else$
	bar3();
	$endif$ */

Although the preprocessor must do an insignificant amount more work, it is much easier for the human reader to see what is going on. Readability pays big divendends in maintainability.

The BEGIN/END brackets in preprocessor statements do not have to be written correctly in the input. The only requirements are:

• The leading $if of each preprocessor statement block must be preceded by BEGIN comment brackets.
• Within a preprocessor statement block, $if, $elseif, $else and $endif statements must be written on a single line preceded by no more than whitespace and an optional BEGIN bracket.

Note that anything after the second $ in a preprocessor statement is discarded by the preprocessor.

Multi-line comments that do not contain preprocessor statements are not allowed between [BEGIN] $if and $endif$ [END]. Single-line comments, like // in Java, are fine, as long as they are not on the same line as a preprocessor statement.

Conditions

Preprocessor conditions are boolean expressions involving:

• The names of variables declared in a <var> element,
• The boolean constants true and false,
• Non-negative numbers, beginning with a digit 0-9 and containing digits and up to two decimal points ('.'),
• String constants delimited by " or ' characters,
• The comparison operators ==, !=, <, <=, > and >=,
• The boolean binary operators && and ||, and the unary ! (not) operator, and
• Left and right parenthesis ( ).

There are three types of primitive values: boolean, numeric and string. Only values with like types can be compared; there is no value promotion or coercion.

Boolean values are compared by false < true.

String values are compared using UTF-16 numeric ordering (Java comparison).

Numeric values are compared by comparing the corresponding numeric values at each hierarchical level for up to three levels. For example, 1 < 2, 2 < 2.1, 2.1 < 2.3, 2.3 < 2.3.1, etc. 3 is equal to 3.0 and 3.0.0. This allows standard integer comparisons while integrating version number comparison in a natural way. Two numbers with no decimal points compare like non-negative integers, and two numbers with two decimal points each compare like, e.g., Eclipse version numbers.

Note that numeric values with decimal points do not compare like decimal numbers. For example, in Prebop 1.5 < 1.10.

The complete expression syntax is:

	condition ::= compare (('&&' | '||') compare)*
	compare ::= term (('==' | '!=' | '<' | '<=' | '>' | '>=') term)*
	term ::= '!'? primary
	primary ::= number | string | 'true' | 'false' | '(' condition ')'

Binary operators associate to the left and there is no precedence order between && and ||. Thus, a||b&&c equals (a||b)&&c and 1<2<true equals (1<2)<true (which is false). Full parenthesization will clarify your intentions and increase maintainability.

DTD extension

You can improve your Ant editing experience in Eclipse, using XMLBuddy or the built-in Ant editor, or other IDEs by adding the following lines to the DTD used to drive code assist.

<!ELEMENT preprocess (((var+,filetype)|(filetype+,var)),(var|filetype)*)>
<!ATTLIST preprocess
	indir CDATA #REQUIRED
	outdir CDATA #REQUIRED
	out (create|replace|merge) #IMPLIED
	except CDATA #IMPLIED>

<!ELEMENT var EMPTY>
<!ATTLIST var
	name CDATA #REQUIRED
	value CDATA #REQUIRED>

<!ELEMENT filetype EMPTY>
<!ATTLIST filetype
	name CDATA #IMPLIED
	extensions CDATA #REQUIRED
	commentbegin CDATA #REQUIRED
	commentend CDATA #REQUIRED
	outextensions CDATA #IMPLIED>

Be sure to add the preprocess task to the list of tasks defined at the start of the Ant DTD. E.g.,

<!ENTITY % tasks "preprocess | propertyfile | ...rest of tasks...">

(All XMLBuddy or XMLBuddy Pro versions later than 2.0.4 will have Prebop support built-in to the Ant DTDs supplied with the product.)

Author: Bob Foster

Acknowledgement

The idea for a preprocessor as a comment transformer came from the CodePro Preprocessor from Instantiations.