3.1.1.3. Chunking

The ability to build program files out of order is sometimes called chunking. It is sometimes useful, for example, to define functions after they are used even when the target programming language requires otherwise. However, chunking is not restricted to reordering code sections, but may be considered to include the ability to nest sections hierarchically. This permits the author to represent program structure in manner not provided by the target programming language.

There are some further variations on chunking. The first consideration is that code often has to be repeated, and so a section may be used more than once, possibly in distinct program files, but sometimes even in the same file. Because hierarchical chunking requires naming chunks, reuse is available automatically; the principal issue here is the converse: to ensure when required that a chunk is used exactly once.

Stemming from the use of labelled chunks is the need to be able to locate them: when a chunk is used, one needs to find where it is defined, and sometimes conversely.

More generally: a simple verbatim code file is a special case of breaking a code file into sections separated by documentation but without reordering, which is a special case of reorderable sections which is a special case of hierarchical chunking. Again, a tree is a special case of a directed acyclic graphs, which embodies the notion that a chunk may be used twice and also that it may not be used at all.

Some conventional literate programming tools go further in permitting what might be called macros: chunks containing unbound variables which can be bound at the point of use: a simple case of parameterized macro processing. Macros are particularly useful for generating repetitive forms such as tables.

In addition, we might consider conditional compilation, which is useful for controlling software variations such as platform dependencies and debugging versions. Conditional compilation is heavily used in C.

There is yet a more general form of chunking ... in which the notion of the chunk begins to degenerate, namely generation or arbitrary code by executable script. Such facility is essential, for example, where the code to be generated is sensitive to the environment, for example the inclusion of the current date. Perhaps more interesting is the ability of executable script to generate several files simultaneously, for example both the declaration in a header file and the definition in the body file, of a function in the C programming language. Building specialized scripts tailored to the client's requirement is an essential facility of literate programming tools because, although rarely used, it can save a lot of work and provide considerable coherence, as well as generating tabulated documentation.

To provide all these facilities, so the most specialized, and most common case integrates seamlessly with the most complex and general, interscript leverages the Python scripting engine. The most complex code generation is supported almost effortlessly by simply allowing the client to write arbitrary python script, while the more common simpler requirements are simply provided as pre-built routines: all the facilities are accessed in precisely the same way, by script execution.

This feature is central to the interscript design and cannot be isolation from a discussion of requirements, which can be recast, in some sense, to a discussion of the architectural framework in which such script executes, and the set of pre-built functions which ought be made available.

Macros with parameters, however, are not especially good at expression skeletons. A skeleton, or boilerplate, is essentially a macro with large arguments.