# API File Generation There are certain pieces of `NeuralNetworksTypes.h`, `Types.h`, `OperandTypes.h`, `OperationTypes.h`, and of our various `*.hal`/`*.aidl` files that ought to be kept in sync -- most notably the operand type and operation type definitions and descriptions. To avoid having to do this manually, a tool `generate_api.py` is employed to combine a single *specification file* with one *template file* per API file to produce that API file. The script `generate_api.sh` invokes `generate_api.py` once per API file, passing appropriate arguments. ## `generate_api.sh` The environment variable `ANDROID_BUILD_TOP` must be set. Invoked with no arguments or with the `--mode=update` argument, this script regenerates each API file in place, by invoking `generate_api.py` once per generated file. Invoked with the `--mode=hook` argument, this script checks whether NDK and Canonical files need to be regenerated. When the `--dryrun` argument is present, this script shows how it would invoke `generate_api.py` but does not actually regenerate files or check whether they need to be regenerated. ## `generate_api.py` This tool generates a single output file from an input specification file and an input template file. It takes the following mandatory arguments: * `--output OUTPUT` path to generated output file (such as `Types.h`) * `--specification SPECIFICATION` path to input specification file * `--template TEMPLATE` path to input template file * `--kind KIND` token identifying kind of file to generate The "kind" is an arbitrary token that the specification file can reference with the `%kind` directive to help generate different text in different situations. It has no meaning to the tool itself. Today, the following kinds are used: `ndk` (when generating `NeuralNetworksTypes.h`), `canonical` (when generating `Types.h`, `OperandTypes.h`, and `OperationTypes.h`), `hal_1.0` (when generating `1.0/types.hal`), `hal_1.1`, `hal_1.2`, `hal_1.3` and `aidl` (when generating `OperandType.aidl` and `OperationType.aidl`). ## Template File Syntax Every line of the template file is copied verbatim to the output file *unless* that line begins with `%`. A line that begins with `%%` is a comment, and is ignored. A line that begins with `%` and is not a comment is a *directive*. ### Directives #### `%insert *name*` Copy the *section* with the specified *name* from the specification file to the output file. The section is defined by a `%section` directive in the specification file. #### `%insert-indented *count* *name*` Similar to `%insert *name*`, but each non-empty copied line is prefixed with *count* space characters. *count* must be a non-negative integer. ## Specification File Syntax The specification file consists of comments, *directives*, and other text. A line that begins with `%%` is a comment, and is ignored. A line that begins with `%` and is not a comment is a *directive*. The meaning of a line that is neither a comment nor a directive depends on the context -- the *region* in which that line appears. ### Regions The specification file is divided into *regions*, which are sequences of lines delimited by certain directives. Certain regions can enclose certain other regions, but this is very limited: * A conditional region can enclose a section region. * A section region can enclose a conditional region. Equivalently: * A conditional region can be enclosed by a section region. * A section region can be enclosed by a conditional region. #### null region A *null region* is a sequence of lines that is not part of any other region. For example, a specification file that contains no directives other than `%define` and `%define-kinds` consists of a single null region. Within a null region, all lines other than directives are treated as comments and are ignored. #### conditional region A *conditional region* is a sequence of lines immediately preceded by the `%kind *list*` directive and immediately followed by the `%/kind` directive. The `%kind` directive establishes a condition state **on** or **off** (see the description of the directive for details). When the condition is **on**, the lines in the region are processed normally (i.e., directives have their usual effect, and non-directive lines are added to the enclosing section region, if any). When the condition is **off**, lines in the region other than the `%else` directive are ignored *except* that even ignored directives undergo some level of syntactic and semantic checking. #### section region A *section region* is a sequence of lines immediately preceded by the `%section *name*` directive and immediately followed by the `%/section` directive. Every line in the sequence that doesn't begin with `%` undergoes macro substitution, and the resulting lines are associated with the section name. They can be inserted into the generated output file as directed by the template file's `%insert` and `%insert-indented` directives. They can be added to another section region with the with the specification file's `%insert` and `%insert-indented` directives. This is the mechanism by which a specification file contributes text to the generated output file. ### Directives #### `%define *name* *body*` Defines a macro identified by the token *name*. The *body* is separated from the *name* by exactly one whitespace character, and extends to the end of the line -- it may contain whitespace itself. For example, %define test this body begins and ends with a space character Macro substitution occurs within a section region: a substring `%{*name*}` is replaced with the corresponding *body*. Macro substitution is *not* recursive: A substring `%{*name2*}` in *body* will not undergo macro substitution, except as discussed for *macro arguments* below. Permitted in regions: null, conditional, section ##### macro arguments The more general form of a macro invocation is `%{*name* *arglist*}`, where *arglist* is a list of whitespace-separated arguments. Within the *body*, a substring of the form `%{argnum}` will be replaced by the corresponding argument from *arglist*. For example, if the definition is ``` %define test second is %{2}, first is %{1} ``` then the macro invocation ``` %{test alpha beta} ``` is expanded to ``` second is beta, first is alpha ``` The only check on the number of arguments supplied at macro invocation time is that there must be at least as many arguments as the highest `%{argnum}` reference in the macro body. In the example above, `%{test alpha}` would be an error, but `%{test alpha beta gamma}` would not. #### `%insert *name*` Adds all lines from the named section region to the current section region. Permitted in regions: section #### `%insert-indented *count* *name*` Similar to `%insert *name*`, but each non-empty added line is prefixed with *count* space characters. *count* must be a non-negative integer. Permitted in regions: section #### `%kind *list*`, `%else`, `%/kind` `%kind *list*` creates a *conditional region* terminated by `%/kind`. The *list* consists of a space-delimited list of tokens, any of which may end in `*` to indicate a *wildcard pattern* or `+` to indicate a *lowest version pattern*. Any other pattern is a *simple pattern*. The condition is **on** in three cases: * One of the simple pattern tokens equals the "kind" * One of the wildcard pattern tokens less the `*` is a prefix of the "kind" * One of the lowest version pattern tokens less the `+` matches the "kind" or the "kind" matches any token to the right from the lowest version pattern in the list passed to %define-kinds In all other cases, the condition is **off**. Within the region, the condition is inverted every time the `%else` directive appears. Permitted in regions: null, section #### `%define-kinds *list*` This directive has two purposes: * Validity-checking. If the "kind" is not on the space-delimited *list* of tokens, `generate_api.py` terminates with an error. * Ordering the possible kinds for the *lowest version pattern* (see the section above for the explanation of the pattern). Only one such directive is allowed per specification file. Permitted in regions: null, section #### `%section *name*`, `%/section` `%section *name*` creates a *section region* terminated by `%/section`. Permitted in regions: null, conditional