GtkD - Anatomy of the bindings generator.

To generate the Gtk bindings (and adding other features) for gtkDjs I use a code generator. This is pretty common to all the Gtk bindings. I worked on the PHP-GTK ones a long time ago, which in turn where based loosely off the Python bindings.

For GtkDjs, the starting point was the GtkD binding code. While the current code is still similar to it's parent, It has grown considerably since it's original birth. So In the vain of "if someone ever feels like helping out with the bindings", or wants to write and send me bindings for SQLite, Mysql, curl or libsoup etc. Here's the inside story of them.

The basic concept.

In essence, the bindings work using "generator scripts" called APILookup*.txt, which contain a series of either information, or instructions to do tasks.

Other than the overall configuration settings for the generator, or libraries, the core part of generating bindings is to create classes. In Gtk this is normally done by reading the HTML documentation for a specific Gtk Struct, and using that information to generate data about the Enums, Signals and Methods that are defined. This is all done by GtkClassParser.d

Finally after the File has been parsed, the Output classes "ClassOut, FuncOut, EnumOut, LibraryOut, PhobosOut, SignalOut" generate the code files in the src directory.

Unlike the original GtkD bindings GtkDjs does a double pass at generating each of the packages, as we use the first pass to calculate dependencies and inheritance, rather than relying on the list of imports in the APILookup files. This means that the APILookup*.txt files do not really have to list all the required imports.

More details on the APILookup files.

While I'm not going to explain all the possible commands available in the files, I will explain the concept and a few key ones. The file "GtkWrap.d" contains the parser, with a complete list of keywords and actions, it's a reasonably simple and obvious piece of code. I'm still adding new commands when I see them as necessary, and may even remove a few as some are pretty pointless or unused.. (like code: *** which is ignored but is usefull for reference on how GtkD did things).

The basic concept of the file is a list of data in the formats:

#for simple data
keyword : Value
# for building a list of key value pairs
keyword : key valuepair
#for long data - like code etc.
keyword : subject
......
keyword : end

#for long data - (older style)
keyword : start
......
keyword : end

Order is quite important, as the parser for the APIWrap is not forgiving and can easily go into infinite loops if you get the wrong or unknown keyword in the wrong place. Fortunately using the existing files as a basis for a new one should help out a bit.

The core APILookup.txt files contains a few key pieces of information

• aliases of Gtk base types to D types (and a few kludges to work around other issues)
• mappings of the D-Types to Javascript Types (really related to the method calls on dmdscript.value)
• where the files are, like the gtk documentation and where the generated files should be written to.
• List of packages, and how they related to directories (Note that the GtkDjs bindings also use a jsPackage command in the individual wrapping files to determine how they are exposed to Javascript)
• A small bit of code to copy the dl() loader into the generated directory.
• And finally a "include" like statement "lookup:" which tells the parser to start using the new file as it's input.

The individual Binding for the specific libraries (or packages as the tend to be called) are structured a little differently.

• At present there are alot of add*: commands, which are generally ignored by GtkDjs, and will probably be removed. - As we auto generate most things
• In the Gobject bindings, you will see an example of structCode: which is how structs are described if they can not be done by just pretending they are pointers to void.
• Again in the Gobject you can see a manual enum description. necessary as the real Enum is not actually described in the documentation.
• next are the two key information, wrap and jsPackage - wrap indicates which package it is wrapping (from those listed in APILookup) - so it knows where to write the files. and jsPackage tells it how to expose it to Javascript
• structWrap: is used simply ensure that the struct is available usually as an empty struct.
• nostruct: is used to prevent a struct from being bound.. usually an indicator that something needs fixing..
• finally we have a small batch of code for each class to be written. The class: command indicates the filename of the code to be written, the real classname is based on the struct that is being wrapped usually.
• some of the files have jscode, and jssig elements describing how the automatic code and documentation generation should be overridden.

The generator code.

In order of how they are used.

• GtkWrap.d (includes main()) and is the command interpreter, data from the commands is stored in a ConvParms class (convparms.d), and passed to the Outputters. Any packages that are defined are stored in a class defined in Packages.d.
• DefReader.d is the helper class that does the low level conversion of the APILookup.txt files into commands.
• GtkClassParser.d does the legwork of collection all the described definitions in the HTML file into Classes from GtkStruct.d GtkFunc.d, GtkEnum.d (note signals re-use the GtkFunc class). The HtmlStrip.d library just removes the HTML tags making the documentation files a bit easier to parse.
• Finally after a file is parsed, usually the outFile: command is found and the class code is generated by ClassOut.d. Which in turn loops through the functions, signals and generates the code using FuncOut.d and SignalOut.d
• At the end of each package file, the loader, struct listing and enum lists are generated from LibraryOut.d which uses EnumsOut.d and GtkStruct.d (although that wasn't the best of design hacks)
• PhobosOut.d is used by the non-gtk code to generate code that does not need library links.

The design differs a little from the original GtkD generator in that I separated the parsing of the documentation and the code generation into separate classes. I felt the original code was pretty huge and unwieldy done that way.

Anyway comment away if you have any further thoughts...