libvala testing code and extracting API from the vapis

In the first versions this was done using the gobject introspection API Originally 

Using this API it is possible to work out all the properties of a Gtk.Window for example so that you can pick properties from a palete and add set them appropriately. The work done on the Seed documentation, was one of the drivers that helped create the initial builder code in Javascript. I initially mixed the API calls with an XML parser to read the comments from the Gir files, so that there was some 'helpful' documentation about all the properties.

When converting the code to Vala, it became clear quite early that the Gir and Vala API's while very similar have a few very slight differences, some of the signal names where missing, and some of the properties where not available. Unfortunately at that time, my understanding of the vala compiler, and the libvala library was quite limited, and as a stop gap solution, I decided to just use the Gir XML files as the basis for the properties in the builder. Using a simple xml node handler, and iterating through the xml file it was possible to extract all the data needed to get most of the properties for the builder.

This worked for most things, but I ended up with a few kludges, as things like GtkSourceView is Gtk.SourceView in vala, but GtkSource.View in gobject introspection. so there was a few hacks added to handle this issue.

The ability to access the API, is not only critical in the user interface to add properties and elements, but also in writing the vala source files generated from the builder. The code in NodeToVala.vala  converts the Tree structure used internally in the builder into vala code.

One of the tricks we added early on in the Seed version of the builder was to validate the Javascript event handler code, It would highlight syntax and other errors in the gtksourceview editor, so you would not have to wait until you viewed or run the code to find out where your mistakes where.

The code for validation is in the Editor.js file

This was done by calling Seed.check_syntax, and parsing the exception thrown if there was a problem, In the vala version this was done by directly calling the JSCore engine of the Webkit API, then in the same way as the javascript version using the exception to determine the error location.

Checking Javascript code is now handled by the Palete/Javascript.vala code 

For Vala it was a slightly more complex issue, When I started writing the vala version of the builder, I used Ajunta usually to track down errors, as it has a nice compile output filter, where you can hide notes/depreciated and warnings, and only show the real 'errors'. This was quite a round trip, edit code in the builder, run the compiler, find the error - look at the generated file - work out how that relates to the element in the builder... 

It slowed down progress on adding features to the builder considerably. My first step to solving this was to consider using the vala compiler code to see if we could do something similar to javascript (run a syntax check on some code). I had emailed Florian Brosch (author of valadoc.org), when I started looking at extracting the API from vapi's, he pointed me to the tree builder in the valadoc codebase. At the time I decided to go the easier route in using the GIR files. so I did not really make use of his suggestions.

However as I desperately wanted to get the syntax checking working, Eventually I decided to have another attempt at this. what resulted was  thi ValaSource.vala file, this started off by just creating a code context, and adding the generated file (with some comments to help work out where the error was in relation to the tested code). 

From this I created a SourceReport class to catch all the error messages, and then report back if there where any basic syntax errors. This was a huge improvement over the previous situation, but it did not completely solve the issue, as by just doing a syntax check, I could not really find out if the code would compile. it had no awareness of Object types, and how they related to the libraries that where needed to build the application.

Over time I slowly added features that allowed the builder to know about which vala packages (vapi files) where used to create the application, and all the other files that made up the application. This is all handled by the project properties section for Gtk Applications.

With this extra information, I was able to include all the correct elements needed to do a proper check on the edited code. so not only could it run context.parse(), but also context.check() the code in ValaSouce could actually build a binary from all the included files. 

The next step was to integrate this into the editor. running parse on a file is very fast, it can almost be done transparently on the edit keyup signal. However a full parse/check/emit could take 3-9 seconds on my SSD based machine (maybe even slower on a HHD). So the interface would hang while it tried to do this checking. To resolve this, I ended up working out how create a thread, call it using vala's excellent async infrastructure, and callback (using the correct thread) to the UI, so that while you are typing it is constantly trying to parse/check/emit the code you are writing. and notifying you if there are any errors - this has some simple queuing to prevent it doing the checking to match, basically it can only be running one check at a time.

Having got my hands dirty with the libvala API, I was pretty sure I would be able to figure out how to extract the Gtk API and others from the vapi files using libvala. During the conversations I had with Florien the only bit of the code that he showed me that I did not understand was how after running parse/check etc.. how do I get access to the tree, 

The code he showed me extended the Vala CodeVistor class, I had extended this when I created the code syntax checker (although it's not really needed). in his example he had some callbacks to visit_namespace(), a method overridden in the CodeVistor. however what I missed was how the context knew about the code visitor. - as you can see from the ValaSource file, the context does not know about the ValaSource file, so would not be able to call it.

After reading all the code in valadoc, I finally spotted this line..

This was one of those 'Light bulb moments', this was the call I was missing, basically by doing this, the context starts walking the AST tree - giving you access to all the details about the vapi files that I needed. In most of the examples of CodeVistor, it basically calls all the various visit_* methods on the Vistor as it walks through the tree (you need to call element.accept_children(this) to cause the walk to happen).  For my purpose, walking the tree this was not really that helpful, as I needed to build a Gir tree out of this data so the existing code would work with the new data. I discovered you do not have to walk the tree, basically all the walking code does is just iterate through properties that are available anyway, so you can just do that walking in the code, rather than using the callback walk.

The documentation for libvala is very helpful

The result of this is that I now have good reliable API data to build the Application, and the code writer should be more accurate. Next steps on this is to start using pulldowns for property enums, as I can extract that information from the API.