11 Apr 2007
making XSLT readable - has this been done before?
I had the idea of using XmlHttpRequest, however another member of the team suggested XSLT, and I have to admit, I've never seen that idea before, and the example he showed basically included content from an external file by just adding a simple tag like
<our:include src="mydynamic.html" part="xxx" />
Where the mydynamic.html could use the 404 handler trick that php.net uses. (eg. if the file doesnt exist, it gets generated by the 404 handler, and deleted by a cronjob every 5 minutes etc.)
My only reservation about this is really that XSLT templates are unreadable garbage. Hence if we ever started to get clever with them, they could quickly become unmaintainable.
This however didnt seem to be a problem so much with XSLT but the fact that reading and writing XSLT files is not particulaly simple. Looking at the Basic specification for 1.0, and the example file he had, it seem clear that this file could easily be generated by a simple tool. However I could not find anything after a quick google search that seemed to exist to do this. So I threw together a simple proof of concept.
What if the template looked more like this:
template("/") {
applyTemplates;
}
template("*") {
copy {
applyTemplates("@*|node()");
}
}
template("processing-instruction()") {
copy {
applyTemplates("@*|node()|text");
}
}
// make h1 green
template("html:h1")
{
copy {
attribute("style") <colour:green>
applyTemplates("@*|node()");
}
}
template("our:include")
{
variable("part", "@part");
applyTemplates("document(@src)//*[@id=$part]");
}
Writing a simple parser generator in D, enabled the generation of the xslt file from this file.Compile parser generator:
dmd xsltgen.d -of/tmp/xsltgen -od/tmp
Convert the above file to xslt:
/tmp/xsltgen inctest.xslts
And magically you have a generated file, and a maintainable version.. - Makes me wonder though, has someone done this already?
the source: xsltgen.d
output from the above example: inctest.xslt
25 Mar 2007
Javascript Tookits Part Two
My last post about Javascript toolkits garnered a few comments, which seems to be rare for most of my blog posts... (I must be getting too old..or the spam protection works to well). There where quite a few suggestions for alternatives to dojo, along with a not too complimentary comment about dojo's development status.
As luck had it, I started a new project Last week, a contact manager for one of my clients, and rather than carry on with dojo, that had begun to annoy me a little, with little bugs (for example , the Accordion was broken), or just weird stuff, like getting forms on dialogs to look just right. I thought that it would be a good opportunity to review some of the other toolkit's mentioned on the post.
The extended Entry contains reviews of jQuery and yui-ext (and see which is now the prefered library)
08 Mar 2007
Javascript Toolkits
Like alot of people, I'm really impressed with google's applications. They have created web applications that behave alot like desktop ones. Core to this is the javascript tricks that hold it together.
In the past few years I've been building applications using XUL, that provides a similar set of functionality, and is pretty fast to develop with. However we have run into a few issues
- Its not cross browser, which is ok for intranet style applications, obviously is a little problematic for public internet ones.
- After many projects we began to see that dialog, modal prompts where better for user input, as the prevent the user modifying the application state while entering data, along with making it less likely to forget to press the submit button. Recent builds of Firefox have disabled the modal feature of popup window, and along with the noticeable speed penalty on most modern operating systems for opening new windows, these issues begins to dig into the benefits of this approach.
- WYSIWG editors are still not easy and simple to use...
Since one of my pet projects, an online accounting package needed a bit of sprucing up, and testing my home grown JavaScript code on IE is such a nightmare, I started looking at JavaScript toolkits.
The biggy from countless blog posts recently has been YUI, Yahoo's interface toolkit. I spent quite a while looking at this, the website for it is very complete, along with a comprehensive set of widgets. There was alot too like about it, however as I dug deeper, I began to dislike some of the things i saw.
- silly things like all the JavaScript code libraries appeared to use a weird, double lined coding standard.yes I know it's silly, but it makes a difference when you read code to see something that odd.
- (from what i remember ), alot of those set*, get* methods.
- almost all the demos appeared to create the widgets in JavaScript and using js associative arrays to configure the widgets , rather than using extra attributes on existing html tags, and doing some kind auto conversion.
- the constructors did some strange namespace hack.
In general, it looked like using the widgets was very un-javascripty.
So after some consideration, I started looking for alternatives. The first I came across was prototype which looked more like someone rewriting JavaScript than a toolkit, then i came across dojo,
which looked alot more promising. The website has focused on the demos, which is pretty good. There is documentation (i'll talk about that more later)
Best of all it works alot more like the widget based JavaScript code i've written. it focuses on converting existing dull html widgets into super js enabled ones at the flip of a switch.
Concept
Conceptially, as show on the demo's, takes the HTML page, and when the library is loaded, it hooks into the onload method of the page, and looks for all the widgets with the attribute dojotype="Select" or similar, and has a javascript class to handle each "dojotype".
The dojo widget class takes a look at the attributes, and sets the javascript object properties to match those HTML attributes (hence behaving like the configuration for the widget). Then it builds the new widget based on those attributes, and merges it in with a template file (which is in dojo/src/widgets/templates/*) or uses a text attribute of the javascript object for the template. It then adds all the behaviour and just makes stuff happen.
This means getting upto speed with the widget set is fast, and you only slow down when you want to alter the default behaviours, at which point looking through the code seems to be the best idea.Unfortunatly, I found the documentation for dojo, a little thick and rather pointless, it comes of as more of a overview, and the real information is so far down in the chapters that it doesnt help much.. Rather than a nice reference of all the attributes that each widget type uses. (although the API reference is quite handy)
Features
Dojo has all the basic features that you would expect from a javascript widget toolkit, dialogs, windows, autocompletion, clever select widgets and more. It also has quite a library of backend tools for formating dates, doing xmlhttprequests. and more. What is nice about this, is that these libraries are not forced on you, rather they are used by the widget libraries, and you dont really need to know about them, or learn them. And offer a gradual learning path. So when you are ready you can start diving in and seeing if they are usefull in other parts of your code.
The architecture looks to be extendable as well, so you theoretically could build on the existing widget types and create your own bastard children, suiting your needs better. The syntax for the widget's is realitively straight forward, although they do use a slightly strange javascript syntax, but I doubt there is a better way to do what dojo acheives.Most of the really usefull stuff is in src/widgets/*
Flaws
Well, nothings ever perfect, and looking at the huge number of bugs in the dojo bugtracker, I suspect they developers are having difficulting keeping up with it's popularity.
There are a few design decisions that where annoying (although this has to be taken against the fact that the majority where excellent). The key ones being:
- CSS interaction, Widget's can load their own CSS from the template directory, This makes modifying the CSS for widget's rather difficult. (unless you treat the template directory as your own working space - or override it somehow). The more I worked with it, the more I thought it would be better to provide a generic CSS file that contains all the CSS needed for the widgets (or a file that @imports all the individual ones) , and force you to load it manually. - hence allowing you to replace the CSS quickly and simply. As often the dojo styling can conflict with any generic CSS that you may have already for your application.
In addition, there are quite a few style="..." tags in the HTML templates used by the widgets, these should really have been moved into the CSS files, allowing more flexibility.
- Creating your own widgets. based on the dojo ones, It looks like a theroetically simple task, but an example in the demo's would probably help out alot (eg. a single file with an extended dojo widget example) - not a sexy demo, but would be really handy..
- Documentation of attributes. as I mentioned before, the two set's of documentation - API and Manual miss out on the esential part of dojo, the use of HTML attributes to change settings. This could probably be generated or displayed in the API manual but is missing.
- Missing widgets, or features lacking. There are many little things missing from dojo, that seem obvious, yet are not there (yet),
- Dialogs that are more complete, the basic dialog is very baren, having no OK/Cancel built in, and no styling that makes it look particularly sexy.
- No Basic Dialog set's like prompts, warnings etc. that just display messages or get simple user feedback
- Interaction of the button widget with things like wizards and dialogs - you would expect these to be used by default on the standard wizard, but the default is a rather poor looking button
- The great love of blue - the standard button widget uses rather a inelegant blue colour, a faded grey would probably be more neutral for generic applications.
So far though I've been pretty impressed with dojo, so I would recommend it as a good solution for a adding a stylish interferace to rather dull get/post web sites.
25 Feb 2007
Walter showing off Digitalmars D cool features
Video of Walter Bright at the C++ group.
Slides of Walter Brights talk at the C++ group
Event details for reference
Some of the stuff like scope(exit) - makes Exceptions actually usable. Some of the other stuff like templates I'm still trying to get my head round, let alone decide how the fit into writing maintainable code....
31 Jan 2007
Backtracing segfaults in a daemon in digitalmars D
So after a bit of searching through code.google.com I came across the idea of catching the SIGSEGV signal and calling backtrace and backtrace_symbols
This little trick can product output that looks something like
/path/to/application [0xAAAAAA] << address of codewhich initially seemed a bit cryptic, but by putting it together with
/path/to/application [0xAAAAAA] << address of code
/path/to/application [0xAAAAAA] << address of code
/path/to/application [0xAAAAAA] << address of code
addr2line can result in some great debugging information.
This is my little backtrace logger for the deamon logger.
static void print_trace()of course you need to use a few C externs to make this work:
{
void *btarray[10];
size_t size;
char **strings;
size_t i;
pid_t pid = getpid();
//writefln("SEG");
size = backtrace(cast(void**)btarray, 10);
strings = backtrace_symbols(cast(void**)btarray, size);
std.process.system("/bin/echo '----BACKTRACE------' " ~
"> /var/log/myproject/backtrace.log");
for(i = 0; i < size; i++) {
char[] line = std.string.toString(strings[i]);
char[][] bits = std.string.split(line, "[");
char[] left = std.string.strip(bits[0]);
if (!left.length) {
continue;
}
// skip lines with ( in them...
if (std.string.find(left,"(") > -1) {
continue;
}
char[] addr = bits[1][2..length-1];
std.process.system("/bin/echo '----" ~ addr
~ "------' >> /var/log/myproject/backtrace.log");
std.process.system("/usr/bin/addr2line -f -e " ~
left ~ " " ~ addr ~ " >> /var/log/myproject/myproject.log");
}
free(strings);
}
extern (C) {
int backtrace(void **__array, int __size);
char** backtrace_symbols(void **__array, int __size);
pid_t getpid();
sighandler_t signal(int signum, sighandler_t handler);
void sigsegv(int sig)
{
// reset the handler.
signal(SIGSEGV, cast(sighandler_t) 0);
print_trace();
// really die
exit(SIGSEGV);
}
}
and to add it to you application, stick this in main() somewheresignal(SIGSEGV, &sigsegv);testing it is quite simple, just do this in D
void testSegfault()Now looking at the debug file, you can work out where it failed...
{
class SegTest {
void test() {}
}
SegTest a;
a.test();
}
----BACKTRACE------I'm sure with some more work, you could get it to log to syslog...
----805e971------ (THIS IS MY OUTPUT CODE)
_D9myproject7cmdLine6daemon11print_traceFZv
init.c/src/myproject/cmdLine/daemon.d:306
----805e46b------
sigsegv
init.c/src/myproject/cmdLine/daemon.d:121
----804db18------ (AND NOW FOR THE LOCATION OF THE SEGFAULT)
_D9myprojectfort7manager7manager7runOptsFZAa
init.c/src/myproject/manager.d:50
----805617a------
_D9myproject10webRequest10webRequest5parseFAaAaKAaZAa
init.c/src/myproject/webRequest.d:89
----8050c3e------
_D9pmyproject14myprojectThread14myprojectThread18dealWithWebRequestFAaAaZv
init.c/src/myproject/myprojectThread.d:331
----80503d0------
_D9myproject14myprojectThread14myprojectThread3runFZi
init.c/src/myproject/myprojectThread.d:111
----8076260------
_D3std6thread6Thread11threadstartUPvZPv
??:0
----a7fd10bd------
??
24 Jan 2007
Wing Lung Bank (Hong Kong) not exactly trustable.
Banks, while never perfect corporate citizens usually at least try to behave in a reasonable maner. That is probably why I was fuming with anger at Wing Lung Bank in Hong Kong.
I have a credit card from this bank, that was given out as HKIA uses them for their membership cards, and my spouse being a member, just signed me up for it. I normally use American Express for most payments, but some shops don't accept it so having a Visa card is quite handy.
We always pay off the balance every month, I normally sit down around the 1st or 2nd of each month and clear the bills. I must have been doing this for at least 6 years, and I've had the Wing Lung card for I guess more than 3 years.
So yesterday when I got a letter from them indicateing we had not paid the bill, I was a little supprised. That turned out to be me forgetting to pay my wife's bill last month, as she doesn't use the card normally, so the is normally nothing to pay. However while checking the other bills, I noticed a finance charge, and a late payments charge. I was pretty certain that I had paid last month on time, so I checked closer, I had paid the bill on the second of the month, which was the usual date. Then I spotted the problem..
WING LUNG, THE BUNCH OF CROOKS HAD CHANGED THE PAYMENT DUE DATE!!!!
They had moved it back about 10 days, resulting in the card running up late charges.
Not only that, I started checking the older statements,and realized the had pulled this slight of hand 5 months ago, raking in a tidy sum for their criminal coffers.
By this time I was fuming, and called their hotline, my temper not subdued by their annoying 'press 2 for..' and playing adverts for their services while waiting.
After practicalling screaming at two idiots who refused to refund the money and change the payment date, and listening to them try and justify this crazy action 'your agreement states we can change the terms at any time' They promised that a manager would call back.
Of- course, this call never arrived, so today I cancelled the card, filed a complaint with our consumer council. (Hopefully at least they will issue a warning about them Changing the due dates, so other customers don't get caught out.)
From what I've been hearing, this is not the first bank to pull this scam.. Well, Wing Lung will not be one of the place's I'm going to get a mortgage from...
06 Jan 2007
Hong Kong consultation on copyright changes.
Response to Copyright Changes consultation document:
Please Have a look through my response, (I'm not an qualified activist, or an English major - so you probably want to improve it) but please do respond if you live in Hong Kong. It's already a bad enough police state, without these idiots letting this kind of legislation loose.
To co_review@citb.gov.hk
Introduction:
The whole premise of this document is outrageous, it reads like a music industry wish-list, and follows on from a previous consultancy document that was rejected before. Basically trying to force a similar attitude without care or compassion for citizens of Hong Kong.
It Completely ignores fair use and consumer rights issues, which have already been seriously damaged by the current legislation, and are railroaded in favor of helping the antiquated business models of copyright monopolies. A particularly bad example of collusion between government and business over consumer rights.
22 Dec 2006
local caching and JSON bugs.
In discussing the application with the client, they also requested that the application remember some of the states (eg. the data filters). Normally you might consider cookies or having a preferences database, but I remember seeing something about firefox2 having localized session storage.
You can read the session storage specification here, It allows you to store strings totalling upto about 5Mb on the client side. However, the reality is that you really need to be able to store javascript objects or similar in it. So since we where using JSON to transfer data, I thought 'how about serializing the data back into JSON, and storing it in the session.'
I started off by using the JSON js code, however this code is unusable for 1 major reason - it adds a method to array's/objects, resulting in the fact that foreach() always returns an additional member. (breaking almost all the existing javascript code!) - I also ran into problems with recursion in firefox (but I think thats due to me forgetting to add the String.prototype method.)
So in the extended entry is a quick kludge (without full unicode support) for converting items into JSON in javascript, so that something like this works:
sessionStorage.setItem("mycache_somedata", JSON_toString(mydata));... and onloading..
var mydata = new Array();
if (sessionStorage.getItem("mycache_somedata")) {
eval("mydata = " + sessionStorage.getItem("mycache_somedata").value);
}
12 Dec 2006
Autocompletion in leds for Digitalmars D
I've been spending far to much time looking at the autocompletion in leds for D. The justification, while growing weaker the more time i spend on it, remains the same. Autocompletion and AutoHelp save a huge amount of time when writing code.
The PHP parser in leds is working quite well, is organically written, and has room to grow, the next major step is guessing object types by matching method calls on unknown objects, so while I spend about the same amount of time coding in D as I do in PHP now, I had decided to look at the D parser, and push that along.
Antonio had written the current parser which is in the dantfw project. It aimed to parse all C like languages, C, D, Java and C# etc. This meant that rather than following the D specification, the tokenizer and parser where quite generic. and lead to two issues.
- When I added autocompletion as you type, rather than on demand (ctrl space), I noticed that it was attempting to autocomplete when i was typing "strings" and comments.
- Quite often it appeared to be missing some key variables or methods that it should have known about.
The first issue I resolved by rewriteing the tokenizer (following the D specification closely), and re-tokenizing the document as you type (if it thinks your scope may have changed), then deducing the scope being input.
The second, I began to conclude was more related to the generic nature of the parser. It was making many assuptions about the language that turned out to be incorrect in D's context. The only, rather drastic solution was to rewrite the parser....
So on rather a quiet day i started that thankless task. Using Antonio's original design, of using object contructors to parse and eat their own tokens, I set about writing loads of switch/case combos.
My first effort worked reasonably well, but the more i wrote it, I began to realize that both the complexity, and the use of a series of tokens had a number of flaws. The declaration code for parsing methods, and variables was quite large, and very similar. (ending in a lot of redundant, duplicate code) Declaring multiple variables in one line also made the resulting abstract syntax tree objects quite klunky.
At this point I put it on hold for a while, partly to consider alternative approaches, and more to ensure paid work was not falling behind.
When i returned, I had come up with a few ideas
- look at dmd for inspiration
- check dsource to see if somebody had already done this
- consider preparsing tokens into a tree before sending them to the parser.
From dsource I found codeanalyser, which after a bit of hacking to remove what appeared to be the C++ memory stack allocation routines. I finally got to compile (without crashing dmd), parse and output what looked a little bit like syntax trees.
The downside to this project was that extracting the detailed information I required (type definitions, method declarations, line numbers etc) was not really feasible, and the tree that was produced by the code included a significant amount of noise, (in that it frequently created tree nodes for failed syntax matches). Along with this, it would require some work to store the token data within the tree that was created.
On the upside, in debugging the compiling issues I began to get a better understanding Templates (one of those black magic features of D). I ponder if the introduction to templates in D should basically start with the statement.
"D does not have preprocessor macro's, it has Templates" as while Templates are considerably more powerfull than Macro's, from a comprehension point of view, that is effectively what they are. And, given that D's decision not to have macro's, on what appears to be a sensible view that they obfusicate the code (along with often making it a nightmare to compile), I consider Templates to be on the list of features to be used with extreme caution, as they have a similar obfusicating effect, although perhaps not to the same degree....
Anyway, so the last option was to chuck my first draft, and use an idea of building a token tree, and passing that to the parser, rather than just giving a series of tokens. Having got about half as far as I did before, I consider this one of those little gem's of an idea. That makes writing the Parser considerably simpler..
Consider the simple statement
void main(char[][] argv) { }Which would tokenize into the following.Token.T_void : void
Token.IDENTIFIER main
'('
Token.T_char char;
'['
']'
'['
']'
Token.IDENTIFIER argv
')'
'{'
'}'
Using a post tokenizing tree building routine, it now looks like this, which from the perspective of the parser, is considerably simpler to deal with.
Token.T_void : voidFrom a pattern perspective, we are just looking for
Token.IDENTIFIER main
'('
Token.T_char char;
'['
']'
'['
']'
Token.IDENTIFIER argv
')'
'{'
'}'
BaseType, IDENTIFIER '(' '{' == a method declaration.(although it's a bit more complex than that in real life!)The tokens inside of a '(', '{', '[' collapse into that token, so we can either ignore them, when not needed, or send them as a set to a parsing routine. which doesnt have to keep dealing with nesting or determining the closer.
Anyway, the new parser is slowly ticking away, once it parses correctly, the next step is to work out how to make the resolver cleaner...
24 Nov 2006
Multilanguage setup for Flexy and FlexyFramework.
FlexyFramework, with HTML_Template_Flexy include the ability quickly create multilingual websites, by translating the templates.
I've set this up a few times now, and kept meaning to document it. - So here it goes.
Having set up FlexyFramework, (see the previous post about this),
... The full instructions are in the Extended entry....
Follow us
-
- Some thoughts on the language server and its usefulness in the roobuilder
- Roo Builder for Gtk4 moving forward
- Clustered Web Applications - Mysql and File replication
- GitLive - Branching - Merging
- PDO_DataObject Released
- PDO_DataObject is under way
- Mass email Marketing and anti-spam - some of the how-to..
- Hydra - Recruitment done right
Blog Latest
-
Twitter - @Roojs