Big step forward in Modular Database Applications with DataObjects

HTML_FlexyFramework Provides the basic 'Controller', It's one of those very lightweight, get out of your way, 2  small classes bit of code. In some sense's it's not really  a Framework (well 2 classes could hardly classify as that). More a generic Controller, and a Base class for a Page, just implementing the bare bones stub features, like get/post,getAuth() and implementing the output method to run HTML_Template_Flexy.

What I've built ontop of that is the Pman Components, originaly based on a project codenamed Pman (for project management). Which ended up being the most re-usable code I've developed to date. It's the foundation for around 6 completely different projects now, sharing a large amount of modular code. The basic code of the Pman Project is Pman.Base and Pman.Core, in themselves they do not provide much more than the basic Interface outline. 

There's quite a bit more code and data in those Modules, but what changed yesterday and has a significant benefit to the long term re-usabilty of all this code is the autogeneration of cached data for DB_DataObjects schema files, automatic merging of DB_DataObject links files, and the generation of code for Roo Javascript's Reader data.

The background here is that, as I re-used more and more of these componets, there would frequently be an issue where a particular project needed the modification of a core database component. In our example it was the adding of 'dispatch port' to the Company Table. This was a specific requirement for one project, which could have been impemented using a Companys_Extra table or some kind of 'properties' hack (which is what Midgard used to do back in early 2005). However the best solution is really to modify the base Table, so that it can be searched/ordered etc, and in the long run I'm looking at having a 'Designer Module' similar to SugarCRM / netsuite, that allows the end users to actually modify the database structure, adding colums as they need. 

The Knock-on effect however for this small change is, unfortunatly it often breaks other code depending on the compoents, due to the process that is involved.

a) Modify the database changes file. normally in the DataObjects directory of the Component

b) Run it against the Database, updating the Data Definition for tables in that module.

b) Run the generator on the module, so that the schema.ini file is regenerated.

This works fine on the development server, and usually fine on the Live server for that project, as you remember to run the database changes from the relivant Module when you do the next install, Where this falls apart is usually a month later...

You go back to working on a project that was on hold for a while, and start by updating the modules in the project. Suddenly Authentication fails, or no data is available in the interface... - All this is due to the fact that the Database Queries being built expect certain columns to be there (based on the schema.ini file) and since you have forgotten to update the database all the queries break.. Not usually a big issue as the builder module has some code in it to apply all the database changes code, by runing the bootstrapper in CLI mode. - But still it's annoying.

The other issue is that you really do not need the Company Table to contain a 'dispatch port' when no other project is going to use it.

I have been thinking about the solution to this for some time, however, just before I jumped into refreshing some code in a project, I thought this would be the ideal time to solve this problem, and speed up my development process in the long run.

The change involved

a) HTML_FlexyFramework changes

options:

dataObjectsCache = true (defaults to ON)

dataObjectsCacheExpires = 3600 (database Cache file expires every hour..)

$FlexyFramework->generateDataObjectCache(force=false) method

which can be called to force regeneration of the ini files (stored in the session directory)

HTML_FlexyFramework_Generator - a tiny wrapper around DB_DataObject_Generator, that is loaded and called only when regeneration takes place. 

b) Pman.php

- when you load the main page (not when you do JSON calls) it calls generateDataObjectCache(), which will generate the schema files if they have expired. - if you are running with Pman[isDev] then it will run them every time (eg. force generation)..

c) Pman/Roo.php

- new get Request argument _requestMeta

which adds to the data return value the JsonReader meta arguments which include schema type information, so you do not need the schema information in the UI (reducing the size of the UI considerably). note schema data is only returned if rows of data where also returned..

d) roojs1/Roo/data/JsonReader.js

- new property metaFromRemote - a flag to indicate that the meta data was recieved from a remote source (and hence probably does not need fetching again).

e) roojs1/Roo/data/Store.js

- load method modified to check for metaFromRemote, if it is set to false, then _requestMeta is added to the data Query. (hence schema is only queried once).

All this should mean that I can move application specific database changes to core tables into their respective modules, and out of core..