cue circus music

Executive Summary

Bigtop is a language for describing database backed web applications and a code generator to make web apps from those descriptions.

Bigtop allows for generation of...

It is quite different from code generators provided by web app frameworks. What distinguishes it is...

Background

I'm about to tell you a story. It's the story of invention. As I am the inventor I can't tell whether the invention is the best thing since sliced bread or a candidate for the most outrageous category on the American Inventor TV show. You'll have to decide whether I've made a better bread slicer or a Perfect Pet Petter. (For those who didn't see it: The Perfect Pet Petter has a motion sensor, so it knows when a dog is ready for petting, and a mechanical hand to do the petting. Very scary.)

It was Fred Brooks who told us that revealing our table structure would make the workings of our apps obvious. Some may think that the wide adoption of object-oriented programming and the advent of web delivered apps obviates Mr. Brooks assertion. I don't.

I had been out of the data driven app development business for quite a while, when I took my current job a little more than a year ago. After coming up to speed on mod_perl and Postgres, I discovered that our apps are strikingly similar to the academic records apps I worked on during my first job, as a registrar's office programmer for a small college, twenty years ago.

While the apps look nicer and generally run faster, their basic ideas have not changed. They are still firmly centered around their data models.

As I worked on the tedious bits of a new app last year I found myself violating the first principle of coding I learned in college: don't duplicate code. (Since I used to teach college Math and Computer Science, I don't use the term Don't Repeat Yourself or DRY. Just try to imagine a college teacher who never repeats himself.) Many others had already discovered and named the bit of code I was writing over and over. It was CRUD (Create, Retrieve, Update, and Delete).

There are many web frameworks. My guess is most of them started to avoid hand coding CRUD. They must have all started on the premise of duplication reduction.

Since we were using a framework, yet were forced to repeat ourselves, we began to think that a major revision of the framework was in order. This led to (Gantry), which kept the best bits of our shop's original framework while eliminating yet more duplication.

At the risk of shameless self promotion -- since I wrote the relevant parts -- I'll assert that Gantry has a pair of nice CRUD schemes. The first, called AutoCRUD, does what you want for the 60-80% of your tables which are dull. The second, called CRUD, takes care of the others, with a bit more help from you.

Having test driven the CRUD schemes in Gantry, I wanted more. I didn't want to have to write an SQL schema file describing the table structure, then go back and write object relational mapping code (for Class::DBI or something similar), and still have to return a third time, to the same data model, to describe its appearance for an on-screen form.

I wanted a single place to say everything that needed saying about the data. That place is now a bigtop file. In it you can fully describe your data model and its controllers. A quick generation step builds the app. Further, the code it builds is always separated into two pieces. One piece is for you to revise and expand, to do the things that make your app different from others. The other piece is generated and safely regenerated to allow for changes to the data model.

Bigtop already supports Gantry with a choice of three database engines and three object relational mappers. But it allows for backend plugins to generate whatever is needed for your app. Backends can easily add keywords to the grammar and are free to generate whatever they like. So it is possible for Bigtop to support other frameworks. Since I'm rather committed to Gantry, it probably won't support others unless someone else writes the backends.

Example

Here I will show the smallest app I can think of (that does anything useful) with bigtop. It manages one table which stores names and phone numbers. I built this app with the tentmaker, a browser delivered bigtop editor. (See the following section for more info on tentmaker.) I did beautify it slightly for readability.

If you want to understand better what the bigtop description means, see Bigtop::Docs::Tutorial. I won't show details here.
config { engine CGI; template_engine TT; Init Std { } CGI Gantry { with_server 1; } Control Gantry { } SQL Postgres { } Model GantryCDBI { } SiteLook GantryDefault { gantry_wrapper `/home/pcrow/srcgantry/root/sample_wrapper.tt`; } } app Contact { config { dbconn `dbi:Pg:dbname=contact` => no_accessor; dbuser some_user => no_accessor; dbpass `$ecret` => no_accessor; template_wrapper `genwrapper.tt` => no_accessor; root `/home/pcrow/Contact/html:/home/pcrow/gan/root` => no_accessor; } authors `Phil Crow` => `philcrow2000@yahoo.com`; sequence contact_seq {} table contact { sequence contact_seq; field id { is int4, primary_key, auto; } field name { is varchar; label Name; html_form_type text; } field phone { is varchar; label Phone; html_form_type text; } } controller Contacts is AutoCRUD { rel_location contacts; controls_table contact; text_description Contact; method do_main is main_listing { cols name, phone; header_options Add; row_options Edit, Delete; title `Contact Management`; } method form is AutoCRUD_form { all_fields_but id; } } }

Building and Running

With the bigtop file in hand (I'll call it contact.bigtop), you can be running your app in a few steps:
bigtop --create contact.bigtop all cd Contact createdb contact -U postgres psql contact -U regular_user < docs/schema.postgres ./app.server [ port_defaults_to_8080 ]
(Supply database passwords as needed.)

You are ready to use the app through a browser.

You may need to update the data model, even after you've added code to Contact/Contacts.pm. This is a completely safe operation (but source code control is always a good idea).

Simply edit the bigtop file, in the docs subdirectory of the working directory, (with your favorite editor or tentmaker). Save the result and rebuild:
bigtop docs/contact.bigtop all
Note that you may want to turn off generation of things which shouldn't change. Do this by adding no_gen statements (or checking no_gen boxes in tentmaker):
Init Std { no_gen 1; }
Turning off Init generation is particularly common, since failure to do so invites bigtop to overwrite README, Changes, etc.

You can turn off generation for any backend in this way. Init Std also allows you to control regeneration of individual files, so README can be safe from overwriting while the MANIFEST is kept up to date. Further, you can include no_gen statements at the table and controller levels too (as well as for the whole app, a feature for the truly paranoid).

Making Tents

Even as the bigtop language has grown, the basic structure remains simple. But that simplicity of structure is often lost in the complexity of keyword proliferation. So, tentmaker comes to the rescue.

The structure of bigtop can be described briefly:
There are two top level blocks: config and app. In the config block there are statements and backend blocks. In the app block there are sequence, table, config, and controller blocks (plus a set of literal statements). All blocks allow statements; in addition, table blocks have field blocks, while controller blocks have methods. Finally, controller blocks also have literal statements.
But this structure still leaves a little to be desired. As note above, the problem is the volume of useful keywords. While each has its place and use, remembering them is hard.

After much thought and several abortive attempts at helping you write bigtop files, I began tentmaker. It allows you to edit your bigtop files with a browser (as long as your browser is DOM compliant, think Firefox not IE).

If you've install bigtop and the tentmaker templates that come with it, you can start tentmaker easily enough:

tentmaker [--port=8081]
Then point your browser to the port you chose. tentmaker looks like a complex wizard, but is a reusable editor of bigtop files. It can do everything except remember comments and preserve all whitespace. (Well there is also one statement it can't handle, but you don't need it).

Learn More

To start using Bigtop see Bigtop::Docs::TentTut or Bigtop::Docs::Tutorial.

Replies are listed 'Best First'.
Re: Introducing the Bigtop Web App Language
by sh1tn (Priest) on Jun 10, 2006 at 22:27 UTC
Re: Introducing the Bigtop Web App Language
by zby (Vicar) on Jun 12, 2006 at 08:11 UTC
    To add a shameless plugin - there is also Catalyst::Example::InstantCRUD. It creates a CRUD application with all tables in a database - so you need only create the tables in SQL and than run: instantcrud.pl -name=My::App -dsn='dbi:Pg:dbname=Dname' -user=zby -password='pass'. You can then delete the code for tables that you don't need. It is in a more experimental phase now.
      This sounds like a good idea in some contexts. I like the fact that you can merely alter the tables in the database, then regen the app. But, the key to bigtop is that it leaves hand written code alone in such cases. While some apps are just database front ends, almost all of ours do at least a little bit more than that. So, they have custom code. With bigtop, we can regenerate to take new data model changes into generated code, without overwriting custom code. (Of course, we have to revisit hand written code, to make it reflect the updated data model. But regenerating leaves that code undisturbed.)

      Phil

        It's not that naive as you might think. Actually it generates only a directory structure with files with no code only a 'use base ...' declaration and a database description. I think the regeneration after a schema change can be made safe - that is it only needs to overwrite the database description and eventually add new classes but it does not need to overwrite the hand written code.