I took my time to take this in, because though fairly proficient at objective programming, I tend to use objects as structs on steroids. That is, I abstract data structure manipulation rather than control flow through them. Mostly, I guess, because I'm an autodidact and haven't looked into any formal OO design methodologies.

So what I'd like to hear about is how the choice I would have made differently might affect the flexibility and overhead of the design. The point I depart is around pass 8 to 10, where you winnow out some redundancy by introducing an error state. At that point, I would have submerged the state register in the state object and made the transition-for method a mutator, so my personal roadmap might look something like this:

    state->start
    input->remember-starting-position
    
    while (input->has-more) {
        if (state->attempt-transition-for (input->current-char)) { # m
+ove
            if (state->accepts) {                                  # m
+emo
                last-accept-state = state
                input->remember-accept-position
            }
            input->advance                                         # a
+dvance
        } else {
            push (output-queue, token {                            # t
+oken
                type = last-accept-state->type
                value = input->last-accepted-string
            })
            input->rewind                                          # r
+ewind
            state = start                                          # r
+estart
            input->remember-starting-position
        }
    }
[download]

Applying similar transformations as your following steps, we might arrive at something like the following. (Note: I juxtaposed the token/rewind steps so type and string could become simple accessors to the current configuration.)

    state->start
    while (config->has_more_input) {
        if (state->attempt_transition_for(config->current_input)) { # 
+move
            config->remember(state->type) if state->accepts         # 
+memo
            config->input->advance                                  # 
+advance
        } else {
            config->rewind                # rewind
            push (output_queue, token {   # token
                type = config->type
                value = config->string
            })
            state->start                  # restart
        }
    }
[download]

In this case, the states would be represented not as a state subclass each, but rather as a hash (dispatch?) table inside the state class.

Some further submerging reduces this down to

    config->start
    while (config->has_more_input) {
        if (config->state->attempt_transition) {       # move + advanc
+e
            config->remember if config->state->accepts # memo
        } else {
            config->rewind                        # rewind + restart
            push output_queue, config->token      # token
        }
    }
[download]

What specific drawbacks, if any, do you see in this approach? As far as I can tell, both solutions are equivalent on some level, although the type of abstraction differs. Is that so?

Makeshifts last the longest.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.