I good quote somewhere, that is entirely irrelevant, and mostly forgotten, went something like this: "The two worst ways to solve a problem are neural networks and genetic algorithms". It's not really an insult, but more of a statement of the A.I. pathology: The former must know when it has found the answer, and the later must know the solution and works on how to get there from the problem. In all, this impresses as me as being insanely cool in a computational overkill sort of way. But this is the sort of theory and weirdness I love to play with.

Without a doubt though, it's safer to play with this massive structures in something that has some decent garbage collection and memory management. Your choice on Inline::C shows extreme bravery -- but I guess the speed is required for something which is, by definition, inefficient. If you know how to solve a problem without a Neural Net or genetic algorithm, then you don't need the neural net.

I can't find it, but there was a neural net somewhere used to reproduce circuits humans had already invented. The designs left some extra resistors in strange places, and in many cases, scientists weren't exactly sure what they did. It is possible, in the future, to see more of this kind of work -- many problems exist where we can define the inputs and the outputs, but can't invent the middle layer.

Game on.

The quote actually sounds very reasonable. A neural network will frequently be the worst way of solving a problem. However, it has the advantage that, when there is not clear solution, it can frequently offer a reasonable guess, even with incorrect or incomplete information. The example I like to use when I explain it is "Verbal SQL". Imagine the following exchange:

Ovid: Who's the new guy with the blond hair working in accounting?
Bob: That's 'Jim', but he's actually a brunette and he works in human resources.

In this case, I got the hair color and the employee department wrong, but I still get good results. Further, I left out "hire date", but implied that the person was "new". It would be difficult to write an SQL statement that will get me the correct answer.

On the other hand, what if I were actually meaning the new blond lady working in accounting? Gender probably has a higher chance of being accurate then the department that a person works in, so Bob might have given me the wrong answer. Neural networks are prone to the same types of problems. They can ascertain strange correlations that we might not consider, but they can also give incorrect results, particularly if we ask poor questions.

Cheers,
Ovid

New address of my CGI Course.

Your example sounds more like an expert-systems kind of deal to me.

From what I follow, neural-nets map given input to expected output based on some function, but there is pretty much no way a neural net could respond to personal details of Jim. Unless of course you are talking about writing a neural-net to understand natural language, which is another animal entirely. A very cool animal, but one with spines, pointy teeth, and a bad disposition.

You're right on the "it's role is to guess" based on prior experience. Dead on. A genetic algorithm, on the other hand, seems to understand not the answer to a few questions, but the concept of "between these two, which is better", and it goes from there. Now, the real question is, which is the better choice for natural language-analysis? :)

Probably neither, right off, first some sort of model needs to be constructed, almost like those evil sentence diagrams I had to do back in English class. Man, I hated those.

Most neural-net work I see being done (for exposure to the field) is in photo-analysis. I.e, "is this a stop sign". The problem there is identifing what is a reasonable input -- percentage of red is not the best input, and writing a "eccentricity from standard octagon" detector might be complete heck to implement.

What I'm shooting at is, I guess, is what criteria are used to determine what inputs are quality and how to map the problem-space into reasonable-inputs? For any arbitrary problem, finding the right inputs are key to the neural-net, as you already know the output values you are using to train the net.

This is wonderfully off-topic, but I love it. Good to see there is an A.I. following in Perl as well as Lisp. (Lisp is fun, but it's like coding in Brain[] sometimes).

I've wanted to play with neural nets seriously since they were omitted during my A.I. class in college, and Russel and Norvig (still shelved, mostly unread), while canonical, isn't exactly clear-as-mud on the subject.

While R&N is an excellent general textbook, I'd hesitate to call it canonical.

If you're interested I'd recommend taking a look at Rumelhart & McClelland's classic two volume set "Parallel Distributed Processing: Explorations in the Microstructure of Cognition". While it was first published back in the mid-eighties it's still a great overview of the basics.

Masters' "Practical Neural Network Recipes in C++" is a good practical introduction.

Update: You may find the AI Depot a source of useful info and links too.

I good quote somewhere, that is entirely irrelevant, and mostly forgotten, went something like this: "The two worst ways to solve a problem are neural networks and genetic algorithms". It's not really an insult, but more of a statement of the A.I. pathology: The former must know when it has found the answer, and the later must know the solution and works on how to get there from the problem.

This is a slightly unfair characterisation. GAs progress by having a way of comparing solutions. Knowing whether one thing is better worse than another is very different from knowing what the ideal solution is. The same applies to neural nets.

The AI "pathology" that I have come across is people ignoring slightly less cool solutions. For example, given equal amounts of time, you'll often find that GAs will lose out to other stochastic methods like simulated annealing.

I can't find it, but there was a neural net somewhere used to reproduce circuits humans had already invented. The designs left some extra resistors in strange places, and in many cases, scientists weren't exactly sure what they did. It is possible, in the future, to see more of this kind of work -- many problems exist where we can define the inputs and the outputs, but can't invent the middle layer.

You may be thinking of Adrian Thompson's work (whose robots occasionally prevented me getting to my office back when I worked at Sussex Uni ;-) He evolved hardware solutions by applying evolutionary algorithms to field-programmable gate array's.

This is from memory - so I may be getting the details wrong.

As I recall he had evolved a chip to recognise different tones. When he looked at the solution on the chip it turned out that some of the cells were not connected to input or output, but when he disconnected them the chip suddenly stopped recognising the tones.

It turned out that the solution was taking advantage of capacitive/inductive affects between the connected and disconnected bits. It was hard to investigate because the FGPA only gave digital output so it was impossible to measure directly the analogue values that were causing the affect. Didn't even work if you setup the same configuration on a different chip.

I think they were going to build some specialised hardware to take analogue measurements - can't remember or never knew if they got anywhere.

Fascinating stuff.

I understand the one node for the output layer, but I can't use a linear activation function for a backprop network because that uses the derivative of the activation function for propogating the error back through the network. Sample error progogation code:

      for (out = 0; out < network.size.output; out++) {
        network.error.output[out] 
            = (network.neuron.target[out] - network.neuron.output[out]
+) 
              * sigmoid_derivative(network.neuron.output[out]);
    }
[download]

That fails because the derivative of a linear function will be 1.0, thus not allowing the network to learn from errors. Am I missing something basic here?

Cheers,
Ovid

New address of my CGI Course.

Am I missing something basic here?

hmmm .... I think so. not sure what exactly. so I'll just make some random statements and hope they help

1) if the derivative is one, the derivative is one. back propagation still works. everything 's cool.

2) your code is good if its intent is to back prop a layer that is known to be sigmoid. if you want to back prop a layer that is known to be linear you can just drop the derivative term altogether. you'll have to have some kind of layer-labelling scheme er something to decide which loop to call. (ditto for feed forwards).