Hierarchical representations

I've been reading from Principles of Neural Science (2000) by Kandel, Schwartz, and Jessell. (It's the required text for my Neural Basis of Human Movement class.) It's a big, pretty standard textbook. So far I've gained a lot of info on how the motor control centers are organized, something which my previous class (Neurobiology) didn't touch.

The main underlying idea I keep coming back to is hierarchy (both in terms of sensation and action). It makes so much sense. Low-level sensory inputs are combined into higher-level representations. In the cerebral cortex this occurs within modalities at first. Higher up in the hierarchy it occurs across modalities. So the highest levels of the sensory hierarchy represent very abstract states, combining all sensory input. Most of this seems to occur in the occipital, parietal, and temporal lobes.

Similarly, the spinal cord, brainstem, and parts of the frontal cortex appear to form a motor hierarchy. Currently, my best guess is that this hierarchy is arranged as follows, from lowest to highest level: spinal cord, brainstem, primary motor cortex (MC1), premotor cortex, supplementary motor area, presupplementary motor area, prefrontal cortex. If the prefrontal cortex is at the highest level, it might form an interface between the sensory and motor hierarchies... if they are totally separate structures (see next paragraph).

One of the big confounding issues for me is whether there are separate sensory and motor hierarchies. It might be possible that there is a single sensory-motor hierarchy. This would mimic the idea of hierarchical policies in reinforcement learning; each module in the hierarchy would be a mapping from state to action, so it would require both sensory and motor pathways.

So it might be a while before I (or anyone) understands the overall organization of the sensory and motor systems. The method of thinking that has been most help to me in this kind of situation is this: pick a hypothesis and assume it's true until proven wrong. The brain's overall organization is just so poorly understood that it's easy to get overwhelmed. I enjoy at least having a hypothetical model as a framework.

I plan to start working on an implementable hierarchical model soon. It might use radial basis functions like the current Verve agents do, but maybe not. It would be nice to have a totally self-developing hierarchy that could split nodes and add more resolution in the high-information parts of the state-action space. This is getting a little hard to picture, though, so I might make some drawings...

Verve 0.1.0 Release

I put a 0.1.0 source release on SourceForge last night. This is the "thesis" version that was used to generate the results in my MS thesis. Enjoy!

Pendulum Swing-Up and Cart-Pole Videos

I posted two videos: one of the pendulum swing-up task, and one of the cart-pole task. Check them out here: http://verve-agents.sourceforge.net/gallery

Version 0.1.0 should be ready within the next few days.

First software release soon

I'm planning on releasing an initial version of the Verve library soon. It will probably be the version I used for my master's thesis experiments. Even though I have lots of ideas for further development, I think I'll just release the "thesis version," mainly to have a snapshot of the software at the time I wrote the thesis. Not sure if I'll even reimplement the XML loading and saving code (which worked a looong time ago but got totally outdated after a ton of architecture changes... so I just commented it out). So look for something like version 0.1.0 soon.

Also, I've recently registered the project on SourceForge as 'verve-agents.' So now the main project website is http://verve-agents.sourceforge.net. (For the record, the old site was http://www.vrac.iastate.edu/~streeter/verve/main.html.) The new site is powered by the PHP-based Dokuwiki software, making it much easier for me to manage than a bunch of HTML files.