Battaglia, P. W., Kersten, D., & Schrater, P. R. (2012). The Role of Generative Knowledge in Object Perception. In J. Trommershauser, K. P. Körding, & M. S. Landy (Eds.), Sensory Cue Integration. Oxford University Press.

Summary

In this chapter, Battaglia et al. describe the differing roles of the generative process, people’s generative knowledge, and how they related to perception (particularly for object perception). They first define several challenges/observations regarding perception:

  1. The mapping from the world to our senses is often not invertible (e.g. recovering 3D shape from 2D data)
  2. Many sensory cues are not actually measurements of the relevant property we’re interested in, but provide only “auxiliary” information
  3. Sensory cues vary in quality relative to each other, depending on external and internal factors (e.g. fog, cataracts), and as a function of the world state.
  4. Objects’ spatial and material properties follow highly predictable patterns

Next, they define the sensory generative process to be the true process in the world that generates our sensations. In contrast, sensory generative knowledge is people’s assumptions about how the sensory generative process works. In some cases, the generative process and generative knowledge may be the same, though not necessarily. If generative knowledge is at least a good approximation to the generative process, though, it can provide important information about how to interpret our sensations (addressing the challenges listed previously). Battaglia et al. make the distinction between being subjectively optimal (in which people make optimal use of their generative knowledge, but the knowledge does not match the generative process), objectively optimal (in which people make optimal use of their generative knowledge, which matches the generative process), and suboptimal (in which people do not make optimal use of their generative knowledge).

Battaglia et al. also describe how a Bayesian observer model can account for several basic phenomena in perception: performing basic Bayesian inference, combining multiple cues, discounting nuisance information based on prior knowledge, and explaining away nuisance information based on auxiliary cues. In particular, discounting and explaining away rely heavily on generative knowledge, while basic Bayes and cue combination could theoretically be learned just via a discriminative mapping.

Takeaways

This chapter makes two important points:

  1. Our internal models aren’t necessarily accurate imitations of the true generative process, but that doesn’t mean that people don’t make optimal use of the knowledge they have. Hence, when constructing models of cognition, it is important to be explicit about what the generative process is in the world that is generating people’s observations, and what generative knowledge we as scientists think people actually have.
  2. Having structured generative knowledge is important, because it makes it easier to interpret ambiguous sensations into reliable perceptions. For simpler things, such as cue combination, we don’t necessarily need to have a full generative model since a simple discriminative model can often be sufficient. But, for reasoning about more complex systems, and to be able to explain phenomena like discounting or explaining away, generative knowledge is crucially important.