Generative Versus Discriminative Learning

Abstract

In this chapter, we review discriminative and generative learning more formally. This includes a discussion of their underlying estimation algorithms and the criteria they optimize. A natural intermediate between the two is conditional learning which helps us visualize the coarse continuum between these two extremes. Figure 2.1 depicts the panorama of approaches as we go horizontally from the generative criteria to discriminative criteria. Similarly, on the vertical scale of variation, we see the estimation procedures range from local or direct solutions optimized on training data alone, to regularized solutions that use training data and priors, to fully averaged solutions which attempt to reduce over-fitting to the training data by considering a full distribution on potential solutions.

All models are wrong , but some are useful. George Box, 1979