Shrinkage Estimator

Surprisingly few people I've met have encountered the shrinkage idea, even among the statistically literate ones. If you routinely estimate means of several groups of objects, this will likely come to be a surprising result for you. Here is a paper that describes how and why this estimate works, and when it may not the the best available option to use.

These days, however, with all the so-called "Big Data" hype all around us, shrinkage is a very powerful idea. In my experience, "Big Data" is rarely big in a sense most people think of it. True, the raw data amounts that are available can easily be in terabytes, but often these are data concerning a large number of individual entities, e.g., customers, and results are sought at the granularity of individual entity. For some of them, it is possible (there are customers who shop prolifically), but for most, we'll have a handful of observations at best. This is exactly the kind of situations where shrinkage estimator rocks.