Experimental success, theoretical debacle

(This post is an attempt to catch up with October subjects that were being trendy when I was on leave from blogging, even though I suppose no one cares anymore)

This year's Nobel prizes were by all means exceptional. In blatant disregard of noble traditions, the prize in physics was given for a groundbreaking(!) and recent(!!) discovery without omitting any of the key contributors(!!!). Indeed, the discovery of accelerated expansion is one of the greatest triumphs of modern science. The measurements of supernovae brightness in the 90s and subsequent experiments have demonstrated that the universe is currently dominated by a form of energy characterized by negative pressure. In fact, this "dark energy" has the properties of the vacuum energy aka the cosmological constant, first introduced by Einstein for completely wrong reasons. In science, experimental progress usually brings better theoretical understanding. And that's another exceptional thing about the recent Nobel: almost 15 years after, the understanding of the cosmological constant in the context of particle physics models is as good as non-existent.

The cosmological constant problem has been haunting particle physicists for nearly a century now. We know for a fact that all forms of energy gravitate, including the energy contributed by quantum corrections. Thus, we know that diagrams with a graviton coupled to matter loops, like the one in the upper picture, yield a non-vanishing contribution to scattering amplitudes. On the other hand, the sum of very similar diagrams with graviton coupled to matter loops in vacuum must be nearly zero, otherwise the approximate Minkowski vacuum in which we live in would be destabilized. The contribution of the electron loop alone (the lower picture) is about 50 orders of magnitude larger than the experimental limit. On top of that, there should be classical contributions to the vacuum energy, for example from the QCD condensate and from the Higgs potential, which are also naturally tens of orders of magnitude larger than the limit.

The usual attitude in theory is that when something is predicted infinite one assumes it must be zero, and that was a good enough approach before 1998. The discovery of accelerated expansion was a game-changer, because it experimentally proved that the vacuum energy is real and affects the cosmological evolution, therefore the problem can no longer be swiped under the carpet. In fact, the problems is now double. Not only we need to understand why the cosmological constant takes a highly unnatural value from the point of view of the effective low-energy theory (the old cosmological constant problem), but we need to understand why it is of the same order as the matter energy density today (the coincidence problem).

Neither the first nor the second problem has found a satisfactory solution to date. Not for a lack of trying. People have attacked the problem via IR and/or UV modifications of gravity, quintessence fields, self-tuning or attractor solutions, fancy brane configurations in extra dimensions, elephants standing on turtles, space-time wormholes, etc, see also the comment section for crazier examples. In vain, all these solutions either rely on theoretically uncontrollable assumptions, or they just shift the problem somewhere else. The situation remains so dramatic that there are 2 only solutions that are technically correct:

1. The anthropic principle: the cosmological constant is an environmental quantity that takes different values in different patches of the universe, however more-or-less intelligent observers can see only those tiny patches where it is unnaturally small.
2. The misanthropic principle: the cosmological constant is being adjusted manually by seven invisible dwarfs wearing red hats.

Both of these theories have a comparable predictive power. In the first case we currently have no way to know the fundamental theory that sets the statistical distribution of the cosmological constant. In the second case we don't know what the little bastards are really up to.

Maybe theory needs another clue that may be provide by one of the future experiments. The Planck satellite will publish an update on cosmological parameters in 2013, although the rumor is that there won't be any revolution. In the asymptotic future there is ESA's Euclid satellite who will precisely measure the distribution of dark matter and dark energy in the universe. Will I live to see the day when the problem is solved? My bet is that no, but I'd love to proven wrong...

For the best summary of the cc problem read Section 1 of Polchinki's review.