Wednesday 24 September 2014

BICEP: what was wrong and what was right

As you already know, Planck finally came out of the closet.  The Monday paper shows that the galactic dust polarization fraction in the BICEP window is larger than predicted by pre-Planck models, as previously suggested by an independent theorist's analysis. As a result, the dust contribution to the B-mode power spectrum at moderate multipoles is about 4 times larger than estimated by BICEP. This implies that the dust alone can account for the signal strength reported by BICEP in March this year, without invoking a primordial component from the early universe. See the plot, borrowed from Kyle Helson's twitter, with overlaid BICEP data points and Planck's dust estimates.  For a detailed discussion of Planck's paper I recommend reading other blogs who know better than me, see e.g. here or here or here.  Instead, I will focus and the sociological and ontological aspects of the affair. There's no question that BICEP screwed up big time. But can we identify precisely which steps lead to the downfall, and which were a normal part of the scientific process?  The story is complicated and there are many contradicting opinions, so to clarify it I will provide you with simple right or wrong answers :)

  • BICEP Instrument: Right.
    Whatever happened one should not forget that,  at the instrumental level,  BICEP was a huge success. The sensitivity to B-mode polarization at  angular scales above a degree beats previous CMB experiments by an order of magnitude. Other experiments are following in their tracks, and we should soon obtain better limits on the tensor-to-scalar ratio.  (Though it seems BICEP already comes close to the ultimate sensitivity for single-frequency ground-based experiment, given the dust pollution demonstrated by Planck).  
  • ArXiv first: Right.
    Some complained that the BICEP result were announced before the paper was accepted in a journal. True, peer-review is the pillar of science, but it does not mean we have to adhere to obsolete 20th century standards. The BICEP paper has undergone a thorough peer-review process of the best possible kind that included the whole community. It is highly unlikely the error would have been caught by a random journal referee. 
  • Press conference: Right.  
    Many considered inappropriate that the release of the results was turned into a publicity stunt with a press conference, champagne, and  YouTube videos. My opinion is that,  as long as they believed the signal is robust, they had every right to throw a party, much like CERN did on the occasion of the Higgs discovery.  In the end  it didn't really matter. Given the importance of the discovery and how news spread over the blogosphere, the net effect on the public would be exactly the same if they just submitted to ArXiv.  
  • Data scraping: Right.
    There was a lot of indignation about the fact that, to estimate the dust polarization fraction in their field of view,  BICEP used preliminary Planck data digitized from a slide in a conference presentation. I don't understand what's the problem.  You should always use all publicly available relevant information; it's as simple as that. 
  • Inflation spin: Wrong.
    BICEP sold the discovery as the smoking-gun evidence for cosmic inflation. This narrative was picked by mainstream press, often mixing inflation with the big bang scenario. In reality, the primordial B-mode would be yet another evidence for inflation and a measurement of one crucial  parameter - the energy density during inflation. This would be of course a huge thing, but apparently not big enough for PR departments. The damage is obvious: now that the result does not stand,  the inflation picture and, by association,  the whole big bang scenario is  undermined in public perception. Now Guth and Linde cannot even dream of a Nobel prize, thanks to BICEP...  
  • Quality control: Wrong. 
    Sure, everyone makes mistakes. But, from what I heard, that unfortunate analysis of the dust polarization fraction based on the Planck polarization data was performed by a single collaboration member and never cross-checked. I understand  there's been some bad luck involved: the wrong estimate fell very close to the predictions of faulty pre-Planck dust models. But, for dog's sake, the whole Nobel-prize-worth discovery was hinging on that. There's nothing wrong with being wrong, but not double- and triple-checking crucial elements of the analysis is criminal. 
  • Denial: Wrong.
    The error in the estimate of the dust polarization fraction was understood soon after the initial announcement, and BICEP leaders  were aware of it. Instead of biting the bullet, they chose a we-stand-by-our-results story. This resembled a child sweeping a broken vase under the sofa in the hope that no one would notice...  

To conclude, BICEP goofed it up and deserves ridicule, in the same way a person slipping on a banana skin does. With some minimal precautions the mishap could have been avoided, or at least the damage could have been reduced. On the positive side, science worked once again, and  we all learned something. Astrophysicists learned some exciting stuff about polarized dust in our galaxy. The public learned that science can get it wrong at times but is always self-correcting. And Andrei Linde learned to not open the door to a stranger with a backpack.

29 comments:

Anonymous said...

I disagree about the Youtube video announcing to Linde "smoking gun evidence," showing him tearing up, all before even referring of their result. That showed not only extremely poor political judgement but just plain old bad taste.

Jin He said...

I am always confused why scientists do not both to study the structure of normal galaxies which fall under a few types (thin exponential disks, or "ellipticals") that are available as images on the public internet. Instead they try to claim the evidences of the first a-few-seconds picture of the origin of the whole universe.

Anonymous said...

I think the hype was where they went wrong myself, particularly the Linde video. It got people's backs up. But I don't think this affair undermines "the big bang narrative". Yes, it undermines "the inflation picture", but IMHO that's a good thing. I think big bang cosmology is better off without it.

Anonymous said...

Be very careful of that last "denial" point. My understanding is that the team has stood by the measurement of B-modes on the sky, which no one is casting doubt on. The interpretation in terms of gravitational waves or dust has had a "we'll see what the data show" preface for some time.

Unknown said...

On point 1: BICEP2 has gotten close to the limit of what can be done with a single frequency. Planck has shown that there is dust polarization almost everywhere. However, multiple frequency experiments from the ground, balloon and space can all go much deeper than BICEP2 and should be sensitive to r down to 0.01 and possibly 0.001. BICEP2 was an important step forward in sensitivity; however, there are many more steps forward in the near future.

Jester said...

Thanks David, I wasn't aware one can improve the sensitivity by that much.

Unknown said...

Hopefully the many steps forward in the future have funding.

Anonymous said...

david spergel, you say ground, balloon and space measurements can go down to ~r=0.001, can you clarify the theoretical limits of earth-based vs space-based measurements - cheers?

and, related, Jester, I would say Planck screwed up a bit too, at a far larger cost, they didn't get close to the sensitivity required to conclusively dismiss BICEP2's result. Instead we have to wait for new ground based measurements, probably from the same BICEP team.

In future, when planning hundred millions of euros project, perhaps wait a couple of years until the technology can measure at the required sensitivity. Just a couple of years, not decades..., just sayin'

Anonymous said...

The other problem is that now, thanks to the reverse indignitation press that has been generated by the Planck release the public 'knows just knows' that Bicep is 'wrong'. And so if in the future it is demonstrated that some part of the signal remains and leads to inflation, it will require quite a bit of explaining.

Sigh!

John Burger said...
This comment has been removed by a blog administrator.
Anonymous said...

I went to ask some Planck people over email some details. I was going to click on names to grab e-mail addresses, but they weren't on the author list. I looked at the acknowledgements. BICEP2 isn't listed. Did the Planck team even float this paper by the BICEP2 team at all? They are collaborating, aren't they? There's good, bad, and ugly. This is ugly.

Jester said...

@ John Burger: ."..Comments should not be grossly offensive to third parties..." :)

Torbjörn Larsson said...

I like your right or wrong opinions, except perhaps the last iffy "denial" point. But since I am Swede I have to protest your unsupported taring of the Nobel Prize. =D

[Also, in retrospect, this poor uninformed layman thinks the BICEP collaboration single frequency instrument was too cheap/Antarctic too expensive. As I understand it something like Planck could deduct the dust directly. True, you have to start somewhere...]

Too bad it wasn't a confirmation. Now we have to wait some (years, I guess) more...

Antonio (AKA "Un físico") said...

I do not agree with your Wrong/Right view. From the very first moments back in March people could understand that Bicep2 results had to be wrong. Why?:
From a initial design of two frequencies with the same resolution measuring at the same location of the universe, the design was shifted to only one frequency in order to reduce the signal/noise ratio. This was a huge design mistake. Because the Cl coefficients must be measured not only in relation to noise but also to dust. And what Bicep2 did: setting dust to zero is not science. That obtained results like r not zero at 7 sigmas was then a fictitious discovery. In my opinion, Bicep2 manipulated their simulations to gain worldwide popularity and in my opinion Bicep2 should apologize.

Sesh Nadathur said...

Anonymous: I'm told Planck sent a draft of the paper to the BICEP2 team for comments well in advance of putting it on the arXiv.

Re David Spergel's comment, I wonder if the future is now going to be in some kind of template-based dust removal, somewhat in the spirit of how galactic foreground is removed from the temperature maps. Presumably that will require more than two or three frequencies?

Anonymous said...

A new paper by Colley&Gott suggests that there is a way to show that the BICEP2 results are not entirely from dust. They look at the Gaussian nature of the signal, and estimate that the value of r is 0.11+/- 0.04 (1 sigma).
http://arxiv.org/abs/1409.4491

As such, I think that it's too soon to say that the BICEP2/KECK team have done anything completely wrong.

Remember that 'all dust theory' can't explain the drop in signal at the two lower values of multipole (for both BICEP2 & KECK.)

As such, we should all be a little more patient and wait for new results to show up before judging the merit of their conclusions.

Alex said...

I think this has done significant damage to the repute of the entire field.

I have come across several horribly misleading science news items (some in national media) in which the doubts concerning BICEP were reported, and no clear distinction was made by the journalists between evidence for tensor modes and the evidence for big bang cosmology as a whole.

Because of the lack of understanding and sloppy research on the side of journalists, these results were reported to the general public as calling into question big bang cosmology as a whole, and that's an infuriating PR desaster.

Theo Nieuwenhuizen said...

I agree with most of Jester's "rights" and "wrongs". But obviously they should have waited for the first referee report before claiming victory. From Jester's arguments, it is clear that arXiv posting thus had to be delayed.
This is once more a proof that press-hyping and twitter-physics damage the reputation of science as a whole.
On top of that, it shows once more that down-to-earth explanations need much more credit. Should I recall faster-than-light neutrinos, emergent-gravity, SuSY at LHC, WIMP dark matter? Science seems not to be so concocted; there is a difference between the moderate complications in Nature and the endless complications that modelling offers. Did anybody count the number of epicycles in LCDM?

Anonymous said...

It was reported that one of the BICEP2 PI's admitted that they did not understand what the scraped data represented, even after talking with Planck researchers. They made a guess and went ahead and used the scraped data. This goes way beyond the question of whether it is allowed to use preliminary data. It is clear that the BICEP2 effort was rushing to get their claim of discovery out there, while covering their posteriors with a few standard caveats if it all fell apart. This happens often in our awards, honors, prizes hungry culture, as evident by Theo Nieuwenhuizen's list above. I can add many to his list from outside the field of HEP.

Anonymous said...

If Planck is correct, the new upper bound on the tensor-to-scalar ratio is given by arXiv:1409.7025: r<0.083. Of course, there might be a window for BICEP or KeckArray.

Anonymous said...

I think the BICEP2-story has made a positive contribution for the public understanding of the nature of science.
Science has always been mean and dirty when claims of discovery are at stake (e.g. Newton vs. Leibnitz).

Truth said...

Indeed the BICEP claims have come crashing down. The celebrations with champagne and Andrei Linde was inappropriate and ridiculous. Kind of a sad story in some ways, both from the experimental side and the theorists (Linde especially) too. At least we again see science in action, correcting this terrible mistake. A painful process, but necessary all at once.

The only real result is that the tensor to scalar ratio is now constrained better than ever at r<0.083, which makes inflation marginally less likely than before; though not a huge change.

Anonymous said...

Sorry Jester, there is no wrong and no right in this; this is just science in action: claims of discovery, followed by refutal or confirmation by other experiments looking for the same stuff, and so on. Science, no more, no less. Now, did BICEP2 hyped a little bit the story? this is a matter of appreciation, especially now that research institutions are ran (*marketed*) almost as burger joints. By now, no one should be over-excited by this type of publicity. Examples of this abound: OPERA, LSND, the Cabrera monopole event, DAMA, LEP2 115 GeV Higgs, the 17 KeV neutrino, just to cite a few. Back to work buddies.

Truth said...

@ anonymous. There is right and wrong in science. And there is right and wrong in how to conduct science. Jester is correct in pointing out that one should have double-triple checked the analysis fir dust, and not made a claim of discovery about GWs, when the evidence was inconclusive.

Anonymous said...

Not sure what the author means by "obsolete 20th century standards", but in any case arXiv is and always has been a preprint server. Prior to arXiv, papers when submitted to journals were also distributed to university libraries as 'preprints' for timely dissemination of results. That is what the word 'preprint' means.

Anonymous said...

Dear @Truth, yes there is right and wrong in science, however as far as the science part of BICEP2 is concerned, everything has been done by the book, and the results has been validated for publication in PRL. The foregrounds have been addressed by the BICEP team the best possible way, considering that BICEP is useless without detailed dust maps. These last are the monopole of PLANCK, whose cost is about 2 orders of magnitude BICEP's budget. Now, what was wrong, as i say is the non-science stuff, and this is not unusual nowadays considering the huge competition, the stakes (a Nobel) and the crisis we live in. Take it or leave it, end of story.

Anonymous said...

@Truth: you are confusing things: science is not an opinion. BICEP's science part was done correctly and in the best possible way, and without having access to PLANCK dust maps. At the end, it was validated by a publication in PRL, period. The non-science part is discutable.

Anonymous said...

Now at least several theoretical papers for solveing the "tension" between Planck and BICEP2 in PRL seem wrong! Right?

Truth said...

Making a claim of discovery when the evidence was very inconclusive was not good science. I don't understand why "anonymous" is so confused about this basic idea.