There Is No Right and Wrong

Continuing this series of posts, answering questions from Milky Way Talk, MWP user UncleClover writes:

I’m a very fast learner, but only if there’s some sort of feedback going on letting me know what I’m doing right and what I’m doing wrong. Once an image that we notated has been processed by a professional, is there any way for us to look it up and find out how accurate or inaccurate our notations were?

Feedback wouldn’t even need to necessarily be an extra step, just having access to your own prior notations and being able to compare them to the “final conclusions” of the professional analysis would be useful.

And in a similar vein, user katieofoz writes:

I was just wondering about a few thing that someone here might be able to answer.

• Im assuming the data collected for each image is compiled and if a similar marking shows up a certain number of times someone (with actual qualifications) looks at the image and classifies it? Is this right?
• From the data collected is there anything that looks at how often a person is correct in their markings? Like some sort of way of ranking the user based on their observations?
• What is done with the compiled data afterwards? Is it based off the average of observations or are areas looked at by qualified individuals to map it more accurately?

An important thing I learnt during my PhD is that there’s an awful lot of work involved in research that you really don’t need a PhD for. Or even a degree. Just a pair of eyes will do, and a few fingers. The tasks you’re performing for Milky Way Project are a great example of that. I may well have a PhD, but I’m no better than any of our users at finding or drawing bubbles. So really, we’re all experts in this project, and as long as you’re genuinely drawing what you believe is a bubble, or a knotty thing, or a dark cloud, there is no “right” and “wrong”.

To be clear, all we’re doing in the science team is providing you with the dataset, and then gathering up all the drawings of all the users and merging all that information into a consistent catalog of objects. We implement a few quality control measures, for example we consider a new user’s first few drawings to be practice drawings and don’t enter those into our dataset – as UncleClover says, there is a learning process.

We make no judgment on whether a drawing is “right” or “wrong”, and we don’t process individual images or users’ drawings. Once you’ve had a little practice, every click counts. All the clicks go into a (giant!) database, and a computer program written by Robert Simpson analyses all the data.

This program essentially scans the whole are of the sky covered by our images and locates clusters of ellipses that users have drawn (you can see an example image with all the various drawings made my volunteers, above). Where we find more than a given number in a small region, we mark this as a bubble. The bubble’s properties that go into the catalog are calculated from averages of the individual users’ classifications – as katieofoz rightly suggests. So whether a bubble you drew ends up in the catalog really just depends on whether lots of other people agreed with you. Given that there’s no real “right” or “wrong” in MWP, we don’t rank the users. It’s not a competition! But our processing algorithm does allow the classifications by experienced users to count more heavily than those of newbies.

Rob wrote more about the procedure earlier on this blog, and of course this will all be described in a lot of detail in the forthcoming paper.

If you haven’t done so already and you’re interested in knowing more about the infrared bubbles in the interstellar medium, I’d invite you to read the original bubbles papers of 2006 and 2007, written by our science team members Ed Churchwell (U Wisconsin-Madison), Matt Povich (Penn State), Bob Benjamin (U Wisconsin-Whitewater), Barbara Whitney (U Wisconsin/Space Science Institute) and their collaborators. This paper is packed with information on the Spitzer surveys we’ve taken our data from,the difficulties in picking out bubbles and what the bubbles might physically represent. The references are below, and the paper should be available via the ADS link.

Churchwell, E., Povich, M., Allen, D., Taylor, M., Meade, M., Babler, B., Indebetouw, R., Watson, C., Whitney, B., Wolfire, M., Bania, T., Benjamin, R., Clemens, D., Cohen, M., Cyganowski, C., Jackson, J., Kobulnicky, H., Mathis, J., Mercer, E., Stolovy, S., Uzpen, B., Watson, D., & Wolff, M. (2006). The Bubbling Galactic Disk The Astrophysical Journal, 649 (2), 759-778 DOI: 10.1086/507015 [ADS]

Churchwell, E., Watson, D., Povich, M., Taylor, M., Babler, B., Meade, M., Benjamin, R., Indebetouw, R., & Whitney, B. (2007). The Bubbling Galactic Disk. II. The Inner 20o The Astrophysical Journal, 670 (1), 428-441 DOI: 10.1086/521646 [ADS]