With Lisa at the helm, My Boss Is A Robot is going to take a slightly different path toward our goal of doing crowdsourced journalism. Partly that’s because our first attempt didn’t exactly produce stellar results. But it’s also because Lisa thinks about the problem in a different way. Here’s what she says about her plans going forward:

The initial attempts to create news articles hit two obstacles: low speed and low quality. Unlike simple tasks such as labeling images, fewer workers are willing to write news articles, which require more cognitive effort. Since workers are less willing, it takes longer to assemble a crowd and complete the task. Moreover, either for lack of ability, or because workers are not putting sufficient effort into the task, the quality of the crowd’s work is low.

In order to find talented and motivated workers, we need to build a stable workforce. Our approach is called invitation-based participation. Instead of posting tasks to the public crowdsourcing market, we will invite workers to join a writing team. Workers who accept enjoy special benefits. We’ll also focus on building a team with a shared goal, establishing a bond between the requestors and workers, as well as between workers themselves. The team environment also allows for feedback and integrated learning.

The idea of building a stable workforce in the crowdsourcing market was inspired by theories from several disciplines. Ronald Coase, an economist, explained the emergence of the firms in terms of transaction costs. There are many costs that occur in the processing of market-based transactions: for example, search and information costs and bargaining costs. Firms with long-term employees are able to save such transaction costs. Posting the writing tasks to the crowd in the crowdsourcing market leads to very high transaction costs: long waiting times, lower quality from the unfitted workers, and higher payments. By building a long-term relationship with the right workers, we can reduce such costs.

We have conducted some studies and the preliminary results show that invited-participation leads to better performance. Now we’re gong to experiment with various ways of building up a stable and high-quality workforce. We’re also trying to combine machine learning techniques with the crowd. We will first shorten the source articles using text summarization techniques, then let the crowd work on the shortened articles. This machine-assisted summarization process is expected to be an integrated part of crowdsourced journalism.

Are we stepping back from our original goal? In some ways, yes. We’d hoped to be able to create articles using any workers from Mechanical Turk. We’re admitting that, at least at this stage, we need more control over our workforce. Having said that, our workforce will still consist of untrained journalists. If the system works, we’ll still be creating articles rapidly and at low cost. And if it fails? Maybe we’ll have shown that journalism isn’t something that can easily be done by an algorithm and the crowd.

We’ll be back here with more results as soon as Lisa has them.

We’ve been stalled for a while looking for a new technical collaborator, but we’re happy to announce that the project is now back on track. Lisa (Lixiu) Yu is a new postdoc at the Human Computer Interaction Institute at Carnegie Mellon. She has done extensive research in the areas of crowdsourcing, creativity, and open innovation, with a focus on crowdsourcing complex and creative work. She’s a perfect fit for our project to crowdsource journalism, and we’re happy she’s signed on.

We’ll have a more extensive post coming up, detailing where we are, what we’ve learned, and what the plan is from here.

It turned out that we were a little over-confident when we said that a new researcher would soon be heading up our project. It didn’t work out with the person we had in mind, so My Boss Is A Robot remains temporarily leaderless. Our friends at Carnegie Mellon are on the hunt for a suitable candidate and we hope to have someone in place come January. If you think that person might be you, drop me an email at jg@jimgiles.net. You’ll need expertise in computer science, an interest in crowdsourcing and a desire to experiment with the journalistic process.

It’s been very quiet recently at My Boss Is A Robot. That’s not because we’ve given up. It’s because two team members have become fathers and Susheel Khamkar, who has been posting the jobs on Mechanical Turk and analysing the results, has finished his MS at CMU and taken a position at CrowdFlower. (Congrats, Susheel!) This has set us back a bit, but Susheel’s work is being handed over to another CMU researcher and the new fathers will soon find time to start working again. So don’t give up on us. We’ll have more results soon.

Our first experiments at crowdsourcing journalism hinted that writing would not be our big problem. The quality of the material we’ve been getting back is variable, but basically okay. Certainly good enough that an editor could knock it into publishable shape.

These results seem to be in line with other findings. Marti Motoyama and colleagues at the University of California, San Diego commissioned a bunch of articles from workers at Freelancer, a Mechanical Turk rival. (Freelancer jobs tend to be better paid and to involve more work than the “microtasks” posted on Turk). The results don’t make for thrilling reading, but most are reasonably well written.

Motoyama commissioned the articles as part of a study of the outsourcing of “web service abuse”, meaning offers of payment for shady tasks such as the creation of Gmail accounts for spamming purposes, or the posting of fake positive reviews. It turns out there is a lot of this stuff on Freelancer — check out his paper for details.

Freelancer is also used to generate content for spam purposes. This usually involves the creation of cheap 500-word articles that are used to fool Google and other search engines into giving a website a higher rating. I wrote about the process in a little more detail earlier this year after I interviewed one of the folks behind a similar scheme.

In this case, Motoyama commissioned 400-word articles on skin care products. He shared the results with me and they’re much the same as the stuff we’ve been getting back. Some are contain errors and horrible writing, but many do not. On the Flesch–Kincaid Grade Level, a measure based on word and sentence length, the pieces fall somewhere between Wikipedia and Cosmopolitan in terms of readability.  After a little quality control to weed out the worst workers, it’s seems that it be would relatively easy to recruit writers on Freelancer, just as I think it would be on Mechanical Turk.

That doesn’t mean that we’re finding our challenge easy. In our experiments over the past few weeks we’ve been struggling to get workers to summarize the papers as we’d like. It’s probably because we’re now experimenting with three papers that are more jargon-filled and conceptually complex that the study we used in our initial trials. We’ll update you in full once all the results are in.

As discussed last month, our initial attempt at crowdsourcing the journalism process led to a draft article that was okay, but not great. It certainly wasn’t good enough to be edited. If this were regular journalism, the editor would have emailed the piece back to the author together with a curt message about the need for cleaner writing.

Over the past couple of weeks, we’ve been talking with our collaborators at Carnegie Mellon. We know that we need to do a better job of quality control. Our system builds up each paragraph of the piece in parallel, which can mean that different workers create each chunk of copy. This should make the process quick and easy to outsource. But it also provides lots of opportunity for errors. We’ve been talking about how we can weed out inaccuracies as early as possible, before they become too great in number.

Part of the problem was, we think, carelessness on our behalf. We did a decent job of telling workers what to include in the story, but didn’t explain what to leave out. That led workers to produce jumbled paragraphs that included too much information.

We also want to incorporate fact-checking mechanisms into the early stages of our automated workflow. We’re going to experiment with a new round of tasks in which workers check the copy produced by other workers, so we can chuck out problematic paragraphs straight away. In fact, we think we might get better results just by telling our workers that this is going to happen. Research suggests that crowdsourced workers perform better when they know that their output is going to be compared with that from other workers, or checked for accuracy.

We hope to start running the new tasks this week or next. Results to come as soon as we have them!

First visit to My Boss is a Robot? Try starting with the about the experiment page.

As we dig into the philsophical roots of radical crowdsourcing, we are finding that the idea that humans can be organized and ultimately made subservient to machines is not particularly new.

The book Technopoly, written in 1992 by American cultural critic, Neil Postman, lays out much of the history of society’s relationship with technology and, though it was written before the widespread use of computer networks, proves somewhat prescient about where the unexamined adoption of information technologies might lead.

Postman frames his argument with the story of King Thamus, described in Plato’s Phaedrus. Thamus is presented with a number of inventions–calculation, astronomy, and writing, to name a few–but is unimpressed. He takes specific issue with writing, claiming that it will cause people to stop remembering things in favor of writing them down, leading them to gain much information, but little wisdom. To the inventor, Theuth, he admonsishes, “the discoverer of an art is not the best judge of the good or harm that will accrue to those who practice it.”

In broad terms, Postman then describes the evolution of society from tool-using cultures where technology is integrated into daily life but is bound by a social or religious worldview, through technocracy–where technology plays a central role in a culture, becoming a source of social power and challenging traditional structures–and finally technopoly, which Postman calls the “submission of all forms of culture to the sovereignty of machines and technology.”

Postman writes that we are entering a period of technopoly, the immediate roots of which stretch back at least as far as Frederick Taylor, the father of scientific management. Taylor was famous for efficiency studies in which repetitive tasks were measured and standardized similarly to machine parts. Postman sees in this behavior “the first statement that society is best served when human beings are placed at the disposal of their techniques and technology, that human beings are, in a sense, worth less than their technology.”

The rest of the book examines the dawning information age and its attendant glut of data which, Postman argues, encourages an over-reliance on statistics–an “invisible technology”–to replace social or religious sources of meaning. Postman also has a special loathing for computers, which he sees as usurping decision-making power from people.

Despite immediate impressions, the book is more than just the ranting of a Luddite; Postman readily acknowledges the many conveniences and advances that technology has enabled. He simply cautions that society take a note from King Thamus and look carefully at what is lost when a technology is adopted.

One wonders what Postman, who died in 2003, would make of crowdsourcing sites like Mechancial Turk. The idea seems an almost pure expression of technopoly–people literally take direction from, and are compensated by, computers. Ever the technological skeptic, he might take heart to know that there may be a limit to this trend, perhaps demonstrated in some small way by the fact that our experiment to completely mechanize the process of writing a news article isn’t going very well.

Posted by MacGregor Campbell at 9:49 am on June 1st, 2011. No comments... »
Categories: Uncategorized.