Introducing Orchestra

November will mark the 10-year anniversary of Amazon’s Mechanical Turk, which introduced the concept of a microtask to the world.

28 September 2015 · 5 min read

To computer scientists, microtasks posed a nice abstraction of labor: for a few cents, I can ask a human to do something a machine yet can’t, like tell me what’s in a photo or write up a short snippet of text about a topic. There are certainly benefits to Mechanical Turk’s model, including automating and distributing work to people anywhere in the world who can work where, when, and on what they want. The model opened the door to interesting innovations like near-realtime image labeling for the blind to new work opportunities for the developing world.

On the flipside, microtask labor dehumanizes the humans doing the work. It commoditizes the features that separate humans from machines: Click on the correct photo, and make sure to do it fast. Describe the product in one sentence, not two. We’re training algorithms to replace you, so we’ve filtered all of the nuance and ambiguity out of the work that might have made it interesting. Rather than nurturing and celebrating subject matter expertise as people work on more substantial tasks, the microtask model of doing work has turned work into the smallest bite-sized pieces that anyone can complete with little training.

Luckily, social scientists and computer scientists have kept exploring the role of experts on distributed digital teams. The most important recent paper in the space was the work on Flash Teams by Daniela Retelny et al. at Stanford. Through flash teams, the authors show that it is possible to automate away the tedious work of coordinating a team of experts to the point that they can deliver complex web/mobile apps or educational videos in half the time that they would if left to do their own planning. Two other papers (Peer Management by Anand Kulkarni et al. and Review Hierarchies by our own Daniel Haas et al.) showed that machines could match entry-level workers up with trusted experts who reviewed their work, resulting in higher-quality work and machine-mediated mentorship.

We’ve been pretty excited about and involved in the research in the space because, if harnessed correctly, the research can shape a brighter and more interesting future of work. We’ve also been frustrated by the lack of usable open source projects around these budding ideas. With the refreshing exception of Stanford’s Foundry project, most of the ideas from the papers we’ve read and written have stayed on paper. As a company that cares about the future of how creative and analytical people do work, we felt strongly that one of our first contributions to the world should be a collection of software artifacts that we can build a community around.

And so…


Hello, Orchestra!

Orchestra is an Apache-Licensed workflow management system. Orchestra is our stab at how software-defined work may look in the future. Through Orchestra, users define workflows that combine automation and coordination of expert teams to produce complex creative and analytical projects.

To understand how Orchestra can be used, we’ve thought through a few scenarios. For example, the newsroom brings together folks like reporters, photographers, and editors of various types. In our documentation we’ve put together an example workflow through which all of these different types of experts work together on a story. More experienced experts mentor less experienced ones, and various tasks like image cropping and resizing are completed automatically

There are many other uses we can imagine ranging from recruiting and conference planning to managing various legal workflows at a law firm. We’re even thinking of ways Orchestra can be used in some of the most creative fields, like design, and some of the most analytical, like data science.

What made it into v0.1.0

The version of Orchestra that you can find on github is the one that we’re using in production at Unlimited Labs. Here are some of the most exciting features as of v.0.1.0:

  • Workflows with humans and machines. Orchestra allows you to define workflows through which experts can contribute work and algorithms can contribute computation. This allows, say, a photographer to upload a bunch of photos, and an autocropping algorithm to resize those photos before a copy editor provides captions for them. In addition to a tutorial workflow in our getting started in 5 minutes guide we provide an illustrative example of the following workflow implemented in Orchestra:

  • Review hierarchies. Knowledge work necessitates feedback. From code reviews to design critiques, professionals build review into their process. Orchestra allows a workflow designer to specify that particular workflow steps should be serially reviewed by any number of experts, so that everyone from apprentices to skilled reviewers can learn from someone else.

  • Certifications. Photographers and reporters are often not the same person. Experts in Orchestra receive certification to do certain types of tasks, and each step in a workflow can specify which certifications are required to complete it.

  • Slack/Google Drive integration. Teams in Orchestra collaborate using a number of tools. Orchestra’s Slack integration invites team members to a private Slack group as they join a project. It also does things like automatically creating Google Drive folders so teammates have places to share their work.

  • Project API. Orchestra exposes an HTTP API to create and monitor projects as experts and algorithms contribute to them. This allows other services to kick off projects, report on their progress, and pull structured data out of them after they finish.

The limits of technology

Orchestra is technology, but digital labor is a sociotechnical concern. While we built Orchestra because we believe that the infrastructure to empower distributed teams was lacking, software will only solve some of these problems. Some of the hardest problems we face daily are not technical at all, and we are both humbled and excited to participate in the discussion around the social parts of the infrastructure. One example challenge is in the world of recruiting, vetting, and onboarding experts, which is just as hard to scale and has just as many domain-specific nuances as it ever did.

You should realize that by using this software, you’re going to run head-first into topics like labor models, organizational psychology, and operations research. Tread lightly, and remember that Orchestra, and technology more broadly, is not a silver bullet to address all of these considerations.


Getting involved

We plan to work on Orchestra for the coming decades. There’s so much that Orchestra is not yet capable of, and we’re excited to improve on it. For starters, we’re looking at building a user interface for defining workflows, various interfaces for providing feedback during review, better review policies, and better expert and task modeling.

We can’t wait to see what you do with Orchestra. If we can help with anything, please reach out. Here are a few quick links to get you started:

Read next

See all
Hello World

Hello World

The world of work is changing. Machines are getting smarter and jobs are often defined in minutes, not years. Technology is changing how we work an...

Read now
Introducing B12

Introducing B12

We are thrilled to share today that B12 (formerly known as Unlimited Labs) has raised a $12.4M Series A. We’ll start by telling you about what we a...

Read now
How B12 Hosts Websites at Scale

How B12 Hosts Websites at Scale

At B12, we offer customers a new way to create and manage their website. Customers provide us with some basic business information, and in 60 secon...

Read now
Introducing Kinase, B12’s Web Content Labeling Framework

Introducing Kinase, B12’s Web Content Labeling Framework

The web contains a variety of content that is easily accessible to humans but ill-formed for machines. Many organizations have spent countless hour...

Read now
This website uses cookies to ensure you get the best browsing experience.  Learn more
I agree