Become Your Own Techmeme: Curating Big Data in the Cloud

Easily the largest contributors to ReadWriteWeb's overall traffic on any given day include technology news aggregation services, one of them being Techmeme. It's a service that pulls interesting headlines from amid the ever-flowing sea of content, and compiles them into an amalgamated front page of what's happening; it's directed toward a specific audience, which Techmeme hopes includes you. It's done largely by humans, as opposed to automated services that glean articles' potential relative interest level through semantic analysis.

The CEO of a company called Flow has demonstrated to ReadWriteWeb what he describes as a tool that conceivably enables anyone to do for their businesses or even for a mass audience what Techmeme does for its audience, and with relative ease. Eric Alterman (not the noted professor) showed ReadWriteWeb a tool created for his Flow Platform that essentially enables any user to navigate selected streams of digested Web information on any number of filtered topics, and effectively generate a cultivated, "curated" stream of related content. Imagine a Techmeme that pertains only to what you do and who you are.

Sponsor

One of Flow's clients, Alterman tells us, is a Fortune 15 company. (No, there's not a missing digit.) It has a customer base of 75,000, and currently the complaints system made available to them is a form on a Web page. The complaints are amassed every 72 hours or so, and a report is generated and sent to a representative via email.

For this company, Flow Platform enables the engineering of a kind of automated information stream called, easily enough, a flow. A content item may be directly attached to this flow by way of what's called a drop. Or, content from Web pages (including, for instance, the complaints data form) may be made to stream into the flow. Rules may be generated where specific fields (customer's name, location, product serial number, etc.) are translated into the flow. Filters may be employed to determine whether the complaint is on account of a damaged shipment, a faulty shipment or a bad experience with the salesperson. "You can actually ask questions," Alterman says. "Compare that to a product like Yammer or Chatter, where you really don't have any structure in the stream."

A notice from a customer for a damaged shipment can be routed to a stream directed immediately to the fulfillment department and the salesperson assigned to that customer, who then views that stream the same way they would anyone's Twitter stream. A rule may be created where orders over a particular dollar amount may be directed in a stream to the CFO. Another rule could have the marketing team's stream be updated whenever there's a request for materials.

"So the idea of thinking of the flow in the enterprise as real-time routing of information, begins to give you an idea of how the workflow of a company can be improved in a highly contextual way," Flow's CEO explains.

What's more, Flow Platform can be leveraged by that same Fortune 15 company to cultivate a content channel for its customers, including those who've posted complaints - "a flow that gets routed to your customers based on what they care about." Up until now, Flow's Fortune 15 client had been communicating with all its customers as a mass, by means of a handmade newsletter. "Once you set this up," he says, "you have a publishing system that allows you to contextually reach all your customers with new information, new products, new regulations, best practices... Whether your customers are talking to you or you're talking to your customers, [you have] information flow, and some intelligence to the way that information flows, instead of manually creating spreadsheets and documents and emails that have no way of routing intelligently, and that in no way scales when you don't have the staff to do it."

The tool Alterman demonstrated for us is not the Flow Platform itself, but rather a tool for using resources that is made available to all levels of users of the Flow Platform. "Even if I'm not a person who wants to curate or do anything fancy, every employee wants to tune into the flow that matters to them, my customer flow. I'm on a product team; we have a new product initiative. Show me the flow of all the different ideas, or maybe the roadmap, or maybe some other collection of information about what we're working on."

The Platform upon which this Flow tool is deployed is a kind of IaaS service, through which information from multiple sources may be channeled, directed and filtered. It's through this streaming, like the creative hydromodification of a river, that the Flow company generates the power for its business model.

The output from this filtering tool may be directed to users like a cultivated stream. Here, using an example set of flow production rules on Alterman's own account, is a stream of categorized information reconstituted into a simple Web page, in a format that may be published to an individual user. Alternately, the Platform may produce an RSS stream or simple HTML that may be laid out with a CSS stylesheet.

The tools Flow's customers use are like this one, though modifiable by developers using the Platform's Sandbox, depicted above. Here, developers may craft specific queries of the source data and try them out live in a safe, sterile environment.

But the cultivation process need not look so much like software development. Alterman took us through the process of making a new flow by way of his customized tool (which, by the way, is one he uses personally every day). It's a wizard-like process that guides the user through the selection of known sources (including a personal favorite of mine). Later, if the tool detects the presence of a real database instead of just a Web page - a resource that contains records, usually as indicated by the presence of a form - you can select which fields are to be filtered and added to the flow.

The final step in the process is to set policies for authenticated users. It's this part that determines whether the publication of the flow is public or private, and if it's the latter, who has control over the flow's content. "As I create these different streams, I'll provision only the right people to see them," Alterman explains. He perceives the role of flow controller - of curator of the data that flows to and through an organization - as attaining executive status.

"I'd call that person the 'Chief Content Officer,'" he says. "We believe that's the next most important position in the enterprise."