Taking a social look at the concept of 'big data'

The biggest surprise about the web is that it turned out to be social and not a giant library, shoe catalog, or business-to-business interchange of supply-chain data.

We created a web where all roads lead back to us

The social exhaust of humans at work, traveling, chatting: that's forming the social side of the big data hype. It's as rich and as interesting as a 100 billion sensors, devices, and machines reporting traffic jams, or that your milk's gone off, or how many Almond Joy bars are in the vending machine hear the elevators at the Rite Aid on 146 Main Street.

Paul Kedrosky started to capture some data California was starting to publish in 2009. One stat was the junk they were collecting on the highways that had fallen off cars and trucks.

His Ladder Index derived from that data turned out to be a perfect leading indicator for the housing boom and bust, leading by about four months.

Some researchers claim they can read fluctuations in the stock market based on Twitter sentiment. It's been disputed, but it raises the question: If we actually could do that, what would happen to the markets? If the near-term future becomes knowable by algorithm, how would markets work? Is it the end of speculation, or the end of markets?

It took years for the New York City government to find out where a occasional cloying smell of maple syrup was coming from. Someone correlated the phone number call data with addresses and it turned out to be a factory making ersatz maple syrup from fenugreek.

Nowadays, geolocation can be built into mobile-first complaint websites, and it would be found in days, not years.

A researcher from IBM call cull your personality from 200 tweets.

In the near-term the notion of hiring or promoting people based on their 'fit' with company culture or the demands of a certain job will become dogma. And it will seem normal to hire not for skills and experience, but based on a pre-crime-like estimate of emotional and motivational orientation.

Gild has algorithms to read open source software and evaluate the skills of the programmers behind it. They cross-correlated with social media interactions between programmers, and can say with high confidence who're the stars and who're the wannabes. Now, they don't have to read the code. And they found that being a strong programmer is tightly correlated with being interested in a single Japanese manga site. Will we take people out of the hiring loop, altogether?

Imagine a company of a few years from now, where Watson-grade AIs are monitor all of our work activities, and then will reach in and make a suggestion to someone about an article that might help with that research, or pull three people into a meeting to resolve a project delay, or introduce two marketers and a developer working on similar but unrelated activities.

The cyborg business is coming: an admixture of social and algorithmic logics, working better than either alone.

We are undergoing a rapid loosening of business culture: the decline of the postmodern work contract of long-term loyalties has been replaced with a postnormal detente of short-term mutual expediency.

Now, the rise of the 'bots will accelerate the collapse of consensus and the '00s era of 'collaboration', and all that will be left is cooperation through networks and communication through hyperintelligent social appliances, that are listening to us and connecting dots too small -- or large -- for us to see.