Since selling Urban Mapping last year, I’ve spent more time thinking about how data is can be and is used for alternative purposes. To me, the idea of packaging up organisational data exhaust and redirecting it to non-adjacent markets is an opportunity hidden in plain sight. I’ve been whining that one person’s metadata is another person’s data for years and it’s fun watching it play out. As with most tech trends, hype generally precedes utility, so this will doubtful result in a massive payday anytime soon.
A few current examples of how organisations and industries some are making use of log/event data:
- Farming – Co-ops for agriculture have been around in the US since 1800 as a means of realising efficiencies through marketing, bargaining/labor and credit. As Big Ag continues muscle in on independent farmers and build out the new, vertically integrated value chain, from seed to SQL, newer cooperatives seek to push back against industry by pooling data about yield/performance and meteorological conditions for member benefit. With substantial farming subsidies in the US, is it crazy to think that crops could become a loss leader, subsidising data about crops.
- High Frequency Trading – Brought to attention by Michael Lewis, the simple premise is that retail trades give pricing information to institutional investors. When armed with such metadata, HFT firms front their trades, often at the peril of retail investors. The teaching point here is that without aggregate trading behavior, HFT shops wouldn’t have a strategy to generate billions in annual profit.
- Macroeconomic Forecasting – Traditional measures of economic health, such as unemployment, CPI and consumer confidence were born during the mechanical age: telephone panels, mail-in surveys and on-site visits seemed reasonable in a time when data was sparse and compute was expensive. So no surprise that ADP, payroll processor to 20% of the non-governmental workforce, began to offer its own unemployment survey in 2006. If you can’t get that which you seek, use what you can get!
A wave of “proxy indicators” have emerged to fill an information vacuum (most popular among them is the six year old UBS use of satellite imagery to count parked cars at Wal-Mart). Other more current examples include the Fleetmatics Fleetbeat report to better model retail sales. Proxy indicators is post (or book!)-worthy on its own, so stay tuned for more on this topic.
- Public Records – There is a wealth of value in public records, but a marginal business in simply ingesting/indexing to some kind of machine-readable format. The real value is in performing analysis and generating insight. A few companies are way ahead of the curve in doing this: Lex Machina ingests court filings and allows users to understand relationships between judges, circuits, intellectual property claims, attorneys and more. In the context of litigation, how is an action brought by Plaintiff A on Judge B’s docket likely to fare? Moneyball it!Speaking of moneyball, what about applying similar smarts to the very public voting record or legislation? Washington (and legislatures across the globe) have been cloaked in relationships and whispers and party leaders have had a monopoly on who supports what. FiscalNote has applied smarts to predict voting patterns and a lot more in the arcane legislative and rule-making factories. If you have an interest in public policy, the crystal ball of Washington just became a lot less mystical and more practical.
Stay tuned for more thoughts (and possibly insight!), and please let me know what you think.
Post-employment 