On Thursday and Friday of last week, I attended the Big Boulder data conference, which brings together vendor, academics, analysts and practitioners of social data. The purposes were many: discuss emerging trends, acknowledge the issues and challenges around privacy and security, and make introductions to encourage discussion of how we all envisage social data technology and by extension social data maturing.
I spent two days fastened in on how vendors believed social data could be used and how companies and researchers were ultimately using it. At times, there was a wide gulf and not only because the rate at which technology is evolving is rapid but because we, as an industry, recognize the importance of this data and don’t want to compromise the trust our customers and clients have for us.
The people at GNIP/Twitter are well aware of this and have spearheaded the Big Boulder Initiative, a task force created to address critical issues around stewardship, enablement, availability and value. If you’re interested, you can learn more here.
Over the two-day conference, there were over 45 sessions with topics ranging from Sina Weibo to the challenges of analyzing unstructured data to user-generated content vs. brand-created content. Despite the wide scope of topics discussed, there was an underlying recognition that we were all in this together, that we have an obligation to manage the growth of social data in a responsible and secure manner and that we still had some growing up to do.
I could probably write several pages of themes and insights that I noted during the two days but here are three I thought we’re particularly interesting.
Visualize Whirled Peas
This year there was a lot of discussion around visualization and the impact of Tumblr and Pinterest, respectively. One of the panelist believed that visual channels were happy because people like to engage with images. I’m not sure I entirely buy that and other members of the panel were quick to argue to the contrary. However, watching the world wake up and go to sleep with Twitter was very compelling and did make me smile (if not happy).
Some members of the panels wanted customers to more fully recognize the value in sharing their location via a social platform. I can see the benefits to users of the data; it was amazing to see the outline of common maps reveal themselves not through traditional boundaries but rather through social activities—outlines of cities, airports, etc. emerged as people Tweeted. The panelists didn’t seem to share some of the anxieties I had about sharing my whereabouts in real-time. Issues of safety and cyber-bullying can and should influence what people share online. However, I liked the idea of using imagery to guide discovery and finding someone on say, something like Tumblr, with a similar aesthetic to encourage that connection.
We Do Have Some Standards Around Here, You Know
This was the first year, where I heard the admission that social does not have the same standard of measurement as say TV advertising, print ads, etc.. This wasn’t the familiar beat of the ROI drum but rather a recognition that we need to, as an industry, better define the value of social. To date, we don’t have a verifiably mature model that clearly defines what comprises that value. We don’t have a clear idea of when engagement matters most and how to attribute that activity. But honest conversations are beginning and everyone seems to recognize the importance to sales, marketing, HR, etc. to answer these questions.
Millennials vs. Digital Behavior—Which One Truly Matters
I have to admit this topic really intrigued me and I was excited to learn the digital characteristics of this generation. I don’t know if the resulting information was meant to make us all feel better (read: younger) but some of the panelists felt that generations should be segmented along the lines of digital behavior over age. Susan Etlingersuggested that we’ve been using demographic behavior as a proxy for categorizing customers and it’s losing its value. It’s certainly true that using the blunt instrument of age to determine a person’s online social persona may omit a lot of detail but with each succeeding generation the use and proliferation of online tools can’t be entirely overlooked. Susan certainly wasn’t minimizing the influence of social technology broadly across generations but that we should perhaps adjust our lens to include more than just demographics to segment an audience.
In the two days, I met some great people, discovered that everyone is facing very similar changes and that it’s never been more exciting to be involved with Social Data. Learn more about the Boulder Initiative here and the Big Boulder conference here.
Sooner or later we all feel like throwing up our hands and cursing the complexity of modern life. But while technology may seem the chief culprit in making things unmanageable, it is also the ultimate solution to complexity.
In the Internet of Everything (IoE) era, it is particularly important for business leaders to understand the power of technology to simplify our lives and support informed decision making. And this was a core theme at Sapphire Now 2014, an event in Orlando, Fla., that I was privileged to attend last week.
By using network technology to integrate people, process, data, and things, IoE counters complexity in unprecedented ways. In a city, this can involve something as simple as cutting the time it takes to find a (connected) parking space. Or IoE technologies can scale up to reroute traffic lights; for example, to head-off highway backups before, during, and after a large event.
In a brick-and-mortar retail setting (a key area of discussion at Sapphire Now), IoE can alleviate the complexity of managing customers, staffing, and products. With data from multiple sources comes heightened, real-time awareness, empowering managers to react faster than ever. For example, they can then stock shelves and reorganize staff in response to constantly changing levels of demand. With predictive analytics they can even respond before a customer rush begins.
The idea of hyper-aware, real-time decision-making resonated during a Sapphire Now panel discussion titled Thrive in the Digital Networks of the New Economy. I was honored to share the panel with such luminaries as Erik Brynjolfsson of MIT; Michael Chui of McKinsey Global Institute; and Jai Shekhawat, Deepak Krishnamurthy, and Vivek Bapat of SAP. And there was much discussion on the impact of bad decisions on failed organizations. Which is why we all take such an interest in technology that enables good ones.
In a typical week, I spend about 70 percent of my waking hours on work-related matters. Another 50 percent is devoted to my family. Which leaves 20 percent for taking care of the household, and ….
Yes, that adds up to more that 100 percent. But there simply aren’t enough hours in a day for all that needs to be done — not to mention protecting that crucial time with loved ones.
So, thank you, multitasking! I can’t be the only one who has held a child while writing emails, taken conference calls from the supermarket, or had several online meetings running simultaneously.
All of this occurred to me as I struggled to find time for this blog. Writing forces me to shut off everything around me and reflect on the things that really matter — in a world that is rapidly changing, increasingly complex, and in which technology can sometimes seem a mixed blessing. When I do finally carve out an opportunity to write, it is precious time, which I cherish.
But writing is hard. Trust me, I’ve thought about creating a blog for years, and my past is riddled with failed attempts to start. Each time, I hesitated for too long, wondering whether people would really want to hear what I have to say. Like many writers, I have wondered if my compositions were too long, too short, too personal, too corporate, too banal, too deep ….
But as much as I appreciate your attention, dear reader, this time around I realize that I am writing the blog for me, the writer. Like many of us, I navigate a harried, high-pressure life. And this blog is my time, my space, to do something creative and expressive.
Historical data is now an essential tool for businesses as they struggle to meet increasingly stringent regulatory requirements, manage risk and perform predictive analytics that help improve business outcomes. While recent data is readily accessible in operational systems and some summarized historical data available in the data warehouse, the traditional practice of archiving older, detail-level data on tape makes analysis of that data challenging, if not impossible.
Active Archiving Uses Hadoop Instead of Tape
What if the historical data on tape was loaded into a similar low cost, yet accessible, storage option, such as Hadoop? And then data virtualization applied to access and combine this data along with the operational and data warehouse data, in essence intelligently partitioning data access across hot, warm and cold storage options. Would it work?
Yes it would! And in fact does every day at one of our largest global banking customers. Here’s how:
Adding Historical Data Reduces Risk
The bank uses complex analytics to measure risk exposure in their fixed income trading business by industry, region, credit rating and other parameters. To reduce risk, while making more profitable credit and bond derivative trading decisions, the bank wanted to identify risk trends using five years of fixed income market data rather than the one month (400 million records) they currently stored on line. This longer time frame would allow them to better evaluate trends, and use that information to build a solid foundation for smarter, lower-risk trading decisions.
As a first step, the bank installed Hadoop and loaded five years of historical data that had previously been archived using tape. Next they installed Cisco Data Virtualization to integrate the data sets, providing a common SQL access approach that made it easy for the analysts to integrate the data. Third the analysts extended their risk management analytics to cover five years. Up and running in just a few months, the bank was able to use this long term data to better manage fixed income trading risk.