Big Data, Little Data, and Your Decisions


There is a difference in knowing the path and walking the path.
— Morpheus, The Matrix

You do it every day. Make a decision and take actions with consequences. You change an HBA setting. VMotion VMs as part of an upgrade. Troubleshoot storage slowing application response times. You operate under the gun, with pressure to get the job done. And you do.

But what are the impacts of each step you take? Are there longer-term implications to what you’re doing? Is there risk you can’t see? Are you unknowingly creating waste that will lead to new expenses? You don’t know and you’re not to blame for not knowing. The datacenter is at once both a wildly complex beast and instrumented so heavily it throws off more metrics than a Boeing 787 Dreamliner. You know your operational data can guide you to answers that will make your decisions and actions better – but it’s not consumable with today’s operational management approaches. You have to work much too hard to walk the path you want: driving decisions using data. Educated guessing can only get you so far until you’re surprised by unforeseen waste and risk.

Putting Big Data to Work for You:
Global Insights and Dynamic Benchmarks

CloudPhysics exists to enable you to walk the data path: providing you the right data, at the right time, in the right context. At VMworld this week we’re presenting new types of analysis and simulations you’ve never seen applied to familiar problems. We’ve designed services to put data to work for you, to enable you to make better decisions, to take more effective actions, and to present your work to management in a data-driven framework intersecting with their world of finance and economic rationale.

Big data plays a big role in making this happen for you. For the past two years we have continuously gathered trillions of operational metric objects and applied data science to this unprecedented and growing dataset representing thousands of vSphere datacenters. You can see our initial results in the latest CloudPhysics release, which we’re showcasing at VMworld this week. We are now providing global insights comparing your local configuration, performance, and cost structure to cohorts found in our global dataset. If you are an architect trying to build an outline of your design, CloudPhysics can provide you the initial underpinnings to your thinking. While in operations, our global insights establish the data-driven framework for setting up your key performance metrics and indicators. It’s a new form of dynamic benchmarking by which to set goals and measure effectiveness.

And Little Data, Too: Workload Shapes

Little data, however, is just as important in driving the datacenter with data. By “little” I mean ultra-narrow in scope, the very opposite of the large aggregated queries our system also enables. In the microcosmic view of a running workload, for example, capturing every I/O emitted by an application at the microsend level, is the key to unlocking the mysteries in some of the most frequent and painful problems in a virtualized infrastructure. My co-founder and CTO, Irfan Ahmad, who originally authored storage DRS and many of the storage metrics in vSphere, has addressed these issues with a fundamental innovation: a new view into workloads, producing the “workload shape” characterizing storage access patterns in a manner never seen before.

The interaction of a running workload with storage is multi-dimensional and dynamic – and confounded by the hypervisor that sits between them. Having an understanding of the storage demand “shape” reveals exactly where and how contention exists. It solves the workload puzzle at the core of classic storage management problems – performance troubleshooting and capacity planning – as well as new storage decisions you face, such as deciding whether you should procure and introduce SSD/flash into your infrastructure. Little data in the form of workload shapes has big impacts on your most pressing storage-related decisions. Irfan will be providing a technical preview of this industry-changing technology this week.

Check Us Out This Week

That’s just a taste of what we’ve been up to at CloudPhysics, all of it focused on one thing: helping you make better decisions as admins and architects. Putting data to work for you – rather than making you work for it – sets you on the path to increasing the levels of effectiveness, safety, and predictability beyond your reach today. That data-driven world exists right now. You can find it this week at VMworld in our booth #640.

path to data-driven insights