Are You Big Enough for Big Data?

by | Nov 1, 2012

image with machinery gears

Gartner has recently put forth a formal definition for Big Data: “Big Data are high-volume, high-velocity, and/or high-variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization.” For those of you new to Big Data problems, the definition includes three critical “V”’s often referred to as the “3 V’s of Big Data”: volume, velocity and variety. And the fourth critical piece of the definition is “…require[s] new forms of processing…” – i.e., a novel approach to the management of data.

Here’s why that’s critical. Vendors, companies and individuals love to throw out the dictum that they have handled big data problems before big data was big. But it just ain’t so! Yes, 10 years ago, a gigabyte of data was a lot for a system to handle efficiently. Kudos for those who successfully manipulated data sets of that size. And in 10 years from now, a petabyte may not seem like a lot of data, so the future you may question why you struggled with “only a petabyte”. But just because you have “volume” doesn’t mean that you know how to work with big data problems.

The reality is that these big data problems are of a class that are just plain difficult or impossible to solve with traditional approaches. (For those of you that need specifics – I’m talking about relational data models!) Big data doesn’t mean that you must use NoSQL or other technologies, but it does mean that you need to approach the problem with a new approach, typically by implementing a highly parallel processing algorithm such as Mapreduce (though, again, it doesn’t have to be Mapreduce).

So if you are talking to a vendor who is touting their big data experience, and they say they’ve been solving these types of problems for years and years – be wary. They are giving you a “marketing answer”. Dig in. Understand what they are really doing, and understand if their solution really meets your needs. They may certainly be a great choice for you but, equally, they may not.

READ MORE

The Magic of Mortals

The Magic of Mortals

Daily we wake up to new developments in automation, Artificial Intelligence (AI), and Machine Learning (ML). Across sectors and industries, automated solutions prove highly successful in surpassing the capacity of the human brain for certain tasks, improving...

read more
Leveling Up: How to Hone Your Skills at Home

Leveling Up: How to Hone Your Skills at Home

Leaders have been trying to crack the code on talent development for years. Recent studies have shown, however, that strength-focused leadership [read: intentionally elevating the qualities that already come naturally to us] is the clear winner for developing talent...

read more
Fake Case Study: Jack of all trades vs. Master of One

Fake Case Study: Jack of all trades vs. Master of One

  Listen to any earnings call or executive presentation and you will likely hear the terms “top line” and “bottom line.” These are words used to describe a business’s performance. According to Investopedia, the words are defined as follows: Top line refers to the...

read more
Your Personality Is Showing

Your Personality Is Showing

There I was, minding my own business one evening, digging into my organization's SEO performance (as one does), when I came across something interesting. Search terms related to "MBTI" — or the Myers-Briggs Type Indicator, developed by Katherine Cook Briggs and Isabel...

read more
Lessons From a Change Manager Who Hates Change

Lessons From a Change Manager Who Hates Change

Hello. My name is Monique, and I’m a change manager who hates change.   After years of receiving “consulting therapy” from various mentors, I am now able to say these words out loud and proudly. But for a long time, it felt more like an admission of guilt. I mean, who...

read more
Creativity as a Cure

Creativity as a Cure

The topic of creative solutioning has been front and center these days as we talk more and more about organizational adaptability in the face of dynamic and uncertain times. For example, I recently read about a project that got me thinking about specific priorities...

read more
Thought Ensemble, a Pariveda Company — Why Now?

Thought Ensemble, a Pariveda Company — Why Now?

Big news over here as we close out the year - we have been acquired by Pariveda, a 750-person consulting firm in 12 markets across North America! We are now “Thought Ensemble, a Pariveda Company” and I’ll be serving as the Managing Vice President continuing to lead...

read more
Thought Ensemble Joins Pariveda Solutions!

Thought Ensemble Joins Pariveda Solutions!

Dallas, December 9, 2021 /PRNewswire/ -- Pariveda, a leader specializing in solving complex technology and business problems, announces the acquisition of Thought Ensemble. With the addition of Thought Ensemble, Pariveda now provides holistic business strategy,...

read more
Thoughts on Colorado’s Equal Pay for Equal Work Act

Thoughts on Colorado’s Equal Pay for Equal Work Act

It was about a year ago that we first started hearing about Colorado’s Equal Pay for Equal Work Act (SB19-085) and I knew it was going to be national news. We’d just gotten past the “Rocky Mountain High” jokes, and our lovely state was trying to break new ground...

read more