Are You Big Enough for Big Data?

by

image with machinery gears

Gartner has recently put forth a formal definition for Big Data: “Big Data are high-volume, high-velocity, and/or high-variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization.” For those of you new to Big Data problems, the definition includes three critical “V”’s often referred to as the “3 V’s of Big Data”: volume, velocity and variety. And the fourth critical piece of the definition is “…require[s] new forms of processing…” – i.e., a novel approach to the management of data.

Here’s why that’s critical. Vendors, companies and individuals love to throw out the dictum that they have handled big data problems before big data was big. But it just ain’t so! Yes, 10 years ago, a gigabyte of data was a lot for a system to handle efficiently. Kudos for those who successfully manipulated data sets of that size. And in 10 years from now, a petabyte may not seem like a lot of data, so the future you may question why you struggled with “only a petabyte”. But just because you have “volume” doesn’t mean that you know how to work with big data problems.

The reality is that these big data problems are of a class that are just plain difficult or impossible to solve with traditional approaches. (For those of you that need specifics – I’m talking about relational data models!) Big data doesn’t mean that you must use NoSQL or other technologies, but it does mean that you need to approach the problem with a new approach, typically by implementing a highly parallel processing algorithm such as Mapreduce (though, again, it doesn’t have to be Mapreduce).

So if you are talking to a vendor who is touting their big data experience, and they say they’ve been solving these types of problems for years and years – be wary. They are giving you a “marketing answer”. Dig in. Understand what they are really doing, and understand if their solution really meets your needs. They may certainly be a great choice for you but, equally, they may not.

READ MORE

What’s in a Name?

What’s in a Name?

In today’s impression-obsessed, content-rich, never-ending-narrative-building climate, there is a surplus of articles that exacerbate the importance of developing a strong competitive BRAND for your organization. But what about your company’s name? What about the...

read more
What Is Ted Lasso Teaching Us?

What Is Ted Lasso Teaching Us?

Featuring Dave Allston For me, it started about six months ago… “Do you watch Ted Lasso?” This question pops up on Zoom calls. References often show up deep within the heart of important meetings as a form of verbal punctuation and most certainly are a mainstay of...

read more
Calling Everyone Back to the DEI Table

Calling Everyone Back to the DEI Table

It is no secret that 2020 put a glaring spotlight on the importance of diversity, equity, and inclusion in our everyday lives. As our communities reckoned with difficult questions and conversations around racial equity and justice, those topics found their way into...

read more
Did We Just Fall in Love With Hybrid Meetings?

Did We Just Fall in Love With Hybrid Meetings?

Last week, we held a two-day training session for our extended leadership team. We’d been planning it for months — the meeting room, the dinner activity, the team building exercises … the works! This was, as I’m sure you’re picking up on, more than just a meeting,...

read more
“The Great Resignation” Has a Really Good Publicist

“The Great Resignation” Has a Really Good Publicist

Mass resignations are a big deal right now. Yes. Is it surprising? Absolutely not. People leaving for greener pastures is not a new concept. How many of you — given a choice between two situations — would choose the one that LEAST benefited you? If a better option...

read more
The Human Quotient: Heuristics and Bias in Analytics

The Human Quotient: Heuristics and Bias in Analytics

As individuals, we all have a unique set of heuristics and biases that show up in our daily lives, whether we notice them or not. This means that for a lot of the decisions we make, whether personal or professional, there is an underlying, subconscious process that...

read more
Why Keeping It Real Is a Winning Strategy

Why Keeping It Real Is a Winning Strategy

If I had to sum up my pandemic year, I might say it was the year I got to spend each day of an entire summer with my daughter. It was also the year that I significantly altered my career path by taking the brilliant advice of my coach: to lead with my values (keep it...

read more