A Fresh Perspective: Big Data

Here at The San Francisco Data School we find the term Big Data to be ambiguous, and evolutionary. Both adjectives are being used optimistically. As in, we think there should be more than one interpretation (ambiguous), and that those interpretations should be refined over time (evolutionary).

So much of how we interpret the world is based on our surroundings. This theory is at the core of why we choose to describe Big Data that way. As case in point:

  • An individual that's involved in the engineering efforts of storing data will likely be attuned to a logistical understanding of Big Data
  • An individual who recognizes the massive business potential with data may be best aligned to an economic understanding of Big Data
  • An individual tasked with discovering notable customer behavior with data could have an analytical understanding of Big Data
  • etc...

The bottom line is that we think of Big Data, not as a term to be defined, but as a malleable concept to be understood. And we best understand this malleability through two consistent underlying phenomenons: Social and Technical.

Big Data: The Social Phenomenon

Data has, and will continue to pervade our lives. It's used within the stories we tell, decisions we make, services we use, and more. But why?Because of the potential it holds. Reusing our examples above, we use data in:

  • Stories to help make them objective, and more persuasive
  • Decision making to help hedge against risk (amongst other things)
  • Products and services to get the best outcome (e.g. reviews on Amazon)

And because of this potential, society has built-up a deep vested interest in collecting and sharing as much data as possible, which leads us to...

Big Data: The Technical Phenomenon

Technology has unlocked the power to collect a tremendous amount of data, and as it goes – "with great power comes great responsibility". In this case, we're defining "responsibility" as the collection, storage, processing, and sharing of data (i.e. all of the technical complexities involved in working with data).

The by-product of owning up to these responsibilities has been significant advancements in computer hardware and software, mathematical practices, types of jobs, companies, and more. Thus, the concept of Big Data is embedded in many of the technological developments that have taken place alongside the growth in societal interest for data – increased demand pushes technology to its limits, which propels innovation.

Summary: The Convergence of Social and Technical

To bring this full circle we'll end with our best attempt at articulating Big Data in one sentence:Big Data is a concept sitting at the crossroads of increasing societal needs/wants for data, and all of the technical demands that accompany them.