Why is small data important for your business?

“New ideas are sometimes found in the most granular details of a problem where few others bother to look.” — Nate Silver

Most managers argue that thinking big gives them an edge over their competitors. And probably its true. However, they frequently overlook granular details about their organizations. As a result, when they have to make the decision to reorient their business towards data, they almost automatically think of ‘Big Data.’

It’s also true that we have been hammered with the term “Big Data” for quite some time. But we must be aware that what we mostly have in organizations, (whether they are big or small, global or local) is an enormous number of small data records. Why are these seemingly trivial pieces of data so important? because the most critical information in the organization is encoded here. To use this information we have to options:

1. Collect, centralize, and pre-process this data in order to make it valid for analysis.
2. Analyze this data at the source to get relevant insights and use them locally at the same time we share it across the organization.

The first point is at the core of the majority of corporate data strategies.

Advantages of point 1:

1. All data is centralized and easily available for who needs it at any given moment.

Disadvantages of point 1:

1. What I have written in advantages of point 1 is not true: not all information is collected and sent to the central processing engine; and not everybody that needs this information has access to it.

This is the paradox. Despite the considerable work that has gone into building a cloud-based data infrastructure organizations face with these limitations.

Solution:

Small data. What does it means? To empower people to analyze their small data records using ‘Small data’ tools like AutoML tools, synthetic data, viz. tools and become ‘Organization Data Scientists’ (as the equivalent to ‘Citizen Data Scientists’).

Advantages:

• It’s much cheaper.
• No need to have ‘Data scientist’ advanced skills.
• There arte not data leaks because we think this data is not important.
• Accesible in the needed context by employees that are going to apply this insights.
• Also shareable trough the organization.

Conclusion:

Small data is at the core of any organization. As Nate Silver pointed out, new ideas are sometimes found in the most granular details of a problem where few others bother to look. So, think small (data) can be also a source of competitive advantage.