Making big data projects work
Big data is one of the technology trends where the benefits may outstrip the hype. Almost every organisation is capable of achieving significant benefits from big data, and most organisations already collect significant amounts of useful data. To turn that data into insights, organisations need an effective analytics approach.
Companies that are not making decisions based on effective analysis of big data could be missing the opportunity to outperform competitors. Yet, while most organisations are convinced of the value of big data, some find it difficult to turn theory into practice.
Specifically, many organisations become distracted by the technology around big data and forget that data collection and analytics are only useful if they can be used to solve real business challenges. This includes finding ways to reduce costs, improve processes, services and products, target new markets and increase customer satisfaction and engagement. Keeping the business goals in mind is essential to get maximum value from big data.
Technically speaking, there are a few challenges when it comes to collecting and analysing big data. For example, big data is a term that encompasses many different kinds of data. Structured data, which resides in a database, is readily searchable. It may include sensor data as part of the Internet of Things (IoT), as well as data from customer relationship management (CRM) systems and the like. Semi-structured data is not as easily searchable because it is not stored in a database. It could be in the form of emails, documents, images, video or audio.
To build a truly comprehensive view of the problem an organisation is trying to address, they typically need to bring structured and semi-structured data together, then analyse it to gain insights. The first challenge the organisation encounters, then, is how to combine structured and semi-structured data.
The problem is compounded by the fact that there is a broad range of big data technology options and vendors, which makes it difficult to find the right solution. Like any business process system, a big data solution needs to be as low cost and as simple as possible. It needs to be stable, highly integrated and scalable enough to move the entire organisation towards true data-centricity over time.
Data-centricity refers to an organisation that makes big data and analytics available to all parts of the organisation. Everyone in a data-centric organisation can access data streams and user toolsets to discover valuable insights, make better decisions and solve real business problems.
Organisations looking to make the most of big data should consider three key components:
- data sources such as operational systems, machine logs and sensors, and web and social media
- data platforms that enable the capture and management of data, and
- big data analytics tools and apps that let executives, analysts and managers access customer insights, model scenarios and make decisions.
These components can then be used to enable data analysts to examine current data streams and repositories, as well as solve specific business problems and make longer-term market predictions.
“To build a truly comprehensive view of the problem an organisation is trying to address, they typically need to bring structured and semi-structured data together, then analyse it to gain insights.
Experience has shown that successful and effective big data environments demonstrate three key traits:
- They are flexible and low-cost, able to scale for future needs, yet still simple and easy to use.
- They are stable, which is critical because data volumes are, by definition, massive. Users need to easily access and interact with the data, which is impossible if the platform is unstable. Therefore, the infrastructure needs to be robust.
Once an organisation has the right big data environment in place, it can begin to develop projects that leverage big data analytics to deliver business benefits.
There are three steps to creating a big data project that will fit your business needs:
Choose what’s right for your organisation
Organisations should not necessarily look to copy the technology decisions made by other companies. Those companies may have different goals, restrictions, and environments. It is far more important to choose the right technology based on its merits and potential value.
Technology enables an end goal. It shouldn’t be an end in itself. As long as the technology can meet business requirements, it doesn’t really matter what that technology looks like. Organisations should define the business goals of a big data strategy before choosing the technology to drive it. This will result in a better fit to meet long-term business goals.
Hire the right people
Many of the job descriptions for big data roles look similar, with specific technology expertise and industry experience cropping up again and again. While some of the attributes of potential data scientist candidates will naturally be similar, it is important for organisations to look at the bigger picture.
If companies only look for one standard set of qualifications, this can restrict the broader skillset available to them. Companies need people with different backgrounds and expertise, and good data scientists will pick up new technology and systems easily. By contrast, hiring the wrong people will severely curtail an organisation’s ability to derive significant value from a big data project.
Ask new questions
It’s not uncommon for companies to have similar end goals from successive big data projects. Higher revenue, deeper insight and better customer engagement are recurring themes, for example. As such, companies may have a tendency to ask their data scientists to replicate what has been done previously to achieve these goals.
This is unlikely to yield much additional value. Organisations would be better served to devise new questions to ask. Getting more specific is one way to drill down into data, gaining deeper insights that can drive action.
The author, Ross Farrelly, is chief data scientist at Teradata ANZ, a provider of end-to-end solutions and services for data warehousing, big data and analytics.