For the last few years people and industries have been thinking and talking about the Big Data endlessly. The next round is all about applying the machine learning to the Big Data. But, in reality, the era of Big Data is coming to an end and many of us must have seen media reports regarding the pullbacks in the Big Data industry.
Table of Contents
This is where the Big Data ends and the actual application of technology begins. This means there will be lesser “let’s roll out and see what happens” projects which will compel decision-makers to take a more rational approach starting with the business problem first.
Analyzing the common occurring problems, finding patterns and creating packaged solutions to those problems will be the next big step and this type of implementation can be seen in the finance industry where fraud detection packages are wrapped up inside and are good to go.
These fraud detection packages are not new but distributing them with Hadoop or cloud technology is something very new and fresh. Due to fraud, the finance industry was losing so much money but now they are fighting back and Hadoop and Spark are the weapons in their armoury.
Custom-built solutions are not enough and fraud detection for credit cards isn’t that different than for invoicing, insurance and any other common business application. The next step is not to write very specialised apps for the specific business but rather to identify the big data patterns for solving the common problems which exist across the lines of business.
Custom solutions where everyone is solving a similar problem in different ways will stay in the market but the future is in finding common grounds, developing patterns and spreading those solutions across various lines of business. In the end, it needs to be customized a little bit to be used in specific business lines. This means many of the repetitive tasks need not be done again and again.
Yes, we have already seen it in the market before. Almost a decade ago, accounting software was a hot topic and while one can still find specialized accounting software in the market, most companies are using a pre-packaged solution which has been customized to their business-specific needs. It is very rare for a skilled software developer to write software for a specific kind of business and let alone for a particular company. Everybody buys them right off the shelf because they are general purpose.
In the world of Big Data, the next big step is going to be “data-driven” and using “machine learning” with the help of integrated software. It might be Big Data-driven in the end but Big Data will not be the topic of conversation. By then it would become like Ethernet cards: “Taken for granted”.
