Connected Data Platforms & Internet of Anything

Existing data platforms need to evolve to deal with the tsunami of data-in-motion being generated by the Internet of Anything (IoAT).We need a data strategy that is more enterprise-ready and future proof. Hortonwords Connected Data Strategy provides an overview of how enterprises should utilize this (Connected Data Platforms) strategy that handles both data-in-motion and data-at-rest.

–> According to McKinsey, if you’re a Fortune 1000 company and you digitize an asset, you can grow its value by 5x. If you sell through digital channels you can increase productivity by up to 15%. If you digitize your business strategy you can realize a 3x increase in profit margins. This would be impossible without Apache Hadoop and the data lake. processing of large data sets to a fully-fledged data platform with the necessary services for the enterprise from Security to Operational Management and more. McKinsey Report

–> Today we are moving even beyond the wildest predictions of the growth of data being created, stored, retrieved and analyzed only a few years ago. With 5G the smartphone can potentially deliver a staggering 1Tbps of data! TrustedReviews

–> In just two years from now there will be more devices than people on this planet. 35% of Americans Now Own at Least One Smart Device other than a Phone like thermostats, refrigerators, watches all delivering signals.That’s up to 6.4B of connected things, or 21B devices, by 2020. Truste

–> Data types are constantly changing too. Obviously, it’s not just rows and columns any more;image data, all kinds of streaming data, geospatial coordinates, and time series. The one and a half billion monthly active Facebook users has represents more than 140 billion friend connections to be made, 265 billion photos uploaded, 62 million songs played 22B times. GE.com

–> Gartner tells us that 32% of businesses who undertook a digital business transformation say their businesses are now digital businesses. Gartner.com

According to Forbes: - 59% of businesses consider data and analytics to be “vital” to the running of their organizations, with a further 29% deeming it “very important”. - 69% say there is business case for investing in exploring new ways to add value through data projects. - 83% say it is making existing services and products more profitable. - 60% of businesses claim their data is generating revenue within their organizations. - But 48% feel that their organizations have, in the past, failed to take advantage of opportunities to capitalize on their data.

Connected Platforms Need to Evolve

–> Connected Data Platforms need to handle distributed data across departments, servers and location. This data can be data-at-rest or data-in-motion. It needs to be done securely and cost effectively and take into account bandwidth and connectivity issues.

–> Connected Data Platforms need to go beyond Apache Hadoop and need enhanced data routing, transformation, and system mediation logic that is starting to emerge from projects like Apache NiFi which supports real time ingestion, pattern detection, routing, and analysis.

–> New capabilities and tools like Apache Zeppelin are needed to make the Connected Data Platforms more accessible

–> Connected Data Platforms need to be enterprise ready, enterprise scale, provide predictable performance, support data encryption, security, data governance, HA, DR, operations and debugging.

Six Requirements of Connected Data Platforms (as per Hortonworks research)

1) Secure Data Ingestion: Can you easily and securely ingest data from anywhere across the Internet of Anything and siphon off data quickly, detecting interesting patterns?

2) Actionable Intelligence: Can your platform provide real-time actionable intelligence based on both data-in-motion and data-at-rest?

3) Distributed Connectivity: Can you connect, relate, or put in context all forms of data to provide 360 degrees of context that allows you to predict and personalize data for your modern data applications across all your channels?

4) Mere Mortal Friendly: Does your platform provide data science tooling that make this easier for data scientists and professionals to find interesting patterns and gain actionable intelligence?

5) Future Proof: Is your platform built on 100% open source technology allowing you to capitalize innovation everywhere economy?

6) Enterprise Ready: Do you have enterprise-ready capabilities for security, high availability, disaster recovery and so on? Is the data secure? Is it access controlled? Are all compliance regulations taken care of? Is activity tracked with an audit trail? Is data controlled and managed through a lifecycle? Is your platform supported by experts?

References

Apache Nifi Apache Zeppelin Hortonworks Connected Data Strategy Hortonworks Connected Data Whitepaper