Analytics, Machine Learning and AI has not been restricted to large enterprises anymore; it has been widely adopted to varying degree and success by the small and medium enterprises as well.
The rise in the demand for analytics and advanced analytics has coincided with the rise of SaaS across the spectrum.
SaaS market grew at 17.8% between 2015 – 2020, and BI market at 8.4% during the same time period; furthermore, these markets are projected to grow at 28.3% and 12.2% respectively through to 2025. AI has undoubtedly stolen the march in the recent past – the AI market grew at a whooping 78.5% between 2015 to 2020 and is projected to grow at 47.8% through to 2025.
Organizations – small and large – have embraced the SaaS delivery model and are increasingly relying on SaaS applications to run their businesses. Typically, a firm of 250 employees uses 100 SaaS applications and about 34% firms rely solely on small applications to run their business.
Combination of high rate of SaaS adoption, proliferation of SaaS apps, demand for BI & Analytics, and rapid ascent of AI has resulted in unique data-hungry use-cases. Following are some of the representative BI & AI use-cases:
- Customer 360
- Churn modeling and analysis
- Demand forecasting
- Lead scoring
- Video analytics
- Marketing performance analytics
These and many more complex use-cases/projects lead by the CTO and CIO offices are looking to unlock value from their data. Such use-cases need integration capabilities which enables enterprises to deliver complex projects in an iterative manner and with a shorter time to market. In order to enable such complex use-cases, sophisticated data integration capabilities must be developed. Following are some of the key requirements driving data integration efforts in the age of Analytics and ML/AI:
- Ability to connect to varied data sources: On-premise and SaaS apps, databases, spreadsheets, data lakes et al.
- Integration of real-time/near real-time data, pushed data, and time-series datasets
- Integration with the 3rd party data e.g. social network data for personalized messaging
- Run integrations at scale
- Rapid integration with variety of data sources to enable iterative development and experimentation of the downstream services/apps
While the need for data integration capabilities have been around for decades, it needs to evolve and solve complex data engineering problems to provide the competitive edge to the business. Lack of robust data integration capability may undo investments business makes in developing sophisticated BI and machine learning initiatives.
From the engineering standpoint, at a high level, a data integration engine/platform would need to solve following problems to deliver on the business requirements:
- Data validation
- Data standardization
- Data mastering: Entity matching, golden records, universal ids
- Low-code, configurable data source integration and transformation
- Unified batch, event and stream processing
- Data pipeline monitoring
Enterprises which are able to effectively solve these problems will have a significant advantage over its competition. Choosing the right technologies, tools and the team is key to successfully tackling the data integration challenge and enabling various business functions to achieve data driven business outcomes.