2022/11/27

D4P2 - Data-Driven Design & Development of Product or Platform

How to build a data architecture to drive innovation--today and tomorrow | McKinsey

• Design and deploy new data technologies alongside legacy infrastructure to drive market-driven innovations such as personalized offers, real-time alerts, and predictive maintenance.
• from data lakes to customer analytics platforms to stream processing—have increased the complexity of data architectures enormously, often significantly hampering an organization’s ongoing ability to deliver new capabilities, maintain existing infrastructures, and ensure the integrity of artificial intelligence (AI) models.
• need a new approach to defining, implementing, and integrating their data stacks, leveraging both cloud (beyond infrastructure as a service) and new concepts and components.

Data Architecture.png

six shifts.png

1. From on-premise to cloud-based data platform
   从内部部署到基于云的数据平台
   • Serverless data platform e.g. Amazon S3 and Google BigQuery
   • Containerized data solutions
2. From batch to real-time data processing
   从批处理到实时数据处理
   • Messaging platform e.g. REBAR Messaging
   • Streaming processing and analytics solutions e.g. Data Fabric
   • Alerting platforms e.g. DataDog
3. From pre-integrated commercial solutions to modular, best-of-breed platforms
   从预集成的商业解决方案到模块化、一流的平台
   • Data pipeline and API-based interfaces
   • Analytics workbenches e.g. Amazon Sagemaker and Kubeflow
4. From point-to-point to decoupled data access
   从点到点到解耦数据访问
   • An API management platform e.g. API Gateway and Open API
   • A data platform to "buffer" transactions outside of core systems e.g. Data Lake and Data Mesh
5. From an enterprise warehouse to domain-based architecture
   从企业仓库到基于领域的体系结构
   • Data infrastructure as a platform
   • Data visualization techniques
   • Data cataloging tools
6. From rigid data models to flexible, extensible data schemas
   从刚性数据模型到灵活、可扩展的数据模式
   • Data vault 2.0 techniques
   • Graph databases
   • Technology services
   • JavaScript Object Notion (JSON)

How to get started

• Apply a test-and-learn mindset
• Establish data "tribes
• Invest in DataOps
• Create a data culture"