论文部分内容阅读
本文针对公安交通管理信息系统开发建设中遇到数据处理、融合、迁移的需求,从跨网段数据批量处理、多源数据融合、结构化数据向分布式大数据平台迁移三个应用场景着手,分别介绍Spring Batch批处理框架、Kettle Etl工具及Sqoop数据迁移工具,并结合具体场景介绍其优势。旨在为同样面临数据批处理、数据迁移融合需求的应用提供一种解决方式。
In this paper, in order to meet the data processing, integration and migration requirements in the development and construction of public security traffic management information system, this paper starts with three scenarios of batch processing of cross-network data, multi-source data fusion and structured data migration to distributed big data platform. Introducing the Spring Batch batch framework, the Kettle Etl tool, and the Sqoop data migration tool, respectively, and describes their advantages based on specific scenarios. It is designed to provide a solution for applications that also face the convergence of data batching and data migration requirements.