作者：伯箫，阿里云高级开发工程师。现在在阿里云表格存储团队，负责管控系统的开发，对NOSQL类数据库系统有一些了解。前言从EMR-3.21.0 版本开始将提供Spark Streaming SQL的预览版功能，支持使用SQL来开发流式分析作业。结果数据可以实时写入Tablestore。本文以LogHub为数据源，收集ECS上的日志数据，通过Spark Streaming SQL进行聚合后，.....

文章 2022-02-17 来自：开发者社区

海量监控日志基于EMR Spark Streaming SQL进行实时聚合

前言从EMR-3.21.0 版本开始将提供Spark Streaming SQL的预览版功能，支持使用SQL来开发流式分析作业。结果数据可以实时写入Tablestore。本文以LogHub为数据源，收集ECS上的日志数据，通过Spark Streaming SQL进行聚合后，将流计算结果数据实时写入Tablestore，展示一个简单的日志监控场景。场景设计假设有一个商品表Goods，商品信....

文章 2022-02-17 来自：开发者社区

15【在线日志分析】之基于Spark Streaming+Saprk SQL开发OnLineLogAanlysis2（Final）

1.influxdb创建database[root@sht-sgmhadoopdn-04 app]# influx -precision rfc3339Connected to http://localhost:8086 version 1.2.0InfluxDB shell version: 1.2.0>create database online_log_analysis 2...

文章 2022-02-17 来自：开发者社区

20【在线日志分析】之记录一次Spark Streaming+Spark SQL的数据倾斜

1.现象三台机器都有产生executor，每台都会产生tasks，但是其中只有一台的task有input数据，其他机器的tasks都没有数据。 2.猜想 2.1是不是数据倾斜? 是 2.2是数据量过大，group by时，导致key分布不均? 比如key1 有98万，key2有2万,那么shuffle时，肯定数据倾斜。但是我刚开始数据量不是很大，所以pass （就算数据量大，也很简单...

共有9条

< 1 >

跳转至： GO

更新时间 2024-07-02 16:52:15

本页面内关键词为智能算法引擎基于机器学习所生成，如有任何问题，可在页面下方点击"联系我们"与我们沟通。

SQL日志相关内容

SQL更多日志相关

SQL您可能感兴趣

产品推荐

{"moduleinfo":{"card_count":[{"count_phone":1,"count":1}],"search_count":[{"count_phone":4,"count":4}]},"card":[{"des":"云数据库SQL Server，具有企业许可授权，权限更为开放，引擎功能更为强大。能完美支持Windows平台的.NET架构，支持复杂SQL查询，性能优秀，并有强大的可视化管理工具，帮助您轻松管理数据。","link1":"https://rdsnew.console.aliyun.com/console/buy#/create/rds?initData=%7B%22data%22:%7B%22rds_region%22:%22cn-hangzhou%22,%22rds_dbtype%22:%22mssql%22,%22rds_dbversion%22:%222016_web%22,%22rds_web_type%22:%221%22,%22rds_class%22:%22mssql.x2.medium.w1%22,%22rds_storage%22:%2220%22,%22duration%22:12%7D%7D","link":"https://www.aliyun.com/product/rds/sqlserver","icon":"https://img.alicdn.com/tfs/TB1kuYmv3mTBuNjy1XbXXaMrVXa-200-200.png","btn2":"2016 标准版","tip":"完美兼容windows操作系统，已通过ISO 20000、SOC、PCI-DSS、等保三级等十项安全合规认证。         最新产品和实时动态重磅发布<a href=\"https://www.aliyun.com/product/new?source_type=out_sousuo_feature_0716\" target=\"_blank\">立即查看</a>","btn1":"2016 Web版","link2":"https://rdsnew.console.aliyun.com/console/buy#/create/rds?initData=%7B%22data%22:%7B%22rds_region%22:%22cn-shanghai%22,%22rds_dbtype%22:%22mssql%22,%22rds_dbversion%22:%222016_std_ha%22,%22rds_web_type%22:%221%22,%22rds_class%22:%22mssql.x4.medium.s2%22,%22rds_storage%22:%2220%22,%22duration%22:12%7D%7D","title":"云数据库RDS SQL Server版"}],"search":[{"txt":"SQL Server Web版","link":"https://promotion.aliyun.com/ntms/act/sqlserverweb.html"},{"txt":"SQL Server 标准版/企业版","link":"https://promotion.aliyun.com/ntms/act/sqlservergaokeyong.html"},{"txt":"快速入门SQL Server","link":"https://help.aliyun.com/document_detail/26141.html"},{"txt":"产品动态","link":"https://www.aliyun.com/product/new?source_type=out_sousuo_feature_0716"}],"countinfo":{"search":{"length_pc":0,"length":0},"card":{"length_pc":0,"length":0}},"simplifiedDisplay":"newEdition","newCard":[{"link":"https://www.aliyun.com/product/rds/sqlserver","title":"云数据库 RDS SQL Server 版","des":"云数据库SQL Server，具有企业许可授权，权限更为开放，引擎功能更为强大。能完美支持Windows平台的.NET架构，支持复杂SQL查询，性能优秀，并有强大的可视化管理工具，帮助您轻松管理数据。","btn1":"产品详情","link1":"https://www.aliyun.com/product/rds/sqlserver","btn2":"管理控制台","link2":"https://rdsnext.console.aliyun.com/#/rdsList/cn-hangzhou/basic/","link3":"https://www.aliyun.com/core/online-consult?from=OVtXEBmlH8","btn3":"在线咨询","infoGroup":[{"infoName":"最新活动","infoContent":{"firstContentName":"新用户年付低至3折起","firstContentLink":"https://www.aliyun.com/activity/database/bestoffers","lastContentName":"2核4GB 299元/年","lastContentLink":"https://rdsbuy.console.aliyun.com/newCreate/rds/mssql?initData={\"data\":{\"rds_dbtype\":\"\tmssql\",\"rds_dbversion\":\"2022_web\",\"rds_web_type\":\"1\",\"rds_class\":\"mssql.share2.medium.w1\",\"rds_storage\":\"100\",\"rds_nodetype\":\"Basic\",\"duration\":12,\"rds_storagetype\":\"cloud_auto\",\"rds_arch\":\"X86\"}}"}},{"infoName":"快速入门","infoContent":{"firstContentName":"帮助文档","firstContentLink":"https://help.aliyun.com/product/26090.html","lastContentName":"入门指南","lastContentLink":"https://help.aliyun.com/document_detail/53729.htm"}},{"infoName":"产品动态","infoContent":{"firstContentName":"存储能力全方位升级","firstContentLink":"https://www.aliyun.com/activity/database/rds-storage-upgrade","lastContentLink":"https://help.aliyun.com/zh/rds/bi-monthly-product-report-for-apsaradb-rds-march-april-2025-2","lastContentName":"RDS 产品双月报"}},{"infoName":"客户案例","infoContent":{"firstContentName":"RDS 助力巴黎奥运会系统稳定运行","firstContentLink":"https://help.aliyun.com/zh/rds/apsaradb-rds-for-sql-server/apsaradb-rds-helps-ensure-the-stability-of-the-system-of-the-paris-olympic-games","lastContentName":"","lastContentLink":""}}],"icon":"","ifIcon":"img","iconImg":"https://img.alicdn.com/tfs/TB1FIpvn3gP7K4jSZFqXXamhVXa-200-200.png","contentLink":"https://www.aliyun.com/product/rds/sqlserver"}]}

{"$env":{"JSON":{}},"$page":{"env":"production"},"$context":{"moduleinfo":{"card_count":[{"count_phone":1,"count":1}],"search_count":[{"count_phone":4,"count":4}]},"card":[{"des":"云数据库SQL Server，具有企业许可授权，权限更为开放，引擎功能更为强大。能完美支持Windows平台的.NET架构，支持复杂SQL查询，性能优秀，并有强大的可视化管理工具，帮助您轻松管理数据。","link1":"https://rdsnew.console.aliyun.com/console/buy#/create/rds?initData=%7B%22data%22:%7B%22rds_region%22:%22cn-hangzhou%22,%22rds_dbtype%22:%22mssql%22,%22rds_dbversion%22:%222016_web%22,%22rds_web_type%22:%221%22,%22rds_class%22:%22mssql.x2.medium.w1%22,%22rds_storage%22:%2220%22,%22duration%22:12%7D%7D","link":"https://www.aliyun.com/product/rds/sqlserver","icon":"https://img.alicdn.com/tfs/TB1kuYmv3mTBuNjy1XbXXaMrVXa-200-200.png","btn2":"2016 标准版","tip":"完美兼容windows操作系统，已通过ISO 20000、SOC、PCI-DSS、等保三级等十项安全合规认证。         最新产品和实时动态重磅发布<a href=\"https://www.aliyun.com/product/new?source_type=out_sousuo_feature_0716\" target=\"_blank\">立即查看</a>","btn1":"2016 Web版","link2":"https://rdsnew.console.aliyun.com/console/buy#/create/rds?initData=%7B%22data%22:%7B%22rds_region%22:%22cn-shanghai%22,%22rds_dbtype%22:%22mssql%22,%22rds_dbversion%22:%222016_std_ha%22,%22rds_web_type%22:%221%22,%22rds_class%22:%22mssql.x4.medium.s2%22,%22rds_storage%22:%2220%22,%22duration%22:12%7D%7D","title":"云数据库RDS SQL Server版"}],"search":[{"txt":"SQL Server Web版","link":"https://promotion.aliyun.com/ntms/act/sqlserverweb.html"},{"txt":"SQL Server 标准版/企业版","link":"https://promotion.aliyun.com/ntms/act/sqlservergaokeyong.html"},{"txt":"快速入门SQL Server","link":"https://help.aliyun.com/document_detail/26141.html"},{"txt":"产品动态","link":"https://www.aliyun.com/product/new?source_type=out_sousuo_feature_0716"}],"countinfo":{"search":{"length_pc":0,"length":0},"card":{"length_pc":0,"length":0}},"simplifiedDisplay":"newEdition","newCard":[{"link":"https://www.aliyun.com/product/rds/sqlserver","title":"云数据库 RDS SQL Server 版","des":"云数据库SQL Server，具有企业许可授权，权限更为开放，引擎功能更为强大。能完美支持Windows平台的.NET架构，支持复杂SQL查询，性能优秀，并有强大的可视化管理工具，帮助您轻松管理数据。","btn1":"产品详情","link1":"https://www.aliyun.com/product/rds/sqlserver","btn2":"管理控制台","link2":"https://rdsnext.console.aliyun.com/#/rdsList/cn-hangzhou/basic/","link3":"https://www.aliyun.com/core/online-consult?from=OVtXEBmlH8","btn3":"在线咨询","infoGroup":[{"infoName":"最新活动","infoContent":{"firstContentName":"新用户年付低至3折起","firstContentLink":"https://www.aliyun.com/activity/database/bestoffers","lastContentName":"2核4GB 299元/年","lastContentLink":"https://rdsbuy.console.aliyun.com/newCreate/rds/mssql?initData={\"data\":{\"rds_dbtype\":\"\tmssql\",\"rds_dbversion\":\"2022_web\",\"rds_web_type\":\"1\",\"rds_class\":\"mssql.share2.medium.w1\",\"rds_storage\":\"100\",\"rds_nodetype\":\"Basic\",\"duration\":12,\"rds_storagetype\":\"cloud_auto\",\"rds_arch\":\"X86\"}}"}},{"infoName":"快速入门","infoContent":{"firstContentName":"帮助文档","firstContentLink":"https://help.aliyun.com/product/26090.html","lastContentName":"入门指南","lastContentLink":"https://help.aliyun.com/document_detail/53729.htm"}},{"infoName":"产品动态","infoContent":{"firstContentName":"存储能力全方位升级","firstContentLink":"https://www.aliyun.com/activity/database/rds-storage-upgrade","lastContentLink":"https://help.aliyun.com/zh/rds/bi-monthly-product-report-for-apsaradb-rds-march-april-2025-2","lastContentName":"RDS 产品双月报"}},{"infoName":"客户案例","infoContent":{"firstContentName":"RDS 助力巴黎奥运会系统稳定运行","firstContentLink":"https://help.aliyun.com/zh/rds/apsaradb-rds-for-sql-server/apsaradb-rds-helps-ensure-the-stability-of-the-system-of-the-paris-olympic-games","lastContentName":"","lastContentLink":""}}],"icon":"","ifIcon":"img","iconImg":"https://img.alicdn.com/tfs/TB1FIpvn3gP7K4jSZFqXXamhVXa-200-200.png","contentLink":"https://www.aliyun.com/product/rds/sqlserver"}]}}

云数据库 RDS SQL Server 版

云数据库SQL Server，具有企业许可授权，权限更为开放，引擎功能更为强大。能完美支持Windows平台的.NET架构，支持复杂SQL查询，性能优秀，并有强大的可视化管理工具，帮助您轻松管理数据。

产品详情

管理控制台

在线咨询