Log Aggregation Study

一、 基于日志模板解析的聚合算法（结构化方法）

这类算法的核心思想是**“去变量保结构”**。通过识别日志中的常量部分（模板）和变量部分（参数），将同一模板的日志归为一类。这是解决“时间戳、随机ID变化”最直接、性价比最高的方法。

1. Drain (推荐指数：⭐⭐⭐⭐⭐)

  • 原理：一种基于固定深度树的在线解析算法。它假设日志的开头部分通常是固定的，通过预定义的深度和相似度阈值，快速将日志流分组并提取模板。
  • 适用性：
    • 优势：完全符合文档中Flink流引擎的场景，处理速度极快，几乎不增加延迟。能精准剔除时间戳、SessionID等变量。
    • 解决痛点：完美解决“同一攻击因payload细微差异被拆分”的问题。
    • 落地建议：作为聚合的第一道防线。例如将 SQL Injection id=123 和 SQL Injection id=456 解析为模板 SQL Injection id=<*>，基于模板Hash进行聚合。

2. Spell (流式最长公共子序列)

  • 原理：基于LCS（最长公共子序列）在线提取日志模板。
  • 适用性：适用于日志长度变化较大，但核心关键词位置相对固定的场景。比Drain稍慢，但容错性略好。

3. LogPai / LogParse

  • 说明：这是一系列工业级日志解析工具的集合，集成了多种算法（如IPLoM, LenMa等），适合离线分析或批处理引擎场景，用于建立基线。

二、 基于局部敏感哈希的聚合算法（高性能去重）

当日志无法完全结构化（如加密流量、非标准文本）时，需要比对文本的相似度。传统Hash要求输入完全一致，而LSH允许内容相似但不完全相同的日志生成相同的Hash值。

1. SimHash (推荐指数：⭐⭐⭐⭐)

  • 原理：将高维特征向量映射为低维指纹，通过计算海明距离判断相似度。
  • 适用性：
    • 场景：非常适合文档中提到的“payload细微差异”场景。
    • 落地：提取日志的关键特征（如IP、URL路径、特定关键词），生成SimHash指纹。若两条日志指纹的海明距离小于阈值（如3），则聚合。
    • 优势：计算复杂度低，极适合海量数据实时去重。

2. MinHash

  • 原理：利用Jaccard相似度原理，通过随机置换哈希快速估算集合的相似度。
  • 适用性：比SimHash更精确，适合对聚合准确性要求更高的离线分析场景。

三、 基于语义向量化的聚合算法（智能化方案）

这是文档中提到的“新技术研究院”方向，利用NLP技术理解日志语义，解决变种攻击（如同义词替换、编码变换）的聚合问题。

1. FastText / Word2Vec + K-Means (推荐指数：⭐⭐⭐)

  • 原理：将日志分词后训练或加载预训练词向量，计算句向量。
  • 适用性：作为轻量级AI方案。对日志中的关键词（如 select, union, alert）进行向量化，忽略无关噪声。相比深度模型，FastText推理速度极快，能满足2万EPS的性能指标。

2. LogBERT / BERT (推荐指数：⭐⭐⭐)

  • 原理：利用预训练语言模型提取深层语义特征。
  • 创新点：可以使用专门针对安全日志微调过的BERT模型。
  • 落地难点：BERT计算开销大。若要在流引擎中实现2万+ EPS，必须配合模型蒸馏或ONNX Runtime加速。
  • 策略：建议仅在模板解析无法区分（即结构相似但语义不同，例如都是HTTP 500错误，一个是业务报错，一个是攻击溢出）的高价值日志上启用向量聚合。

四、 针对特定攻击类型的聚合Key提取算法

文档提到“不同威胁类型的聚合关键key不同”，这需要结合规则提取与上下文感知。

1. 正则提取 + 动态打标

  • SQL注入聚合：编写正则提取注入点特征（如 id=1'），忽略后续变长的Payload数据，以注入点特征作为聚合Key。
  • XSS聚合：提取攻击参数中的核心标签（如 <script>），忽略随机化的干扰字符。

2. NER (命名实体识别)

  • 方法：训练一个轻量级NER模型（如BiLSTM-CRF），从日志中提取实体：Attack_Vector（攻击载荷）、Target_Asset（目标资产）、Method（方法）。
  • 聚合逻辑：只有当 (Attack_Vector, Target_Asset, Method) 三元组相似时才聚合。

五、 推荐的综合落地方案（混合架构）

为了满足文档中 “SG-6000-ISC6320 高端机型 25,000 EPS” 的性能指标，同时解决噪音问题，建议构建三级流水线聚合架构：

层级 算法/技术 目的 性能开销 准确度
L1: 粗粒度过滤 Drain + 模板Hash 快速剔除时间戳、随机ID等显性噪声，将结构相同的日志聚合。 极低 高（针对结构化日志）
L2: 模糊去重 SimHash / MinHash 针对L1无法命中的半结构化日志，进行相似度快速比对。 中 中
L3: 深度语义聚合 FastText / DistilBERT 仅针对L1/L2判定为“未知类型”或“高可疑”的日志进行向量化计算，解决语义变种问题。 高 极高

实施路径建议：

  1. 第一阶段：部署 Drain算法 集成到Flink，解决80%的基础噪音问题（时间戳、ID），确保性能指标达标。
  2. 第二阶段：引入 SimHash 解决HTTP URL参数变异等模糊匹配问题。
  3. 第三阶段：接入新技术研究院的 轻量化向量化模型，解决高级变种攻击的语义聚合问题。

Comments

Post a Comment

Popular posts from this blog

2023: On the Road

There're so many dramas happened in the past 3 years. People say there'll be more in the coming 2023. Basically, I agree with this. But no matter what happens in the physical world, we need to concentrate on something really important and interesting. In the past 2022, my most profound change was, that I started thinking seriously about the future living style. My family and friends mostly live like plants, fixed by our roots, and change ourselves to adapt to the environment. We are where we are, proud of its history and culture, and contained by it. It'll not be the case anymore. In 2022, I learned, read, listened, wrote, and spoke in second languages, especially English and Japanese. They bring me a new world. I also read several books on Chinese and middle east histories. The world before me is fusing as a whole sphere. China and Chinese people are a part, and the most familiar part for me, the starting point of my exploration. It gives me enough confidence as a foreig...
Read more

Yet another advice to kids

Sharon posted some advice to the youth (although she herself is young). She suggested being independent, with no fear of failure, which is excellent advice. I'd like to give some more "concrete" tips to kids or myself in childhood (if I have the chance to tell that me). First things first, the following suggestions can't make you more competitive than your peers or perform better in exams. or win the girl make you fascinated. They make your everyday life more exciting and fancy even if you're not rich or powerful. They're neither expensive (so you can achieve these goals even if not born into a rich family) nor painful. What you need is to know them, and keep doing them. OK, here we go. 1. Find a sport, enjoy it, and be good at it. Sport is a way to talk with your body, with the physical world, express yourself, and communicate with people. 2. Be honest, especially to yourself. Living honestly in Chinese society looks stupid and clumsy. Never mind. Being hones...
Read more

The Joy of Reading in Natural Light

Nowadays I read books in paperbacks much less than ebooks. Not because I don't like reading paper books, but after reading for so many years on screen, my eyes are very sensitive to the brightness of reading materials. It's very hard for me to read characters when the material is not bright enough. Unfortunately, the light in my living room is not bright enough for me. This morning it was cloudy, while in the afternoon it got clear, and the blue sky and white clouds appeared. So I moved the desk near the window. To my surprise, the characters on paper books are much more clear and more comfortable to read than in artificial light. Maybe such a simple fact is not worthy to mention. But to me, a programmer spent too much time with the keyboard and screen, this reminds me of the memories of childhood, reading a book, or writing homework in the yard. The characters blurred at sunset. That's when we were going to have super. The window of my living room is east-forwarding, so to...
Read more