Python后端AI专题18：分块策略实测：块多大、重叠多少才不会拆坏答案

Python后端AI专题18分块策略实测块多大、重叠多少才不会拆坏答案块越小检索定位更精确却容易把条件与结论拆开块越大上下文完整却带入更多噪声并消耗 Tokenoverlap 能缓解边界问题但会重复存储和重复 Embedding。本篇不用“推荐 500 字”结束讨论而是让同一份语料跑九组参数。清洗来源保真测试完整答案deftest_cleaning_removes_repeated_headers_without_losing_source()-None:blocks[ParsedBlock(内部资料,0,page1),ParsedBlock(第一章\t 请假流程\n\n需要审批。,1,page1,section请假,metadata{source:p1}),ParsedBlock(内部资料,2,page2),ParsedBlock(第二章\t报销流程\n\n需要发票。,3,page2,section报销,metadata{source:p2}),ParsedBlock(内部资料,4,page3),ParsedBlock(第三章 采购流程\n\n需要询价。,5,page3,section采购,metadata{source:p3}),]cleanedclean_blocks(blocks)assert[block.textforblockincleaned][第一章 请假流程\n需要审批。,第二章 报销流程\n需要发票。,第三章 采购流程\n需要询价。,]assert[block.ordinalforblockincleaned][0,1,2]assert[block.pageforblockincleaned][1,2,3]assert[block.sectionforblockincleaned][请假,报销,采购]assert[block.metadataforblockincleaned][{source:p1},{source:p2},{source:p3}]真实结果1 passed in 0.33s。先看分块算法怎样找边界每个块先计算硬上限start chunk_size若不是文末就在后 2/3 区间逆向寻找双换行、换行、中文句号/叹号/问号、英文句号空格。找到则在语义边界结束找不到才硬切。candidates[block.text.rfind(marker,startmax(1,chunk_size//3),hard_end)formarkerin(\n\n,\n,。,,,. )]boundarymax(candidates)搜索从块的 1/3 后开始避免为了一个很早的句号产生极短块。每个结果保留start_char/end_char后续可以回到原块高亮而不只保存复制文本。overlap 怎样推进游标startmax(start1,end-overlap)end - overlap让下一块重读上一块尾部start 1防止极端边界下游标不前进。参数入口要求chunk_size overlap 0否则可能无限循环。完整分块模块from__future__importannotationsfromdataclassesimportdataclassfromtypingimportSequencefromapp.services.ingestion.documentsimportParsedBlockdataclass(frozenTrue,slotsTrue)classTextChunk:text:strordinal:intsource_ordinal:intstart_char:intend_char:intpage:int|Nonesection:str|Nonedefchunk_blocks(blocks:Sequence[ParsedBlock],*,chunk_size:int,overlap:int)-list[TextChunk]:ifchunk_size0oroverlap0oroverlapchunk_size:raiseValueError(require chunk_size overlap 0)result:list[TextChunk][]forblockinblocks:start0whilestartlen(block.text):hard_endmin(startchunk_size,len(block.text))endhard_endifhard_endlen(block.text):candidates[block.text.rfind(marker,startmax(1,chunk_size//3),hard_end,)formarkerin(\n\n,\n,。,,,. )]boundarymax(candidates)ifboundarystart:marker_length(2ifblock.text[boundary:boundary2]in{\n\n,. }else1)endboundarymarker_length pieceblock.text[start:end]result.append(TextChunk(textpiece,ordinallen(result),source_ordinalblock.ordinal,start_charstart,end_charend,pageblock.page,sectionblock.section,))ifendlen(block.text):breakstartmax(start1,end-overlap)returnresult实验不是用三句短文本糊弄实验包含退款、SSE 和多租户三份约 700 字的语料把目标答案放在前后背景之间。每组统计块数、平均/最大长度、目标答案是否完整出现在某个块、重复率。命令$env:PYTHONIOENCODINGutf-8.\.venv\Scripts\python scripts\run_chunking_experiment.py结果摘要size/overlap块数平均长度答案覆盖重复率256/010206.301.00.000256/6411234.091.00.248256/12815239.931.00.745512/06343.831.00.000512/646375.831.00.093512/1286407.831.00.186800/03687.671.00.000800/643687.671.00.000800/1283687.671.00.000第二张图中蓝柱是重复率黄线是块数。256/128 把 2063 个唯一字符扩成约 3600 个待 Embedding 字符重复率 74.5%代价明显。800 的 overlap 没生效不是脚本坏了而是每份样本都短于 800、各自只有一块这提醒我们参数效果依赖文档长度分布。九组答案覆盖都为 1不能据此宣布 800 最优实验只有三个目标短语尚未测长条件、表格和跨段问答。参数选择要加入真实标注问答以 RecallK 和成本共同决策。本篇最终完整模块chunking.py前面的代码片段用于解释本次改动下面是本篇结束时可直接核对和替换的磁盘完整版本。from__future__importannotationsfromdataclassesimportdataclassfromtypingimportSequencefromapp.services.ingestion.documentsimportParsedBlockdataclass(frozenTrue,slotsTrue)classTextChunk:text:strordinal:intsource_ordinal:intstart_char:intend_char:intpage:int|Nonesection:str|Nonedefchunk_blocks(blocks:Sequence[ParsedBlock],*,chunk_size:int,overlap:int)-list[TextChunk]:ifchunk_size0oroverlap0oroverlapchunk_size:raiseValueError(require chunk_size overlap 0)result:list[TextChunk][]forblockinblocks:start0whilestartlen(block.text):hard_endmin(startchunk_size,len(block.text))endhard_endifhard_endlen(block.text):candidates[block.text.rfind(marker,startmax(1,chunk_size//3),hard_end)formarkerin(\n\n,\n,。,,,. )]boundarymax(candidates)ifboundarystart:marker_length2ifblock.text[boundary:boundary2]in{\n\n,. }else1endboundarymarker_length pieceblock.text[start:end]result.append(TextChunk(textpiece,ordinallen(result),source_ordinalblock.ordinal,start_charstart,end_charend,pageblock.page,sectionblock.section,))ifendlen(block.text):breakstartmax(start1,end-overlap)returnresult本篇练习设计一条会被切坏的样本在实验 CASES 中加入一份长文让完整答案“定制商品不适用七天无理由退款”恰好可能跨 256 字符附近。不要直接调整字符串到某个魔法下标用多个自然段构造。比较 256/0 与 256/64输出哪些 chunk 包含“定制商品”和“无理由退款”判断完整答案覆盖是否改善计算新增重复字符用结果说明为什么 overlap 只能缓解边界不能替代按标题/段落分块。下一篇会给出可运行实验改法并进入批量 Embedding当第三批失败时如何从持久化游标继续而不重复写入。