文档分割方式

1 固定长度/分割符

如：RecursiveCharacterTextSplitter, CharacterTextSplitter
CharacterTextSplitter: 按照固定长度顺序分割，同时有一定的overlap
RecursiveCharacterTextSplitter: 按一个字符优先级（如 \n\n, \n, 括号... ）递归地分割，很适合处理类似括号这样的嵌套引用
TokenTextSplitter：基于Token的长度进行顺序分割，同时有一定的overlap
也可以直接基于某个分割符简单分割：

docs = text.split(".")

CharacterTextSplitter:

from langchain.text_splitter import CharacterTextSplitter
text_splitter = CharacterTextSplitter(
    separator = "\n",
    chunk_size = 64,
    chunk_overlap  = 20
)
docs = text_splitter.create_documents([text])
print(docs)

2 规范格式的分割

MarkdownHeaderTextSplitter: 适用于Markdown文件
LatexTextSplitter: 适用于Latex文件
PyPDFLoader: 适用于PDF文件

from langchain.document_loaders import PyPDFLoader
loader = PyPDFLoader("xxx.pdf")
pages = loader.load()

3 按规则分句工具

NLTK 或者 spaCy
NLTK使用的句子拆分原理是基于训练好的模型和规则。它使用了一种称为句子边界检测（Sentence Boundary Detection，SBD）的技术，该技术利用了标点符号、缩略词、数字和其他语言特定的规则来确定句子的边界。依赖语言学上的规则和模式，而不是句子之间的语义关系。
拆英文句子：

import nltk
nltk.download('punkt')
from nltk.tokenize import sent_tokenize
text = "Hello, how are you? I'm doing well. Thanks for asking."
sentences = sent_tokenize(text)
print(sentences)
# ['Hello, how are you?', "I'm doing well.", 'Thanks for asking.']

拆中文句子要用jieba分词库：

import nltk
import jieba
from nltk.tokenize import sent_tokenize

# 载入中文分词词典
jieba.initialize()

# 使用nltk的sent_tokenize函数来拆分中文文本
def chinese_sent_tokenize(text):
    sentences = []
    seg_list = jieba.cut(text, cut_all=False)
    seg_str = ' '.join(seg_list)
    for sent in seg_str.split('。'):
        sentences.append(sent.strip() + '。')
    return sentences

sentences = chinese_sent_tokenize(text)

4 基于语义进行分割

1. 基于 BERT 的 cross-segment 模型
2. seqModel:
   一个实例: nlp_bert_document-segmentation_chinese-base

https://modelscope.cn/models/iic/nlp_bert_document-segmentation_chinese-base/summary

from modelscope.outputs import OutputKeys
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

p = pipeline(
    task=Tasks.document_segmentation,
    model='damo/nlp_bert_document-segmentation_chinese-base')

result = p(documents=text)

print(result[OutputKeys.TEXT])

参考：
https://zhuanlan.zhihu.com/p/673906072
https://zhuanlan.zhihu.com/p/666273413
https://blog.csdn.net/hmywillstronger/article/details/130073676
LangChain+LLM 本地知识库：
https://blog.csdn.net/v_JULY_v/article/details/131552592
seqModel:
https://blog.csdn.net/weixin_48827824/article/details/126952959
从cross-segment到seqModel：
https://blog.csdn.net/v_JULY_v/article/details/135386202