Package dev.langchain4j.data.document.splitter

Class HierarchicalDocumentSplitter

java.lang.Object

dev.langchain4j.data.document.splitter.HierarchicalDocumentSplitter

All Implemented Interfaces:

dev.langchain4j.data.document.DocumentSplitter

Direct Known Subclasses:

DocumentByCharacterSplitter, DocumentByLineSplitter, DocumentByParagraphSplitter, DocumentByRegexSplitter, DocumentBySentenceSplitter, DocumentByWordSplitter

public abstract class HierarchicalDocumentSplitter
extends Object
implements dev.langchain4j.data.document.DocumentSplitter

Base class for hierarchical document splitters.

Extends DocumentSplitter and provides machinery for sub-splitting documents when a single segment is too long.

Field Summary

Fields

Modifier and Type	Field	Description
protected final int	maxOverlapSize
protected final int	maxSegmentSize
protected final dev.langchain4j.data.document.DocumentSplitter	subSplitter
protected final dev.langchain4j.model.Tokenizer	tokenizer

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	HierarchicalDocumentSplitter(int maxSegmentSizeInChars, int maxOverlapSizeInChars)	Creates a new instance of HierarchicalDocumentSplitter.
protected	HierarchicalDocumentSplitter(int maxSegmentSizeInChars, int maxOverlapSizeInChars, HierarchicalDocumentSplitter subSplitter)	Creates a new instance of HierarchicalDocumentSplitter.
protected	HierarchicalDocumentSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, dev.langchain4j.model.Tokenizer tokenizer)	Creates a new instance of HierarchicalDocumentSplitter.
protected	HierarchicalDocumentSplitter(int maxSegmentSizeInTokens, int maxOverlapSizeInTokens, dev.langchain4j.model.Tokenizer tokenizer, dev.langchain4j.data.document.DocumentSplitter subSplitter)	Creates a new instance of HierarchicalDocumentSplitter.

Method Summary

Modifier and Type	Method	Description
protected abstract dev.langchain4j.data.document.DocumentSplitter	defaultSubSplitter()	The default sub-splitter to use when a single segment is too long.
protected abstract String	joinDelimiter()	Delimiter string to use to re-join the parts.
List<dev.langchain4j.data.segment.TextSegment>	split(dev.langchain4j.data.document.Document document)
protected abstract String[]	split(String text)	Splits the provided text into parts.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface dev.langchain4j.data.document.DocumentSplitter

splitAll

Field Details

maxSegmentSize

protected final int maxSegmentSize

maxOverlapSize

protected final int maxOverlapSize

tokenizer

protected final dev.langchain4j.model.Tokenizer tokenizer

subSplitter

protected final dev.langchain4j.data.document.DocumentSplitter subSplitter

Constructor Details

HierarchicalDocumentSplitter

protected HierarchicalDocumentSplitter(int maxSegmentSizeInChars,
 int maxOverlapSizeInChars)

Creates a new instance of HierarchicalDocumentSplitter.

Parameters:

maxSegmentSizeInChars - The maximum size of a segment in characters.

maxOverlapSizeInChars - The maximum size of the overlap between segments in characters.

HierarchicalDocumentSplitter

protected HierarchicalDocumentSplitter(int maxSegmentSizeInChars,
 int maxOverlapSizeInChars,
 HierarchicalDocumentSplitter subSplitter)

Creates a new instance of HierarchicalDocumentSplitter.

Parameters:

maxSegmentSizeInChars - The maximum size of a segment in characters.

maxOverlapSizeInChars - The maximum size of the overlap between segments in characters.

subSplitter - The sub-splitter to use when a single segment is too long.

HierarchicalDocumentSplitter

protected HierarchicalDocumentSplitter(int maxSegmentSizeInTokens,
 int maxOverlapSizeInTokens,
 dev.langchain4j.model.Tokenizer tokenizer)

Creates a new instance of HierarchicalDocumentSplitter.

Parameters:

maxSegmentSizeInTokens - The maximum size of a segment in tokens.

maxOverlapSizeInTokens - The maximum size of the overlap between segments in tokens.

tokenizer - The tokenizer to use to estimate the number of tokens in a text.

HierarchicalDocumentSplitter

protected HierarchicalDocumentSplitter(int maxSegmentSizeInTokens,
 int maxOverlapSizeInTokens,
 dev.langchain4j.model.Tokenizer tokenizer,
 dev.langchain4j.data.document.DocumentSplitter subSplitter)

Creates a new instance of HierarchicalDocumentSplitter.

Parameters:

maxSegmentSizeInTokens - The maximum size of a segment in tokens.

maxOverlapSizeInTokens - The maximum size of the overlap between segments in tokens.

tokenizer - The tokenizer to use to estimate the number of tokens in a text.

subSplitter - The sub-splitter to use when a single segment is too long.

Method Details

split

protected abstract String[] split(String text)

Splits the provided text into parts. Implementation API.

Parameters:

text - The text to be split.

Returns:

An array of parts.

joinDelimiter

protected abstract String joinDelimiter()

Delimiter string to use to re-join the parts.

Returns:

The delimiter.

defaultSubSplitter

protected abstract dev.langchain4j.data.document.DocumentSplitter defaultSubSplitter()

The default sub-splitter to use when a single segment is too long.

Returns:

The default sub-splitter.

split

public List<dev.langchain4j.data.segment.TextSegment> split(dev.langchain4j.data.document.Document document)

Specified by:

split in interface dev.langchain4j.data.document.DocumentSplitter