public interface Tokenizer
List<Integer> encodeOrdinary(String text)
text - The input.List<Integer> encode(String text, String allowedSpecial) throws NoSpecialTokenExists, UnSupportedSpecialTokenMode
text - The input to be encode.allowedSpecial - The special token options can be "all"|"none"|"none_raise", if
none_raise, then an `NoSpecialTokenExists` is throw if any special token is encountered in
text, if null, use "all"NoSpecialTokenExists - No special token in the input.UnSupportedSpecialTokenMode - the allowedSpecial is not["all"|"none"|"none_raise"]Copyright © 2024. All rights reserved.