com.alibaba.dashscope.tokenizers

接口 Tokenizer

所有已知实现类:

QwenTokenizer

public interface Tokenizer

方法概要

限定符和类型	方法和说明
String	decode(List<Integer> ids) Decode token ids to String
List<Integer>	encode(String text, String allowedSpecial) Encode the input text, handles special tokens.
List<Integer>	encodeOrdinary(String text) Encoding that ignores any special tokens.

方法详细资料

encodeOrdinary

List<Integer> encodeOrdinary(String text)

Encoding that ignores any special tokens. There is no guarantee that the results will be completely consistent with the model output. Compared with the python version, there are some differences. It is known that the results of processing 2 or 4 consecutive spaces are different. Detailed information for qwen reference: https://github.com/QwenLM/Qwen/blob/main/tokenization_note_zh.md

参数:

text - The input.

返回:

The list of token ids.

encode

List<Integer> encode(String text,
                     String allowedSpecial)
              throws NoSpecialTokenExists,
                     UnSupportedSpecialTokenMode

Encode the input text, handles special tokens. There is no guarantee that the results will be completely consistent with the model output. Compared with the python version, there are some differences. It is known that the results of processing 2 or 4 consecutive spaces are different. Detailed information for qwen reference: https://github.com/QwenLM/Qwen/blob/main/tokenization_note_zh.md

参数:

text - The input to be encode.

allowedSpecial - The special token options can be "all"|"none"|"none_raise", if none_raise, then an `NoSpecialTokenExists` is throw if any special token is encountered in text, if null, use "all"

返回:

The list of token encode.

抛出:

NoSpecialTokenExists - No special token in the input.

UnSupportedSpecialTokenMode - the allowedSpecial is not["all"|"none"|"none_raise"]

decode

String decode(List<Integer> ids)

Decode token ids to String

参数:

ids - The input token ids, eg: [2610, 525, 264, 10950, 17847, 13], will be Decoded to "You are a helpful assistant."

返回:

The token ids corresponding string.