com.alibaba.dashscope.tokenizers

类 QwenTokenizer

java.lang.Object

com.alibaba.dashscope.tokenizers.QwenTokenizer

所有已实现的接口:

Tokenizer

public class QwenTokenizer
extends Object
implements Tokenizer

BPE encode and decode, implementation reference https://github.com/openai/tiktoken and https://github.com/karpathy/minbpe

构造器概要

构造器

构造器和说明
QwenTokenizer()

方法概要

限定符和类型	方法和说明
String	decode(List<Integer> tokens) Decode token ids to String
List<Integer>	encode(String text, String allowedSpecial) Encode the input text, handles special tokens.
List<Integer>	encodeOrdinary(String text) Encoding that ignores any special tokens.

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

QwenTokenizer

public QwenTokenizer()

方法详细资料

encodeOrdinary

public List<Integer> encodeOrdinary(String text)

Encoding that ignores any special tokens.

指定者:

encodeOrdinary 在接口中 Tokenizer

参数:

text - The input.

返回:

The list of token ids.

encode

public List<Integer> encode(String text,
                            String allowedSpecial)
                     throws NoSpecialTokenExists,
                            UnSupportedSpecialTokenMode

Encode the input text, handles special tokens.

指定者:

encode 在接口中 Tokenizer

参数:

text - The input to be encode.

allowedSpecial - The special token options can be "all"|"none"|"none_raise", if none_raise, then an error is raised if any special token is encountered in text, if null, use "all"

返回:

The list of token encode.

抛出:

NoSpecialTokenExists - No special token in the input.

UnSupportedSpecialTokenMode - the allowedSpecial is not["all"|"none"|"none_raise"]

decode

public String decode(List<Integer> tokens)

从接口复制的说明: Tokenizer

Decode token ids to String

指定者:

decode 在接口中 Tokenizer

参数:

tokens - The input token ids, eg: [2610, 525, 264, 10950, 17847, 13], will be Decoded to "You are a helpful assistant."

返回:

The token ids corresponding string.