qianfan.resources.tools package

Submodules

Tokenizer

class qianfan.resources.tools.tokenizer.Tokenizer[source]

Bases: object

Class for Tokenizer API

classmethod count_tokens(text: str, mode: Literal['local', 'remote'] = 'local', model: str = 'ERNIE-Bot', **kwargs: Any) → int[source]

Count the number of tokens in a given text.

Parameters:

text (str):

The input text for which tokens need to be counted.

mode (str, optional):

local (default):: local SIMULATION (Chinese characters count + English word count * 1.3)
remote:: use qianfan api to calculate the token count. API will return accurate token count, but only ERNIE-Bot series models are supported.

model (str, optional):

The name of the model to be used for token counting, which may influence the counting strategy. Default is ‘ERNIE-Bot’.

kwargs (Any):

Additional keyword arguments that can be passed to customize the request.

Utils for qianfan api

qianfan.resources.tools.utils.async_qianfan_api_request(func: Callable[[P], Awaitable[QfRequest]]) → Callable[[P], Awaitable[QfResponse]][source]: wrapper for all functions in sdk for qianfan api, so that the function only needs to provide the request this decorator will: 1. extract ak and sk from kwargs 2. extract retry config from kwargs 3. use the requestor to send request 4. return the response to the user

qianfan.resources.tools.utils.qianfan_api_request(func: Callable[[P], QfRequest]) → Callable[[P], QfResponse][source]: wrapper for all functions in sdk for qianfan api, so that the function only needs to provide the request this decorator will: 1. extract ak and sk from kwargs 2. extract retry config from kwargs 3. use the requestor to send request 4. return the response to the user