feature/onnx/com.shifthackz.aisdv1.feature.onnx.ai.tokenizer/EnglishTextTokenizer/bpe

bpe

private fun bpe(token: String): List<String>(source)

Executes the bpe step in the SDAI ONNX local diffusion feature layer.

Return

Result produced by bpe.

Author

Dmitriy Moroz

Parameters

token

token value consumed by the API.