feature/onnx/com.shifthackz.aisdv1.feature.onnx.ai.tokenizer/EnglishTextTokenizer

EnglishTextTokenizer

internal class EnglishTextTokenizer(val ortEnvironmentProvider: OrtEnvironmentProvider, val fileProviderDescriptor: FileProviderDescriptor, val localModelIdProvider: LocalModelIdProvider, val preferenceManager: PreferenceManager) : LocalDiffusionTextTokenizer(source)

Coordinates EnglishTextTokenizer behavior in the SDAI ONNX local diffusion feature layer.

Author

Dmitriy Moroz

Constructors

EnglishTextTokenizer

constructor(ortEnvironmentProvider: OrtEnvironmentProvider, fileProviderDescriptor: FileProviderDescriptor, localModelIdProvider: LocalModelIdProvider, preferenceManager: PreferenceManager)

Types

Companion

object Companion

Provides the companion object singleton used by the SDAI ONNX local diffusion feature layer.

Properties

bpeRanks

private val bpeRanks: MutableMap<Pair<String, String>, Int?>

Exposes the bpeRanks value used by the SDAI ONNX local diffusion feature layer.

decoder

private val decoder: MutableMap<Int, String>

Exposes the decoder value used by the SDAI ONNX local diffusion feature layer.

encoder

private val encoder: MutableMap<String, Int>

Exposes the encoder value used by the SDAI ONNX local diffusion feature layer.

fileProviderDescriptor

private val fileProviderDescriptor: FileProviderDescriptor

Exposes the fileProviderDescriptor value used by the SDAI ONNX local diffusion feature layer.

isInitMap

private var isInitMap: Boolean

Exposes the isInitMap value used by the SDAI ONNX local diffusion feature layer.

localModelIdProvider

private val localModelIdProvider: LocalModelIdProvider

Exposes the localModelIdProvider value used by the SDAI ONNX local diffusion feature layer.

maxLength

open override val maxLength: Int = 77

Exposes the maxLength value used by the SDAI ONNX local diffusion feature layer.

ortEnvironmentProvider

private val ortEnvironmentProvider: OrtEnvironmentProvider

Exposes the ortEnvironmentProvider value used by the SDAI ONNX local diffusion feature layer.

pattern

private val pattern: Pattern?

Exposes the pattern value used by the SDAI ONNX local diffusion feature layer.

preferenceManager

private val preferenceManager: PreferenceManager

Exposes the preferenceManager value used by the SDAI ONNX local diffusion feature layer.

session

private var session: OrtSession?

Exposes the session value used by the SDAI ONNX local diffusion feature layer.

Functions

bpe

private fun bpe(token: String): List<String>

Executes the bpe step in the SDAI ONNX local diffusion feature layer.

open override fun close()

Executes the close step in the SDAI ONNX local diffusion feature layer.

createUnconditionalInput

open override fun createUnconditionalInput(text: String?): IntArray

Creates the SDAI value produced by createUnconditionalInput.

decode

open override fun decode(ids: IntArray?): String

Executes the decode step in the SDAI ONNX local diffusion feature layer.

encode

open override fun encode(text: String?): IntArray

Executes the encode step in the SDAI ONNX local diffusion feature layer.

getPairs

private fun getPairs(word: List<String?>): Set<Pair<String, String>>

Loads SDAI data through getPairs.

initialize

open override fun initialize()

Executes the initialize step in the SDAI ONNX local diffusion feature layer.

loadBpeRanks

private fun loadBpeRanks(): Map<Pair<String, String>, Int?>

Loads SDAI data through loadBpeRanks.

loadDecoder

private fun loadDecoder(encoder: Map<String, Int>): Map<Int, String>

Loads SDAI data through loadDecoder.

loadEncoder

private fun loadEncoder(): Map<String, Int>

Loads SDAI data through loadEncoder.

tensor

open override fun tensor(ids: IntArray?): OnnxTensor?

Executes the tensor step in the SDAI ONNX local diffusion feature layer.