huggingface-tokenizers

Name: huggingface-tokenizers
Rating: 5 (7 reviews)
Author: braxtonROSE4

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.

7stars

Updated 2 months ago

View on GitHub ↗License: MIT

How to add

/plugin marketplace add braxtonROSE4/zorro-agent

The exact command may vary by repository. Check the README on GitHub.

For the skill author

Drop this on your repo README

Shows your skill is listed on Skillteca, generates a backlink and trackable traffic.

[![Listada na Skillteca](https://www.skillteca.com.br/api/badge/huggingface-tokenizers-braxtonrose4/svg)](https://www.skillteca.com.br/skills/huggingface-tokenizers-braxtonrose4?utm_source=badge&utm_medium=readme&utm_campaign=badge)

#ai #word

Related skills

See all in Pesquisa e Web →

dev-browser

60.3k

Automates browser interactions with persistent page state. Use for navigating websites, filling forms, taking screenshots, extracting web data, testing web apps, or automating browser workflows.

Pesquisa e Web#testby code-yeongyu

agent-browser

55.6k

Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to "open a website", "fill out a form", "click a button", "take a screenshot", "scrape data from a page", "test this web app", "login to

Pesquisa e Web#ai#testby shanraisshan