$ cutoken --help

FEATURES

Every feature is designed for one goal: reduce tokens without losing meaning.

PASS 1

NORMALIZE

Zero-semantic-change cleanup. Runs first on every mode.

NFKC

Unicode NFKC Normalization

Composes characters, standardizes compatibility equivalents.

QUOT

Smart Quote Conversion

Converts curly quotes (""''‚„) to straight ASCII equivalents.

DASH

Em/En Dash Handling

Converts — and – to simple hyphens for token efficiency.

PUNCT

Punctuation Cleanup

Collapses ... / !!! / ???, removes decorative lines (--- === ~~).

WS

Whitespace Normalization

CRLF→LF, tabs→spaces, collapse multi-spaces, trim trailing.

$ input: "Hello\u201cWorld\u201d\u2014this\u2026\u2026\u2026"
$ output: "Hello\"World\"-this..."
$ cutoken normalize --input prompt.txt 
✓ NFKC applied
✓ 4 fancy quotes converted
✓ 2 em-dashes normalized
✓ 6 excess punctuation removed
✓ whitespace collapsed

TOKENS: 142 → 138 (-2.8%)
$ cutoken lexical --mode=balanced 
✓ Stutters collapsed: 3
✓ Commands stripped: "please note that"
✓ Hedges removed: 7
✓ Fillers dropped: 12
✓ Greetings stripped: "Hello there"
✓ Phrases compressed: 5
✓ Synonyms swapped: 3
✓ Numbers→digits: 2

TOKENS: 523 → 341 (-34.8%)
GRAMMAR: PASSED
PASS 2

LEXICAL

Content-aware removal & compression. Available in Balanced and Aggressive modes.

COMMAND STARTS

Removes instructional preambles at sentence boundaries: "please note that", "kindly note that", "i just want to say that", "let me just say that", etc.

HEDGE PHRASES

Multi-word hedges: "you know", "i mean", "sort of", "kind of", "i guess", "if that makes sense", "to be honest", "at the end of the day", "for what it's worth".

SINGLE-WORD FILLERS

Context-aware removal: "um", "uh", "basically", "literally", "actually", "really", "very", "just", "simply", "totally", "obviously", "clearly", "essentially", "rather" (guarded), etc.

PHRASE COMPRESSION

50+ mappings: "in order to"→"to", "due to the fact that"→"because", "it is important to note that"→"", "make sure that"→"ensure", "take into consideration"→"consider", etc.

SYNONYM SWAPS (AGGRESSIVE)

"utilize"→"use", "facilitate"→"help", "implement"→"do", "commence"→"start", "subsequently"→"then", "additional"→"more", "approximately"→"about", "individuals"→"people".

NUMBER WORDS → DIGITS

"twenty minutes"→"20 minutes", "five users"→"5 users". Only when followed by recognized unit. Quote-aware.

GRAMMAR GUARD

Every lexical change is validated. Reverts if result has: dangling prepositions/conjunctions, doubled determiners ("the the"), modal+"to" without verb ("would to propose"), "important to [subject]" without verb, unbalanced quotes/parens.

PASS 3

STRUCTURAL

Semantic deduplication & structural compression. Aggressive mode uses Jaccard similarity; Balanced uses exact n-gram fallback.

JACCARD

Semantic Clause Deduplication

Splits text into clauses, extracts content words (stemmed + synonym-canonicalized), computes Jaccard similarity. Restatement markers ("to confirm", "in other words", "to reiterate") trigger relaxed threshold (0.25) against immediate predecessor. General threshold: 0.55.

N-GRAM

Exact Repeat Fallback

5-gram exact match detection for copy-paste repeats. Runs in Balanced mode. Prevents overlapping span conflicts with semantic pass.

NL

Newline Collapsing

Reduces 3+ consecutive newlines to double newline. Token-saving for verbose formatted prompts.

$ input: "We need to ship fast. To confirm, we need to ship fast. In other words, shipping fast is required."
$ output: "We need to ship fast."
$ changelog: semantic_dedup: 2 clauses collapsed
$ cutoken structural --mode=aggressive 
✓ Clauses detected: 12
✓ Restatement markers: 3
✓ Jaccard pairs evaluated: 22
✓ Clauses collapsed: 4
✓ Newlines collapsed: 3→2

TOKENS: 1,247 → 982 (-21.2%)
CONFIDENCE: 0.94
$ cutoken --template=list

16 OUTPUT TEMPLATES

Applied after optimization. Structural templates (role_context_task) restructure; others prepend directives.

CONCISE

concise

No wrapper. Pure compressed output.

STRUCT

structured

"Respond in short bullet points."

ROLE

role_based

"Expert answer." prefix.

STEPS

step_by_step

"Numbered steps." prefix.

COMPARE

comparison

"Compare options for: {prompt}"

QUERY

query

Clean wording only. No prefix.

TUTOR

tutorial

"Tutorial style." prefix.

CHAT

chatbot

"Reply conversationally." prefix. Preserves greetings.

ZERO

zero_shot

Compact zero-shot task format.

FEW

few_shot

"From examples below, complete:"

SUM

summarization

"Summarize concisely." prefix.

TRANS

translation

"Translate accurately." prefix.

CODE

code_gen

"Write clean code.\nTask: {prompt}"

DATA

data_extraction

"Extract: {prompt}\nReturn structured."

RAG

rag_context

"Use context above for: {prompt}"

SYS

system_prompt

"System: {prompt}\nRespond accordingly."

RCT

role_context_task

Classifies into Role / Context / Task sections. Structural restructure.

SUPPORTED MODELS

Token counting uses model-specific encodings (cl100k_base / o200k_base approximations).

OPENAI

  • GPT-4o (o200k_base)
  • GPT-4 Turbo (cl100k_base)
  • GPT-4 (cl100k_base)
  • GPT-3.5 Turbo (cl100k_base)
  • o1-preview / o1-mini (o200k_base)

ANTHROPIC

  • Claude 3.5 Sonnet
  • Claude 3 Opus
  • Claude 3 Haiku
  • Claude 4 Sonnet (o200k_base)

OTHERS

  • Grok-1 (cl100k_base)
  • Llama 3 70B / 8B
  • Mistral Large
  • Gemini Pro

OPTIMIZATION MODES

SAFE LOW RISK

Safe

Normalize pass only. Whitespace, unicode, punctuation. Zero semantic risk.

  • concise, structured, role_based
  • step_by_step, comparison, query
  • summarization, translation, custom
BALANCED MEDIUM RISK

Balanced DEFAULT

Normalize + Lexical. Filler/hedge/command removal, phrase compression, greeting stripping.

  • All safe templates +
  • tutorial, chatbot, zero_shot
  • few_shot
AGGRESSIVE HIGH RISK

Aggressive

All passes. Semantic clause dedup, synonym swaps, number→digit, structural compression.

  • ALL 16 templates available
  • Including code_gen, data_extraction
  • rag_context, system_prompt
  • role_context_task
$ pip install cutoken

TERMINAL CLI & DEVELOPER TOOLS

Use CuToken directly from your terminal, scripts, and workflows.

DEVELOPER TOOL

CuToken for Terminal

A developer-grade CLI inspired by Claude Code. Compress prompts with one command, pipe files, or run an interactive REPL session with auto-clipboard copy.

CLI

Interactive Mode (REPL)

Launch an interactive session with live token stats, intensity switching, and history.

PIPE

Unix Piping & Scripting

Stream file contents or git diffs directly through CuToken: cat prompt.txt | cutoken.

CLIP

Automatic Clipboard Sync

Results are automatically copied to your system clipboard ready to paste into any model.

API

Unified Credits

Uses the same API key and shares the exact same Pro credit balance across CLI, Extension, and Web.

$ cutoken "Rewrite this prompt to be punchy" 
✔ Optimized in 340ms
─────────────────────────────────────────────────────────────
 Original: 38 tok  Optimized: 18 tok  Saved: -20 (52.6%)
─────────────────────────────────────────────────────────────
📋 Copied to clipboard!

# Install globally:
$ pip install cutoken
$ cutoken login