$ cutoken --help

FEATURES

Every feature is designed for one goal: reduce tokens without losing meaning.

PASS 1

NORMALIZE

Zero-semantic-change cleanup. Runs first on every mode.

NFKC

Unicode NFKC Normalization

Composes characters, standardizes compatibility equivalents.

QUOT

Smart Quote Conversion

Converts curly quotes (""''‚„) to straight ASCII equivalents.

DASH

Em/En Dash Handling

Converts — and – to simple hyphens for token efficiency.

PUNCT

Punctuation Cleanup

Collapses ... / !!! / ???, removes decorative lines (--- === ~~).

WS

Whitespace Normalization

CRLF→LF, tabs→spaces, collapse multi-spaces, trim trailing.

                        $
                        input: "Hello\u201cWorld\u201d\u2014this\u2026\u2026\u2026"
                    
                        $
                        output: "Hello\"World\"-this..."

$ cutoken normalize --input prompt.txt

✓ NFKC applied
✓ 4 fancy quotes converted
✓ 2 em-dashes normalized
✓ 6 excess punctuation removed
✓ whitespace collapsed

TOKENS: 142 → 138 (-2.8%)

$ cutoken lexical --mode=balanced

✓ Stutters collapsed: 3
✓ Commands stripped: "please note that"
✓ Hedges removed: 7
✓ Fillers dropped: 12
✓ Greetings stripped: "Hello there"
✓ Phrases compressed: 5
✓ Synonyms swapped: 3
✓ Numbers→digits: 2

TOKENS: 523 → 341 (-34.8%)
GRAMMAR: PASSED

PASS 2

LEXICAL

Content-aware removal & compression. Available in Balanced and Aggressive modes.

COMMAND STARTS

Removes instructional preambles at sentence boundaries: "please note that", "kindly note that", "i just want to say that", "let me just say that", etc.

HEDGE PHRASES

Multi-word hedges: "you know", "i mean", "sort of", "kind of", "i guess", "if that makes sense", "to be honest", "at the end of the day", "for what it's worth".

SINGLE-WORD FILLERS

Context-aware removal: "um", "uh", "basically", "literally", "actually", "really", "very", "just", "simply", "totally", "obviously", "clearly", "essentially", "rather" (guarded), etc.

PHRASE COMPRESSION

50+ mappings: "in order to"→"to", "due to the fact that"→"because", "it is important to note that"→"", "make sure that"→"ensure", "take into consideration"→"consider", etc.

SYNONYM SWAPS (AGGRESSIVE)

"utilize"→"use", "facilitate"→"help", "implement"→"do", "commence"→"start", "subsequently"→"then", "additional"→"more", "approximately"→"about", "individuals"→"people".

NUMBER WORDS → DIGITS

"twenty minutes"→"20 minutes", "five users"→"5 users". Only when followed by recognized unit. Quote-aware.

GRAMMAR GUARD

Every lexical change is validated. Reverts if result has: dangling prepositions/conjunctions, doubled determiners ("the the"), modal+"to" without verb ("would to propose"), "important to [subject]" without verb, unbalanced quotes/parens.

PASS 3

STRUCTURAL

Semantic deduplication & structural compression. Aggressive mode uses Jaccard similarity; Balanced uses exact n-gram fallback.

JACCARD

Semantic Clause Deduplication

Splits text into clauses, extracts content words (stemmed + synonym-canonicalized), computes Jaccard similarity. Restatement markers ("to confirm", "in other words", "to reiterate") trigger relaxed threshold (0.25) against immediate predecessor. General threshold: 0.55.

N-GRAM

Exact Repeat Fallback

5-gram exact match detection for copy-paste repeats. Runs in Balanced mode. Prevents overlapping span conflicts with semantic pass.

NL

Newline Collapsing

Reduces 3+ consecutive newlines to double newline. Token-saving for verbose formatted prompts.

                        $
                        input: "We need to ship fast. To confirm, we need to ship fast. In other words, shipping fast is required."
                    
                        $
                        output: "We need to ship fast."
                    
                        $
                        changelog: semantic_dedup: 2 clauses collapsed

$ cutoken structural --mode=aggressive

✓ Clauses detected: 12
✓ Restatement markers: 3
✓ Jaccard pairs evaluated: 22
✓ Clauses collapsed: 4
✓ Newlines collapsed: 3→2

TOKENS: 1,247 → 982 (-21.2%)
CONFIDENCE: 0.94

$ cutoken --template=list

16 OUTPUT TEMPLATES

Applied after optimization. Structural templates (role_context_task) restructure; others prepend directives.

CONCISE

concise

No wrapper. Pure compressed output.

STRUCT

structured

"Respond in short bullet points."

ROLE

role_based

"Expert answer." prefix.

STEPS

step_by_step

"Numbered steps." prefix.

COMPARE

comparison

"Compare options for: {prompt}"

QUERY

query

Clean wording only. No prefix.

TUTOR

tutorial

"Tutorial style." prefix.

CHAT

chatbot

"Reply conversationally." prefix. Preserves greetings.

ZERO

zero_shot

Compact zero-shot task format.

FEW

few_shot

"From examples below, complete:"

SUM

summarization

"Summarize concisely." prefix.

TRANS

translation

"Translate accurately." prefix.

CODE

code_gen

"Write clean code.\nTask: {prompt}"

DATA

data_extraction

"Extract: {prompt}\nReturn structured."

RAG

rag_context

"Use context above for: {prompt}"

SYS

system_prompt

"System: {prompt}\nRespond accordingly."

RCT

role_context_task

Classifies into Role / Context / Task sections. Structural restructure.

SUPPORTED MODELS

Token counting uses model-specific encodings (cl100k_base / o200k_base approximations).

OPENAI

GPT-4o (o200k_base)
GPT-4 Turbo (cl100k_base)
GPT-4 (cl100k_base)
GPT-3.5 Turbo (cl100k_base)
o1-preview / o1-mini (o200k_base)

ANTHROPIC

Claude 3.5 Sonnet
Claude 3 Opus
Claude 3 Haiku
Claude 4 Sonnet (o200k_base)

OTHERS

Grok-1 (cl100k_base)
Llama 3 70B / 8B
Mistral Large
Gemini Pro

OPTIMIZATION MODES

SAFE LOW RISK

Safe

Normalize pass only. Whitespace, unicode, punctuation. Zero semantic risk.

concise, structured, role_based
step_by_step, comparison, query
summarization, translation, custom

BALANCED MEDIUM RISK

Balanced DEFAULT

Normalize + Lexical. Filler/hedge/command removal, phrase compression, greeting stripping.

All safe templates +
tutorial, chatbot, zero_shot
few_shot

AGGRESSIVE HIGH RISK

Aggressive

All passes. Semantic clause dedup, synonym swaps, number→digit, structural compression.

ALL 16 templates available
Including code_gen, data_extraction
rag_context, system_prompt
role_context_task

$ cutoken api --docs

REST API

Integrate CuToken into your pipeline.

POST /api.php

Optimize a prompt and return full result with stats.

Request

{
  "prompt": "your prompt here",
  "mode": "balanced",
  "template": "concise",
  "model": "gpt-4o"
}

Response

{
  "original": "...",
  "optimized": "...",
  "original_tokens": 123,
  "optimized_tokens": 78,
  "tokens_saved": 45,
  "savings_pct": 36.6,
  "confidence": 0.97,
  "changelog": [["pass_lexical", "filler:basically"]]
}

cURL Example

curl -X POST https://yourdomain.com/api.php \
  -H "Content-Type: application/json" \
  -d '{"prompt":"please note that I need help...","mode":"balanced"}'