What is a Token?

A token is a unit of text that AI models read and process. It is not exactly the same as a word — some words may be split into multiple tokens, while punctuation and spaces can also count as tokens. As a rough guide, one word typically equals 1 to 1.5 tokens.

Why Tokens Matter?

All of our AI models operate within token limits. These limits apply to:

• Your input (prompts, documents, questions)
• The model’s response
• The combined total of both

If the total exceeds the model’s capacity, it may:

• Produce incomplete or no outputs
• Hallucinate or generate fabricated/incorrect results

Understanding token usage helps ensure clarity, completeness, and reliability.

What is Chunking?

Chunking is the practice of breaking long text into smaller, manageable sections before sending it to the AI. Instead of submitting one long document, you divide it into logical parts — such as sections, themes, or paragraphs — and process them step by step.

Why Prepare Chunking Before Using The Viyug AI Studio

Preparing chunks in advance allows you to:

• Maintain full context across long analyses (thereby reducing hallucinations and incorrect results)
• Avoid accidental truncation
• Ask more focused and precise questions
• Compare outputs across sections easily

How our Token Planner Tool Helps?

The Token Planner provides a quick estimation of:

• How many tokens your text may consume
• How many safe chunks are recommended for processing

This helps you plan your workflow before engaging the model, saving time and avoiding trial-and-error.

Best Practice for Working with Viyug AI Studio

• Use the Token Planner to assess length
• Split long inputs into logical chunks
• Process one chunk at a time when needed
• Synthesise insights across outputs using your own judgment

Chunking is an AI discipline which once you practice you don’t require tokenisation tool. This discipline is also helpful for getting better results when you are working with public facing AI tools.