I have read about tokens before, and one of the problems is that the definition of a token is pretty fluid (as far as I can tell) and not only is it inconsistent across platforms, it’s inconsistent within the same platform. So it’s not like character count or word count, where the user could monitor their token use with 100% accuracy.
From @axtremus ’s article:
Tokens represent a unit of output, either in text or image, from an LLM. For the Ask Sage tool, a single token equates to about 3.7 characters, according to documents viewed by WIRED
From OpenAI (https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them)
Spaces, punctuation, and partial words all contribute to token counts. This is how the API internally segments your text before generating a response.
One thing that makes it particularly hard to monitor token usage is that the same amount of text could be interpreted as different amounts of tokens, depending on how it’s entered. At least, that’s according to what I’ve read, and the OpenAi article recommends to “break large text into smaller chunks” if you exceed your token limit.
So what does a “chunk” mean? Does that mean that the same prompt content would be interpreted as fewer tokens if you entered it little by little, rather than all at once? It would seem so, but that doesn’t make sense to me.
