</think> tokenization in the training set
#8
by
L29Ah
- opened
I would like to discuss LLM thinking with the LLM, so i need a way to tokenize </think> so that it doesn't treat it as its own thought termination and getting confused instead of treating it as a concept seen in the training set. What is the right way to tokenize it?