If I asked you to separate helloworld into meaningful words, you would immediately jump to hello and world before your mind could even consider that the first word might be he or hell. You’ve just created a token boundary using only a pre-existing knowledge of the language. In other words, the token boundary wasn’t encoded in the source material — you used outside information to infer one. This isn’t how parsers work. ...
