Chapter 6 questions

[SaulLu]

@dipetkov

I admit that by “strict substring” I understood that a string is not its own substring. But with either definition there is an issue. Either that or I am confused about something obvious to everyone else.

In our case, we consider hug because it is a strict substring of “hugs”. The notion of strict substring is only used here to select the initial tokens for this toy example (in a real use case, we will use a BPE algorithm for example). Then we calculate their frequency of appearance, independently of the fact that they are a strict substring or not.