MarkAssPandi (@MarkAssPandi)

@sodiboo it's actually not because of training data (at least not fully, but it's rather a minor part of it)
it's because since those models are autoregressive, they are affected by exposure bias
since next token prediction is based on previous tokens, repeating pattern increases probability of the next token (or sequence of tokens) being the same, which reinforces that pattern
It's basically pretty big problem with LLMs and autoregresive models in general, and isn't rly caused by training data itself