LLM limits for novel coding
LLMs struggle to write code for original languages or projects outside their training data, despite performing well on familiar codebases.
- LLMs fail to produce functional code when dealing with completely novel projects or languages absent from their training sets
- Models succeed on niche tasks like programming calculator games when existing source examples are supplied or present in training data
- Transformer-based models act as prediction engines, making them reliable primarily for familiar programming domains


