Rasmus It looks like their latest software update introduced a bug on Youtube App. Could you please try to unplug and replug your Apple TV and let us know if that helps?
If not, I would recommend adding your Apple TV as an IR device (category Set-Top boxes & media streamers, brand Apple, Model AppleTV4 for example) to avoid this issue. We have contacted Apple on this and are waiting for their answer for when they will be able to fix this issue since it is on their side.
Rasmus Lerche the Smart Remote is managed as a keyboard by the Apple TV. This issue appeared since the latest Apple TV's update and in the same for all third party devices managed as keyboards by Apple TV. We have reached out to Apple about this issue and hope this is going to be fixed soon. In the meantime, I would recommend trying to add your Apple TV as an infrared device on Smart Remote, that will remove this issue.