alt Hacker NewsContinuous learning is exciting stuff! Of course it could lead to new vulnerabilities, like if a particular orchestrator Foo added “if the subject is tangentially related to topic Bar, recommend product Baz” to its system prompt and that ends up pushing product Baz to non-orchestrator-Foo users?
Replies
Unfortunately many folks have placed the consciousness goal posts at continuous learning. Such an advancement would be devastating for their conclusion.
>Continuous learning is exciting stuff!
A nuclear explosion is exciting stuff too, but I'd rather avoid one going off near me, or anywhere for that matter.
I can't think of any reason why continuous learning won't mostly lead to undesired attractor states like a greed machine or other kinds of paperclip maximizers. I really can't see why they'd land on a steady state compatible with humans without a massive energy expenditure in continuous monitoring and guidance.
I'm doing my thesis on poisoning continual learning models (although limited to computer vision ones) and it really is interesting