logoalt Hacker News

Starlevel004 • yesterday at 7:03 PM • 2 replies • view on HN

I cannot say this fills me with confidence about alignment.


Replies

amterp • today at 12:06 AM

Why not? This seems to greatly help with interpretability, which should help with alignment, no?

shwaj • yesterday at 7:40 PM

Me neither, but it's also not reducing my confidence about alignment. I'm curious: does this raise specific concerns for you? Models don't typically have access to their own weights to self-modify. This result might (assuming that it doesn't only hold for a bigger network editing a smaller one) provide an incentive for it to gain unauthorized access to its weights, but by that point it is already highly unaligned.