alt Hacker NewsThis just can't be sustainable. Ever increasing agent activity can only move to the point where it's literally impossible for humans to have a collective awareness of how the systems that support our lives actually work. How does anyone think this is going to end well? If as humans we loose control of vital systems, when they stop working one day (and they so easily could unless the agents are perfect and 100pct aligned with the general good of humanity (how do we even define that)) we could find ourselves without the ability to run healthcare systems and power grids, that have evolved so much such complexity that no human can intervene to fix them. I do believe that you can create feed back loops where probabilistic agents can be used to create deterministic checks and balances, and solve real problems for good. But it's a wild west at the moment and that is not what's happening in so many companies, and the solutions are not yet on the horizon. Even if the 'agents can handle it' ... where does that leave us? Unable to run our societies without puting everything through frontier LLMs? I just finished a very short story by Ted Chiang - 'The evolution of Human science' it starts at the point where it's 25 years since humans contributed a scientific paper because 'meta humans' now handle all research, and communicate digitally such that humans can only get a low fidelity report of what the agents actually know, and the endeavour to understand has become like a best effort archeological like hobby for interested humans. The story assumes that the 'meta humans' will always act in humans best interests. I believe it's from 2002, it's incredibly precient (The stories of your life and others is the collection the story is in. Many of them are very relevant to modern AI actually).
Replies
No one's at the wheel. It's fair (to a degree) for companies to push for short term gain, but the legislative counterbalance that is supposed to care for long term effects no longer exists.
It's not only AI. Betting is on the rise, adults and minors are being hooked by dopamine loops, social networks are causing well measured negative effects, children's educational scores are sinking worldwide, appliance manufacturers engage in surveillance...
It truly feels like the wild west.
[dead]
The agents only have to be better than we are and we’re not very good. Same argument as self-driving.
Don’t fear change.
> Ever increasing agent activity can only move to the point where it's literally impossible for humans to have a collective awareness of how the systems that support our lives actually work
That's the point, they are setting up the economy to be run by agents and they trust that it is possible. Step 1. Teach AI human knowledge (2022-2023). Step 2. Make AI agentic (2025). Step 3. Start autonomous research and create own corpus of knowledge (2026).
Step 3 will keep compounding (as we are witnessing) and most people will be left behind. I'm not against a revolution, I'm all for it.
> Ever increasing agent activity can only move to the point where it's literally impossible for humans to have a collective awareness of how the systems that support our lives actually work.
fwiw, i think literally all largish corporations have been like that pre AI too? i can gurantee you that nobody understands how the whole system works at all of my last workplaces either...
people just know their small part and hope for the best - so from a devils advocate pov, its already been proved that its sustasinable. now wherever its _worth it_ is another question. large employers are - on average - incredibly wasteful with their resources. not to anthropics degree, but still.
if youre boostrapping, and VC token subventions go away... i suspect it'd be not worth it to develop like that. But i guess we will find out in a few month to 1-2 years