alt Hacker NewsJust reading the first paragraph and I've already started to experience it when attempting to apply AI to Acceptance Criteria that testers have to test against.
The software is necessarily complex due to legislative requirements, and the corpus of documentation the AI has access to just doesn't seem to capture the complexities and subtleties of the system and its related platforms.
I can churn out ACs quicker, but if I just move on to the next thing as if they're 'done' then quality is going to decline sharply. I'm currently entirely re-writing the first set of ACs it generated because the base premise was off.
This is both a prompt engineering and an availability-of-enough-context documentation problem, but both of those have fairly long learning curve work. Not many places do knowledge management very well, and so the requisite base information just may not be complete enough, and one missing 'patch' can very much change a lot of contexts.
I work with Australian tax - lots of regulatory complexity, add the documentation often assumes the reader is a CPA. I've got decent results by telling the chat bot to ask questions instead of making assumptions, and then grilling it to find edge cases.
I did a live demo in front of the CPAs, using their documentation, and Claude asked clarification questions they hadn't thought of and exposed gaps in the old manual processes.