alt Hacker NewsI would ask for it to give me one line of a song in another language, broken down into sections, explaining the vocabulary and grammar used in the song, with call out to anything that is non-standard outside of a lyrical or poetic setting.
I know no one wants to hear this from the cursed IP attorney, but this would be enough to show in court that the song lyrics were used in the training set. So depending on the jurisdiction you're being sued in, there's some liability there. This is usually solved by the model labs getting some kind of licensing agreements in place first and then throwing all that in the training set. Alternatively, they could also set up some kind of RAG workflow where the search goes out and finds the lyrics. But they would have to both know that the found lyrics where genuine, and ensure that they don't save any of that chat for training. At scale, neither of those are trivial problems to solve.
Now, how many labs have those agreements in place? Not really sure? But issues such as these are probably why you get silliness like DeepMind models not being licensed for use in the EU for instance.
I didn't really say this in my previous point as it was going to get a bit too detailed about something not quite related to what I was describing, but when models do give me lyrics without using a web search, it has hallucinated every time.
As for searching for the lyrics, I often have to give it the title and the artist to find the song, and sometimes even have to give context of where the song is from, otherwise it'll either find a more popular English song with a similar title or still hallucinate. Luckily I know enough of the language to identify when the song is fully wrong.
No clue how well it would work with popular English songs as I've never tried those.