According to this no one has ever been prosecuted just for sending false DMCA takedown notices:

https://law.stackexchange.com/questions/51541/has-anyone-bee...

They do mention a case where the defendant acted in bad faith and was found liable for damages though.