Free speech in America is specifically about protecting you against the government. Your neighbor is still not allowed to defame you.