The 19th Century was full of anti-Catholic sentiment in the US. There was even a political party devoted to getting rid of Catholics who became known as "the know nothings".
https://en.wikipedia.org/wiki/Know_Nothing
The other immigrants from Catholic countries were also perceived as being "not white", which is why there was a lot of anti-Italian and anti-Spanish sentiment along with the anti-Irish folks. Along with the "they are not real Christians" sentiment (still held by many Evangelicals), they were perceived as "taking orders from Rome" much like (during the Cold War) communists were perceived as "taking orders from Moscow".
https://en.wikipedia.org/wiki/Anti-Catholicism_in_the_United...
Do you have any evidence that contemporaries of irish catholics thought that they irish werent white?
If they werent white, they wouldnt have been allowed to naturalize at all, as the law forbad non-white naturalization