The Catholic Church has excommunicated a Brazilian priest after he defended homosexuality, open marriage and other practices counter to Church teaching in online videos.