In Fort Wayne, Halsted led the effort to organize Trinity Parish on 25 May 1844, naming it after Trinity Episcopal Church on Wall Street, his former parish. The church held worship services in the courthouse in the mornings, sharing it with the New School Presbyterians, who worshiped there in the afternoon. With Peter Bailey, he also helped organize a parish library of books with the goal of circulating "Christian truths gratuitously throughout the Parish and to inculcate sound Church doctrine without money and without price."
Halsted failed to win support of the church because of his strict and at times bombastic demeanor. He resigned his rectorship on 1 December 1845 and departed the parish for good in the spring, eventually becoming rector at St. Paul's Mishawaka. Kemper wrote him a letter of advice: "In your purity, devotedness, and orthodoxy, I have perfect confidence. Perhaps at times you are rather abrupt - perhaps your opposition to little things is sometimes too decided. Perhaps by reflection and care you might improve the severity of your manners. I have often thought if it could be greatly impressed upon my mind that my great duty is to win souls to Chirst, my influence would increase and I would never offend."
Halsted left Mishawaka in 1851, and by 1853 had moved south to Holt County, Mississippi.. He helped to found three missions, including Calvary Church at Tchula and St. Mary's at Lexington, Mississippi. He died at Lexington on 13 September 1888.