James, his wife Jane, and eldest daughter Isabella are in Wilberforce in the 1861 census (although their last name is spelled “Sweny”). He is listed as a farmer. At the time, they were living in a log house, built in 1858.