Daisy Moore (through E. Howes) mentions that sometime after their 4th child was born, Mary took the oldest and the youngest and left Fred, and went to either Scotland or South Africa, where one of her brothers lived.