James was preceded in death by his parents, James Alfred Noel II and Mattie Webb Noel; his first wife, D. J. Noel and daughter, Rebekah Ruth McGahey.
He is survived by his wife, Dr. Rebekah Noel; daughter, Cynthia McGahey Trowell; two sons, Jay Miller McGahey and Andrew Swanner McGahey; grandchildren, Ashley, Cole, Lindsey and Britty Trowell; and Eric and Alyssa Noelle McGahey; and cousins.