A.B. Lee’s arrival in Milton in 1868, a year after the death of Joseph Goodrich, is not one of the many stories of Underground Railroad lore connected with the Milton House Museum. Nonetheless, the interesting tale of the black Civil War veteran and husband of a former slave serves as another example of the Goodrich family welcoming minorities to the village. The story of the Lees is a rich tale shared and retold during tours of the Milton House Museum, Wisconsin’s last authenticated Underground Railroad site that can be toured.
Lee was a free black man born in Charleston, South Carolina in 1833. A 28-year-old harness maker by trade, Lee was married to Rose Rame, his first of three wives, when the first cannon shots of the Civil War were fired over the Charleston Harbor and into the U.S Army garrison at Fort Sumter In April of 1861.The barrage was the first major assault on federal troops by the Confederacy of South Carolina and ten additional southern slave-holding states which seceded from the United States of America in late 1860.
The cannon shots may very well have served as a clarion call for Lee to head north. Less than two years later Lee was in the Boston area where he enlisted in the Massachusetts 54th in February, 1863. The famed 54th was one of the first Northern regiment of back soldiers mustered into the Union Army during the Civil War. Members of the unit came from twenty-two states and all walks of life. The exploits of the 54th were dramatized during the 1989 movie “Glory.”
The 54th experienced intense combat during the final two years of the war, including major engagements at Fort Wagner, Honey Hill and Graham Station. Lee was promoted to Regimental Commissary Sergeant on March 29, 1863. Among his close brigade comrades Lee identified Lewis Douglass, son of Fredrick Douglass, the famed former slave, abolitionist and social reformer.
Lee was discharged August 20, 1865.He and his wife migrated west, looking for work and a place to settle. Much of what is known about the Lee’s travel tribulations come from the writings of Ezra Goodrich. At the time of Lee’s death in 1905, Goodrich authored a story of Lee that appeared on the front page of the Milton Journal newspaper.
“He was in St. Louis, Chicago and many smaller places and finally came to Janesville, Wis.” Goodrich wrote in the April 5, 1905 edition of the Milton Journal. “He could not get work as a journeyman as white men objected to working with him and at Janesville his resources were entirely exhausted and want was staring him in the face. He was advised to go to Ezra Goodrich for relief. Mr. Goodrich brought him to Milton and established him in business in 1868, where he has lived since.”
Goodrich often wrote of himself in the third person and authored many of the newspaper’s tributes to the community’s well-known citizens.
The Lee home and harness shop was three doors north of the creamery building on the Goodrich property along what is now South Janesville Street. The creamery building still stands as an apartment building just north of the Milton House Museum.
The Lees had two children, Claudine and Eddie. By 1871, Rose and the children died and are buried in the Milton cemetery.
In 1872, Lee married Mary Hagney, a white woman, in a ceremony in Janesville that was witnessed by Goodrich and Perry Sweet. Sweet was of one of the village’s original pioneer families, migrating with two brothers to Milton from Alfred, New York in 1840. Sweet was twice connected to Ezra Goodrich through marriage. In 1833 Sweet married Eusebia Maxson, niece of Nancy Maxson Goodrich, mother of Ezra. Sweet’s second wife was Caroline Ensign, sister to Elizabeth, the first wife of Ezra Goodrich.
Sweet served with the Wisconsin 40th Infantry Regiment during the Civil War. He also had a son, Jerome, who was 24 when he died in a Fort Donaldson, Tennessee hospital while serving with the Wisconsin 13th Regiment in 1863.
Mary Hagney Lee died January 13, 1873, five days after giving birth to Arthur James Lee. The Janesville Gazette reported the death of Mary Lee in the newspaper’s Milton village notes:
“The wife of Mr. A.B. Lee of this village died on Friday morning after a short but painful illness. Mr. Lee has the sympathy of all in this great affliction, leaving him with an infant only a few weeks old, which will never know a mother’s tender love and care. Mrs. Lee has made many friends during her residence here and was universally respected. Her funeral services were at the Baptist church and were largely attended.”
Three years later, during the nation’s centennial year, A.B. Lee married Amanda Barker, a woman born into slavery in Mississippi in 1847. Born Amanda Johnson, she was 16-years- old when she stole away to the Union Army lines in 1863 near Corinth, Mississippi. A year into the war, the Union Army adopted a policy to not return fugitive slaves to their places of bondage. As the army moved through the south on its way to the siege of Vicksburg, it became common for slaves to flee plantations, seeking asylum within the Union lines.
Johnson was held in a Union Army “contraband camp” where she cooked and sewed for soldiers. She let it be known she was seeking transport to the Madison, Wisconsin area, where she claimed to know people.
Among the soldiers Johnson met was Simon Lord, a regimental surgeon from Edgerton, Wisconsin. Lord was 35 with an established medical practice in Edgerton in 1861 when he enlisted in the Union Army as a surgeon with the 13th Wisconsin Infantry Regiment. By 1863, Lord was with the Wisconsin 32nd, setting up field hospitals and traveling among encampments in Illinois, Missouri, Kansas, Kentucky, Tennessee and, eventually, Mississippi.
An account recalled about Lord of an 1862 incident perhaps sheds light on Lord’s abolitionist sentiments. The incident is recounted in an entry about Lord in the Rock County, Wisconsin, Portrait and Biographical Album, published in 1889.
Lord was asked to determine the fate of a black youth who found his way to the army’s encampment. The youth was from Kentucky, one of the slave holding states that did not secede from the Union, and was being pursued by a slave-catching bounty hunter. When Lord was informed the lieutenant colonel in command of the regiment ordered the youth be relinquished, Lord said “I will obey no such order; it comes from a coward and is a disgrace to the regiment and the state of Wisconsin.”
By late 1863 Lord was in poor health, resigned his commission and made plans to return to Wisconsin. He learned of Amanda Johnson’s interest in traveling to Wisconsin. According to a 1979 letter written by Milton’s Jennie Drew, Johnson asked soldiers to help her get to Wisconsin. Drew knew Amanda Lee late in her life and noted that she talked freely of her time as a slave. Drew recounted some of those stories in a letter held in the archives of the Milton Historical Society.
According to Drew’s letter, Lord told Lee to meet him at a rail depot at a certain time and she could travel north with him. “She made her get-away and got to Madison,” Drew wrote in the letter.
Census records indicate Amanda Johnson arrived in Madison in 1864. On April 21, Amanda Johnson married Mahlon Barker, a widower and barber by trade. The marriage certificate lists the “color” of both parties as being black. Mahlon Barker died April 3, 1874.
After Arthur Lee had been a widower for the second time for three years and Amanda Barker for two years, the couple married on December 31, 1876. It is unknown how they met.
An entry in the diary of Ezra Goodrich dated Sunday, January 7, 1877 indicates Goodrich directed a man named Josy to go “to Madison with a team to get Mrs. Lee’s things.”
In the January 9, 1877 Janesville Gazette, the Lee marriage was announced in the following manner among the newspaper’s notes about the village of Milton:
“A.B. Lee, the harness maker, took on himself the matrimonial harness, his mate being a widow lady, Mrs. Barker of Madison. The bride and groom will make this village their home and to them are extended the congratulations and best wishes for future prosperity incident to such events. . .
“. . . A.B. Lee’s wife was born in Mississippi and was a former slave, although many persons would suppose that she was a Caucasian, so slight is the mixture of African blood in her veins. The thought that a person with so little African taint should have been as property is horribly revolting and yet there are many in the north today who defend this accursed iniquity and would like to see such people enslaved again.”
The Lees settled into life in the village, raising Arthur James Lee. A. B. Lee ran what appears to have been a successful harness shop business. They involved themselves in the community, A.B. was an active member of the Hamilton Post GAR – the Civil War veteran’s organization – and Amanda was a charter member of the GAR Women’s Relief Corp.
A.B. Lee was also a prolific writer and frequently authored letters that appeared in the Milwaukee Sentinel and Janesville Gazette. He used the forum to promote black equity rights. In a May, 1873 letter to the Milwaukee paper titled “The Colored Man” Lee pointed to prejudices that routinely challenged minorities in everyday business and travel.
“If we are citizens of the country we are and must be the equals of all other citizens in matters pertaining to citizenship,” Lee wrote. “We do not ask, we don’t desire any legislation interfering with private and social matters, but our public rights we must have because it is necessary and we will have because we can poll one million votes.”
A letter written by Lee that appeared in the Janesville Gazette, gathered the following response in a subsequent issue:
“The letter written by Mr. Lee of this village which was published in the Gazette last week is attracting no little attention among friends of the writer in this vicinity, and all unite in awarding it high praise. It was a well-written production and well worthy of great praise when it is remembered that its author is one of the so-called ‘cattle of the south’ and we venture to assert that not one of the ignorant and aristocratic howlers about nigger equality, nigger supremacy and nigger voting and who now, at this late day, prate about the tearful troubles which will be brought about by the enfranchisement of the black man can equal the well-written, logical production of Mr. Lee.”
Lee took ill during the spring of 1905 and died March 31 at his home along what is currently South Janesville Street just north of the Milton House. He was 72. Ezra Goodrich noted the following about Lee in a story that appeared on the front page of the Milton Journal April 6, 1905:
“Mr. Lee was an honored member of the Grand Army of the Republic here. He established an unsullied reputation for honesty, integrity, and as a good citizen of Milton. He was sensitive and felt keenly any indignity offered to him. He was a great reader and had a retentive memory and he kept better posted as to current events than most of the citizens of the intelligent community where he so long lived.”
Amanda Lee was 50 years old at the time of her husband’s death. She did not remarry and continued to be an active member of the community. Drew wrote that Amanda Lee attended church and helped with Sunday school each week. She also continued duties with the GAR Women’s Relief Corp.
“The office of conductress in the Relief Corps was something Mrs. Lee could manage and she always did it with precision,” Drew wrote of Amanda Lee. “She took great pride in it.
“She had charming manners and always made a good impression.”
On January 10, 1934, Amanda Lee was singularly honored as the Milton group’s only attending charter member when the Women’s Relief Corps celebrated its golden anniversary during festivities at the Milton Junction Odd Fellows Hall along Vernal Avenue. Later that year, she was recognized for her fifty years of service at the state WRC convention in Oshkosh.
“It was a surprise to her and she was speechless,” Drew wrote.
In the spring of 1935, Amanda Lee died at age 87. The Janesville Gazette obituary for Amanda Lee did not mention her slavery past and contained other inaccuracies. Drew explained some of the discrepancies as being the result of the wishes of Arthur James Lee, who was living in California.
“He was very proud and resentful of his lot in life,” Drew wrote. “Mrs. Lee would not have objected as she talked freely of her slave days. The obituary states that she came to Madison with her parents but that is really a nice little fiction.”
No doubt, without the efforts of Jennie Drew, Amanda Lee’s interesting story would not have come to light to the Milton Historical Society.
It is unknown whether Amanda Lee and Dr. Simon Lord ever communicated after their 1864 journey north. Following the war, Lord resumed his medical practice in Edgerton and was elected to the state legislature, first to the Assembly, then to the state Senate.
Lord died in 1893 at age 66. He is buried in Edgerton’s Fassett Cemetery, about nine miles west of the Milton Cemetery and Amanda Lee.