Frederick Baily was born a slave in February 1818 on Holmes Hill Farm, near the town of Easton on Maryland's Eastern Shore. The farm was part of an estate owned by Aaron Anthony, who also managed the plantations of Edward Lloyd V, one of the wealthiest men in Maryland. The main Lloyd Plantation was near the eastern side of Chesapeake Bay, 12 miles from Holmes Hill Farm, in a home Anthony had built near the Lloyd mansion, was where Frederick's first master lived.

Frederick's mother, Harriet Baily, worked the cornfields surrounding Holmes Hill. He knew little of his father except that the man was white. As a child, he had heard rumors that the master, Aaron Anthony, had sired him. Because Harriet Baily was required to work long hours in the fields, Frederick had been sent to live with his grandmother, Betsey Baily. Betsy Baily lived in a cabin a short distance from Holmes Hill Farm. Her job was to look after Harriet's children until they were old enough to work. Frederick's mother visited him when she could, but he had only a hazy memory of her. He spent his childhood playing in the woods near his grandmother's cabin. He did not think of himself as a slave during these years. Only gradually did Frederick learn about a person his grandmother would refer to as Old Master and when she spoke of Old Master it was with certain fear.

At age 6, Frederick's grandmother had told him that they were taking a long journey. They set out westward, with Frederick clinging to his grandmother's skirt with fear and uncertainty They had approached a large elegant home, the Lloyd Plantation, where several children were playing on the grounds. Betsy Baily had pointed out 3 children which were his brother Perry, and his sisters Sara and Eliza. His grandmother had told him to join his siblings and he did so reluctantly. After a while one of the children yelled out to Frederick that his grandmother was gone. Frederick fell to the ground and wept, he was about to learn the harsh realities