Lillian was born in Jasper, Florida, to Calvin and Annie Smith, native Georgians. Well educated but of modest means, she gave up a career in music and her missionary work in China to run Laurel Falls summer camp owned by her father in north Georgia when he became ill. Smith used this opportunity for writing and social justice work, primarily for civil rights for African-Americans, by publishing journals and expanding the goals of her camp to include encouraging young white women to imagine the future of the South and their place in it.