In 1851 architect Calvert Vaux and landscape designer Andrew Jackson Downing collaborated to create the original dwelling for New York dry-goods merchant Daniel Parish. William Backhouse Astor, Jr., grandson of John Jacob Astor, purchased the mansion in 1881, and he and his wife Caroline presided there for the next two decades, expanding and renovating it to set a new standard for their socially prominent friends and family.
Caroline Astor, or “The Mrs. Astor,” as she insisted upon being called, brought in Richard Morris Hunt to add the glittering ballroom, with mirrored walls, gilt ornamentation, crystal chandeliers, and French doors that open to expansive lawns and the shimmering sea beyond. Here the upper crust of New York society gathered for the celebrated Summer Ball. Mrs. Astor retired from social life in 1906 and lived quietly until her death in 1908.