James Madison, Sr. American planter and politician (1723–1801) Parents Frances Taylor Ambrose Madison Children James Madison William Madison Nelly Conway Madison Francis Madison Elizabeth Madison Frances Taylor Madison Rose Catlett Madison unknown son Madison Ambrose Madison Reuben Madison Sarah Catlett Madison Siblings Elizabeth [Willis] Beale Partners Eleanor Rose Conway