George was born in Great Neck, New York, United States. He is the youngest of four children; his oldest brother, John, worked in the hops brokerage business and was an innovator in the cultivation of new hop varieties.
Segal's family was Jewish, but he was raised in a secular household. His paternal great-grandfather ran for governor of Massachusetts as a socialist.
Segal served in the United States Army. While there, he also played in a band, which was called Corporal Bruno's Sad Sack Six.
Segal became popular in the 1960s and 1970s for playing both dramatic and comedic roles. Some of his most acclaimed roles are in films such as Ship of Fools (1965), King Rat (1965), Who's Afraid of Virginia Woolf? (1966), The St. Valentine's Day Massacre (1967), Where's Poppa? (1970), The Hot Rock (1972), Blume in Love (1973), A Touch of Class (1973), California Split (1974), For the Boys (1991), and Flirting with Disaster (1996). He was one of the first American film actors to rise to leading man status with an unchanged Jewish surname—thus paving the way for Dustin Hoffman and Barbra Streisand.