Johnny Hartman was born John Maurice Hartman on July 3, 1923, in Houma, Louisiana, and was brought to Chicago as an infant when his family joined the Great Migration north. Chicago, where he grew up, is often given as his birthplace, but his biographer Gregg Akkerman places the birth in Houma. He sang in a Baptist church as a boy (Dinah Washington belonged to the same congregation), studied music at DuSable High School under the school's music director Walter Dyett, and in 1940 enrolled at the Chicago Music College of Roosevelt University to work on diction, pitch and vocal production. He was drafted in 1943 and spent the war singing for troops in the Army's Special Services.

Back in Chicago in 1946 he worked South Side clubs and won a local talent contest. His first record date came in February 1947, for the Sunbeam label with a string group led by Marl Young. That year he also sang with the Earl Hines orchestra, recording with it in Chicago in November 1947, and cut sides for the Regent label in New York. In 1948 he joined the big band of Dizzy Gillespie, broadcasting with it from the Royal Roost in New York in October 1948 and recording for RCA Victor in 1949. In August 1949 he made Mercury singles with the trio of pianist Erroll Garner, and by early 1950 he was working as a solo singer. Through the early 1950s he recorded singles for RCA Victor (with the orchestras of Norman Leyden, Pérez Prado and Howard Biggs), Jubilee and other small labels, and returned to Hines for a King session in August 1953, but none of it produced a hit.

His first album under his own name, Songs from the Heart, was recorded for Bethlehem around October 1955 and released in 1956, with a quartet led by trumpeter Howard McGhee and including pianist Ralph Sharon. All of Me: The Debonair Mr. Hartman followed in 1957, recorded in November 1956 with an Ernie Wilkins group that included pianist Hank Jones and bassist Milt Hinton, plus a string orchestra conducted by Frank Hunter. A third album, And I Thought About You, came out on Roost in 1959. In 1961 he sang with pianist Andrew Hill's trio in St. Louis; four of those performances were issued in 1987.

The break came in 1963. John Coltrane sought Hartman out for a ballad album, and Hartman, unsure the saxophonist would suit his kind of material, went to hear him play ballads at Birdland and came away convinced. With McCoy Tyner at the piano, Jimmy Garrison on bass and Elvin Jones on drums, they recorded John Coltrane and Johnny Hartman on March 7, 1963, at Rudy Van Gelder's studio in Englewood Cliffs, New Jersey, for producer Bob Thiele at Impulse! Records. It runs six songs and about thirty-one minutes, and it is the only album Coltrane made with a singer. It was inducted into the Grammy Hall of Fame in 2013. Impulse! then recorded Hartman as a leader: I Just Dropped by to Say Hello (recorded October 9 and 17, 1963, with tenor saxophonist Illinois Jacquet, guitarists Kenny Burrell and Jim Hall, Hank Jones, Hinton and Elvin Jones) and The Voice That Is! (September 1964). Two further albums for ABC Records, Unforgettable Songs (arranged by Gerald Wilson, February 1966) and I Love Everybody (arranged by Oliver Nelson, 1966), completed a run of four albums after the Coltrane date, all produced by Thiele.

Changing tastes in the later 1960s cut into the work available to a ballad singer, and through the 1970s Hartman played cocktail lounges and clubs in New York and Chicago, toured abroad, and recorded for small labels such as Perception (Today, 1972, with saxophonist George Coleman and drummer Billy Higgins; I've Been There, 1973, with saxophonist Jimmy Heath) and Musicor (1977). In Tokyo in November and December 1972 he recorded two albums for Capitol, Hartman Meets Hino with trumpeter Terumasa Hino and Hartman Sings Trane's Favorites, a tribute to Coltrane, who had died in 1967. On October 13, 1977, he was recorded live at the Sometime club in Kichijoji, Tokyo, with pianist Roland Hanna and bassist George Mraz. He also did a television special in Australia.

He returned to a small jazz group for Once in Every Life, recorded on August 11, 1980, at Master Sound in Franklin Square, New York, with pianist Billy Taylor, Frank Wess on tenor saxophone and flute, trumpeter Joe Wilder, guitarist Al Gafa, bassist Victor Gaskin and drummer Keith Copeland. It earned him a nomination for Best Jazz Vocal Performance, Male, at the 24th Annual Grammy Awards in 1982, the only Grammy nomination of his career; Al Jarreau won. Twelve days after that session he recorded This One's for Tedi in Hamilton, Ontario, a tribute to his wife, Theodora ("Tedi"). Hartman died of lung cancer on September 15, 1983, at Memorial Sloan Kettering Cancer Center in New York City, at the age of sixty. In 1984 the New York City Council named Johnny Hartman Plaza, at Amsterdam Avenue, Hamilton Place and West 143rd Street in Hamilton Heights. In 1995 Clint Eastwood used four of the Once in Every Life recordings in the film The Bridges of Madison County, at a time when the album was out of print. Akkerman's biography The Last Balladeer: The Johnny Hartman Story appeared in 2012.

Hartman was a baritone with a warm, unhurried, dark-toned delivery, and the ballad was his home ground. He described a lyric as a story, close to speaking, that the singer has to make believable, and his phrasing keeps close to the words and lets the tempo breathe; he could swing, as on "The Birth of the Blues" and "All of Me," but the work that lasts is slow. Coltrane said he chose him on the strength of his sound, and the 1963 pairing is regularly named among the best vocal-and-saxophone ballad records, with "Lush Life," "My One and Only Love" and "They Say It's Wonderful" often cited as definitive readings. His commercial standing was modest in his lifetime, and the reappraisal that began after his death has rested largely on that album and on Once in Every Life. Tubist Bob Stewart has said Hartman's range matched the tuba's and taught him how to shape a melody.