James Victor Scott was born on July 17, 1925, in Cleveland, Ohio, the third of ten children of Arthur Claude Scott and Justine Hazel Stanard Scott. He first sang beside his mother at the family piano and in a church choir, and he later told the National Endowment for the Arts that he learned songs from the radio, where Bing Crosby and the Tommy Dorsey band were everywhere, and from lyric books and sheet music bought for a few cents. His mother was killed by a drunk driver when he was 13. He had Kallmann syndrome, a rare genetic condition that kept him from going through puberty, so his voice never dropped; he stood 4 feet 11 inches until he was 37, when he grew about eight inches. The high contralto the condition left him with became the most recognizable thing about his singing.

In 1949 Lionel Hampton hired him as a featured vocalist and billed him as Little Jimmy Scott, a name that stayed with him for life. With Hampton's orchestra he recorded "Everybody's Somebody's Fool" in December 1949, and it became a top-10 R&B hit in 1950, although the label credited "Lionel Hampton and vocalists" and Scott received no credit on any of the songs. Credit was missed again on One Night in Birdland, the 1977 release of a 1950 Birdland set by Charlie Parker, where his vocal on "Embraceable You" was attributed to another singer, Chubby Newsom. In the early 1950s he made singles for Roost and Coral and recorded a live session in New Orleans for Regal Records, and by his own account a one-week booking at Harlem's Baby Grand stretched to a month and began a lasting friendship with the songwriter Doc Pomus. He sang "Why Was I Born?" at President Eisenhower's inauguration in 1953.

His recording career then ran through Savoy Records, whose owner Herman Lubinsky held him to a contract that Scott later disputed. Very Truly Yours (1955), which included his hit "When Did You Leave Heaven," and If You Only Knew (1956) were followed by The Fabulous Songs of Jimmy Scott (1960), and Lubinsky also loaned him to Syd Nathan's King Records for 45 recordings in 1957 and 1958. Scott's girlfriend Mary Ann Fisher, who sang with Ray Charles, helped him sign with Charles's Tangerine label, and he recorded Falling in Love Is Wonderful in Hollywood with arrangements by Gerald Wilson and Marty Paich, Joe Adams producing, and Charles supervising and playing piano. The album came out in 1963 and was pulled from sale while Scott was on his honeymoon, because of the Lubinsky contract; it stayed unavailable for about four decades until Rhino Handmade reissued it in 2002. The Source, recorded for Atlantic in March 1969 with Joel Dorn producing, Arif Mardin arranging, and liner notes by Nancy Wilson, was released in 1970 and then left unpromoted after another breach-of-contract threat from Lubinsky. By the late 1960s his career had stalled, and he returned to Cleveland, where he worked as a hospital orderly, a shipping clerk, and an elevator operator.

He returned to music in 1989, when his manager Alan Eichler booked him on a late-night bill with Johnnie Ray at New York's Ballroom, and in August 1990 he recorded Doesn't Love Mean More with his band the Jazz Expressions, a CD sold at his performances. The turning point came when he sang at the funeral of Doc Pomus in 1991; Seymour Stein, who founded Sire Records, was in the audience and signed him. That year he sang "Sycamore Trees" in the final episode of the original Twin Peaks (the song, composed by Angelo Badalamenti with words by David Lynch, was later issued on the Fire Walk with Me soundtrack), and Lou Reed had him sing backing vocals on "Power and Glory" on Magic and Loss (1992). All the Way (1992), produced by Tommy LiPuma with Kenny Barron, Ron Carter, drummer Grady Tate, and saxophonist David "Fathead" Newman, earned a Grammy nomination; it sold only about 49,000 copies in the United States, according to the Guardian, but won him a devoted audience in Europe and Japan. Dream (1994) was produced by Mitchell Froom and featured Milt Jackson on vibraphone, Heaven (1996) was a set of spirituals and hymns produced by Craig Street, and Holding Back the Years (1998) took on songs by Prince, John Lennon, Elvis Costello, Elton John, and Simply Red, with liner notes by Lou Reed. It peaked at No. 14 on Billboard's jazz albums chart and won the Swing Journal award for best jazz album in Japan for 2000. After seeing him perform in 1994, Madonna told the New York Times that he was the only singer who ever made her cry.

In 2000 he signed with Milestone Records, which issued four albums produced by Todd Barkan: Mood Indigo (2000), Over the Rainbow (2001), But Beautiful (2002), and Moon Glow (2003). The guests across the four included Wynton Marsalis, Renee Rosnes, George Mraz, Lewis Nash, Eric Alexander, Lew Soloff, Cyrus Chestnut, and Hank Crawford, and two live albums recorded in Japan followed. The PBS documentary If You Only Knew aired in 2003 and 2004, and David Ritz published his biography Faith in Time in 2002. Scott received the NEA Jazz Masters award in 2007, a Living Legend Award from the Kennedy Center, and a Lifetime Achievement Award from the Jazz Foundation of America in 2010; he sang at President Clinton's inauguration in 1993 and was inducted into the R&B Music Hall of Fame in 2013. He and his wife, Jeanie, bought a house in Las Vegas in 2006 after about ten years in Euclid, Ohio. His last recording session took place on May 10, 2014, in the living room of his home, for a song written for him, "The 26th of May," on Grégoire Maret's album Wanted. He passed away in his sleep at his home in Las Vegas on June 12, 2014, at 88, and was buried at Knollwood Cemetery in Mayfield Heights, Ohio. A stretch of East 101st Street in Cleveland was renamed Jimmy Scott Way the following month, and the duets album I Go Back Home, with guests including Kenny Barron, Dee Dee Bridgewater, and James Moody, was released in 2016.

Scott's natural contralto, a result of Kallmann syndrome, sounded at once very young and very old, and he sang it unhurriedly: he drew lyrics out, often well behind the beat, and gave ballads and love songs a stillness that the NEA described as emotionally penetrating. He said he learned from the radio singers of the big-band era, and that the years in Lionel Hampton's orchestra taught him to accept large audiences. The NEA reports that Billie Holiday named him as the one vocalist she admired, and Ray Charles, Dinah Washington, and Nancy Wilson are among the artists reported to have favored his phrasing; Wilson, who wrote the liner notes for The Source, told the NEA in 2007, according to her own profile, that Scott was the source of her sound. His repertoire ran from standards and the R&B hit of his Hampton years to spirituals, and in the 1990s to songs by Prince, John Lennon, and Elvis Costello; on Over the Rainbow (2001) he sang "Strange Fruit," a song closely tied to Holiday. His career was shaped as much by contract disputes as by taste: it faltered in the 1960s, and he was largely out of public view until the revival of the 1990s, which is the part of his story most listeners know first.