Jean-Baptiste Frédéric Isidor Thielemans was born on April 29, 1922, in Brussels, where his parents ran a café. He began on a homemade accordion at the age of three and taught himself the harmonica as a teenager, settling on the chromatic model around seventeen. During the German occupation of Belgium he came to jazz through records, first a Louis Armstrong disc he heard in 1940 and then the guitar playing of Django Reinhardt, whose recordings he used to teach himself the guitar while he was a mathematics student. By the end of the war in 1945 he thought of himself as a full-time musician, and the bebop records of Charlie Parker and Dizzy Gillespie that reached Belgium after the liberation became what he later called his prophets.

A trip to New York in 1948 led an agent for Benny Goodman to hear him in a small club, and Goodman invited him to join the band's European tours of 1949 and 1950. He was hired as a guitarist and played harmonica at the London Palladium only because of union restrictions; Goodman was reportedly startled to learn that the tours were the first money Thielemans had ever earned from music. In those years he also made his first record, with the tenor saxophonist Zoot Sims, a fellow member of the Goodman group, and in 1949 he sat in on a Paris jam session with Sidney Bechet, Parker, Miles Davis, and Max Roach. He settled in the United States in the early 1950s, became a U.S. citizen in 1957, and played with Charlie Parker's All-Stars and worked with Miles Davis and Dinah Washington.

From 1953 to 1959 he was a member of the George Shearing Quintet, playing guitar and harmonica and adding whistling to his work there. He made his first album as a leader, The Sound, in 1955, and his second, Man Bites Harmonica!, was recorded for Riverside on December 30, 1957, and January 7, 1958, with Pepper Adams on baritone saxophone; it presented the chromatic harmonica as an improvising instrument on a par with the trumpet, the saxophone, or the piano. Through the 1950s he dominated the miscellaneous-instrument category in Down Beat polls, and from 1959 he toured internationally with his own small group while recording only now and then in the studio.

His best-known composition, "Bluesette," was first recorded by him in 1962, with Thielemans doubling the melody on guitar and whistling in unison. Norman Gimbel later wrote English lyrics, the song earned him a Grammy nomination for best instrumental theme in 1964, and more than a hundred artists have since recorded it. In the 1960s and 1970s he became one of the most called-on soloists in studio work: he played harmonica, guitar, and whistling on a string of albums by Quincy Jones beginning in 1964, appeared on film scores including Midnight Cowboy (1969), The Getaway (1972), The Sugarland Express (1974), and Looking for Mr. Goodbar (1977), whistled in Old Spice commercials, and played the closing theme of Sesame Street, which ran for forty years. He recorded a 1969 album with the Brazilian singer Elis Regina, played harmonica on "Night Game" on Paul Simon's Still Crazy After All These Years (1975), and took part in the Oscar Peterson Big 6 date at the Montreux Jazz Festival in 1975.

In October and November 1978 he joined the pianist Bill Evans's trio for Affinity (released 1979), which he called one of his favorite recordings. He worked with Jaco Pastorius, appeared on Pastorius's Word of Mouth (1981), joined Pastorius's big band as a special guest on its 1982 tour of Japan, recorded with Gillespie at Montreux in 1980, and in 1983 played on Billy Joel's "Leave a Tender Moment Alone." In the 1990s he built albums around themes: The Brasil Project (1992) and its sequel (1993) surrounded his harmonica with Brazilian songwriters and players including Caetano Veloso, Milton Nascimento, and Ivan Lins, and Chez Toots (1998) was a French-flavored set made largely in Paris, with guest vocals from Johnny Mathis, Diana Krall, Dianne Reeves, and Shirley Horn. Later records included a live duo album with Kenny Werner, recorded in Kalmthout, Belgium, in June 2001, and One More for the Road (2006), a set of Harold Arlen songs that Belgium's Octaves de la Musique awards named album of the year.

King Albert II made him a baron for life in 2001, and he held a joint honorary doctorate from the two universities of Brussels. Carnegie Hall gave him an all-star tribute concert in 2006, the National Endowment for the Arts named him a Jazz Master in 2009, and Jazz at Lincoln Center marked his ninetieth birthday in 2012 with Herbie Hancock, Kenny Werner, and others. Health problems led him to announce his retirement on March 12, 2014, though he made one unannounced appearance at the Jazz Middelheim festival in Antwerp that August. He passed away on August 22, 2016, in Braine-l'Alleud at the age of 94, and was buried on August 27 in La Hulpe, where Werner read a personal message of condolence from President Barack Obama to his widow. Google marked his hundredth birthday with a Doodle on April 29, 2022.

Thielemans made the chromatic harmonica a credible jazz solo voice. The historian Ted Gioia judged his chief contribution to be "championing the humble harmonica," and the Penguin Jazz Guide called him the preeminent jazz harmonica player. He played it with bebop phrasing absorbed from Charlie Parker and Dizzy Gillespie and a lyricism he traced to Django Reinhardt, who he said remained one of his main influences, and his tone was warm and singing. The guitar and the whistle gave him two more voices, and the pairing of whistle and guitar in unison, as on "Bluesette," was his signature sound. Quincy Jones called him one of the greatest musicians of our time, and he won Down Beat's miscellaneous-instrument (harmonica) category every year from 1978 to 1996, from 1999 to 2008, and again in 2011 and 2012. According to Stevie Wonder's profile, Thielemans said Wonder's harmonica tone and technique were an influence on his own playing, and later players such as Hendrik Meurkens, who bought his first chromatic harmonica the day after hearing Thielemans, and Yvonnick Prene, who recorded the tribute album Merci Toots (2015), worked directly in the line he established. His work was jazz at its core, with a large share of it done as a film, television, and pop session player, which is why his name appears on so many records that are not jazz records.