John Theodore Klemmer was born in Chicago on July 3, 1946. He started on guitar at five, took up alto saxophone at eleven and had moved to tenor by high school, while also studying visual art at the Art Institute of Chicago and attending the National Music Camp at Interlochen. His saxophone and improvisation teacher was the Chicago saxophonist Joe Daley. Early on he played in commercial groups and toured with big bands, and Wikipedia names the road bands of Les Elgart and Woody Herman among them.

Chess Records signed him as he left high school, brought in by the producer Esmond Edwards, and his first record, Involvement (Cadet, 1967), is billed to the John Klemmer Quartets and was produced by Edwards, with Wilbur Campbell on drums, Melvin Jackson on bass, and Sam Thomas on guitar or Jodie Christian on piano depending on the track. Four more albums followed on Cadet and its Cadet Concept imprint: And We Were Lovers (1968), which put standards with strings and brass on one side and straight-ahead originals on the other; Blowin' Gold (1969), co-produced by Marshall Chess and often cited as one of the first jazz-rock fusion albums; and All the Children Cried and Eruptions (both 1970). On Blowin' Gold he began using the Echoplex tape-delay unit, the echo effect that became his trademark. The Chicago players around him in these years included the bassist Cleveland Eaton.

By the end of the decade he had moved to Los Angeles and joined the big band of the trumpeter Don Ellis, where he played tenor saxophone and clarinet on Autumn (Columbia, recorded 1968), tenor saxophone and flute on The New Don Ellis Band Goes Underground (1969) and tenor saxophone on the double LP Don Ellis at Fillmore (1970), which includes two of his own pieces, "Excursion #2" and "Old Man's Tear." He has said in an interview that it was in Ellis's band that he first plugged his horn into an echo unit. In 1969 he was one of the saxophonists in the small group Oliver Nelson took to West Africa on a State Department tour, and he is credited as one of two tenor soloists on the opening track of Nelson's Black, Brown and Beautiful (Flying Dutchman, 1969).

From 1971 he recorded for Impulse!. Constant Throb was cut at Western Recorders in Los Angeles on August 12 and 19, 1971 and released in 1972, with Wilton Felder and Reggie Johnson on bass, Jim Keltner and Shelly Manne on drums, Howard Roberts on guitar, Mike Wofford and Mike Lang on keyboards, Don Menza on alto flute and bass clarinet, and Marni Nixon singing on two tracks. Waterfalls (1972) and Intensity (1973) followed. Intensity contains a "Prayer for John Coltrane" and a "Waltz for John Coltrane," and AllMusic describes him as a very strong, Coltrane-inspired acoustic tenor player. Magic and Movement (1974) joins a June 22, 1972 set at the Ash Grove in Los Angeles, with Mike Nock on electric piano, Wilton Felder on bass and Eddie Marshall on drums, to a July 6, 1973 performance at the Montreux Jazz Festival with Tom Canning on electric piano, Cecil McBee on bass and Alphonse Mouzon on drums, with overdubs added in 1974.

The records that made him widely known came next, on ABC. Touch (1975) was a soft, melodic album built around his echo-treated saxophone, and it became one of the earliest titles in Mobile Fidelity Sound Lab's half-speed audiophile reissue series. Barefoot Ballet (1976), Lifestyle (Living and Loving) (1977), Arabesque (1978), the unaccompanied Solo Saxophone: Cry (1978) and Brazilia (1979) continued in the same direction, and Wikipedia and AllMusic both place the Touch series and the solo saxophone records among the forerunners of smooth jazz and new age music. The shift did not please everyone. Klemmer has said the jazz press began to doubt that he could still play once he had a popular album, and he maintains that the music on Touch and Barefoot Ballet was the most honest he had made. He wrote for other singers as well: with the lyricist David Batteau he wrote "Walk in Love," recorded by The Manhattan Transfer. As a session player he took the tenor solo on "The Caves of Altamira" on Steely Dan's The Royal Scam (ABC, 1976), and Wikipedia credits him on John Lee Hooker's Born in Mississippi, Raised Up in Tennessee (ABC, 1973), Gloria Lynne's I Don't Know How to Love Him (ABC/Impulse!, 1976) and Nancy Wilson's Life, Love and Harmony (Capitol, 1979).

Through the same years he kept returning to straight jazz. He is the tenor soloist on the opening track of Roy Haynes's Thank You Thank You (Galaxy, recorded July 1977 at Fantasy Studios in Berkeley), alongside Bobby Hutcherson on vibraphone, George Cables on electric piano and Ron Carter on bass. In December 1978 he went back to Fantasy for the Galaxy sessions that produced the label's all-star saxophone records, playing in a quartet with the pianist Stanley Cowell, the bassist John Heard and Haynes, and appearing beside Art Pepper, Johnny Griffin, Joe Henderson, Joe Farrell and Harold Land; his "God Bless the Child" was issued on Ballads by Four (1981) and his "'Round Midnight" on 5 Birds and a Monk (1981). His two-LP Nexus for Duo and Trio (Arista/Novus, 1979), with the drummer Carl Burnett and the bassist Bob Magnusson, is the album AllMusic calls his finest jazz set, and he recorded Simpatico with the Brazilian guitarist Oscar Castro-Neves. He also made Straight from the Heart for the audiophile label Nautilus Recordings (1980) and signed with Elektra, for which he recorded Magnificent Madness (1980), then Hush, Solo Saxophone II: Life and Finesse (all 1981), followed by Two Tone (Crusaders Records, 1982), a split LP on which he plays side one with Joe Sample on keyboards and Eddie Harris takes side two.

Music (MCA, 1989) was followed by a long break from touring and recording to concentrate on composing; AllMusic notes that the rumors of ill health at the time were untrue. He founded his own Touch Records label, released Making Love, Vol. 1 in 1998 and appeared as a guest on new age albums by 3rd Force, David Arkenstone and Craig Chaquico. His records, the early Cadet ones in particular, have been sampled by hip-hop producers since the 1990s.

On April 26, 2024 he released Chateau Love on Touch Records, a set of twelve tracks recorded live with the bassist Abe Laboriel Sr., the drummer Harvey Mason and the percussionist Lenny Castro, with Ronnie Foster on keyboards, John Tropea on guitar and the singers Katie Christopher and Darlene Koldenhoven. It is his first album of new material since Making Love, Vol. 1. Klemmer plays tenor and soprano saxophone, synthesizer and piano on it and sings on some tracks.

Klemmer worked at both ends of the line between jazz and pop, and the two bodies of work sound like different players. The Impulse! albums and Nexus show a hard-toned, Coltrane-influenced tenor saxophonist whose early-1970s live sets were long and intense. The ABC records that followed are another proposition: a warm, breathy tone, simple melodies, a rhythm section that stays out of the way, and the Echoplex delay, which lets one saxophone answer itself and gives the playing its chordal, hovering quality. That sound, together with the unaccompanied Cry, is why he is often listed among the precursors of smooth jazz and new age saxophone, and it is also why many of his straight-ahead listeners felt he had left them. His own position is that the later records were the music he most wanted to make, and that the solos on them are not background music.

His Chicago teacher Joe Daley gave him his grounding in improvisation, and John Coltrane is the figure he honors by name on record, in the two dedications on Intensity and in the album Nexus, a duo and trio set he made as a straight-ahead statement. Don Ellis's electrified big band is where he says he first heard what echo could do to a saxophone, and Eddie Harris, another early player of the electric saxophone, was a Chicago colleague and later a recording partner. As a composer he has two lives: the instrumentals for his own records, and pop songs written with others, such as "Walk in Love" with David Batteau.