James Peter Giuffre was born on April 26, 1921, in Dallas, Texas, the son of Joseph Francis Giuffre, an Italian immigrant from Termini Imerese in Sicily, and Everet McDaniel Giuffre. He began on clarinet at nine, graduated from Dallas Technical High School, and studied at North Texas State Teachers College (now the University of North Texas), finishing in 1942. After wartime service in which he played in Army bands, he took jobs with the bandleaders Boyd Raeburn, Jimmy Dorsey and Buddy Rich before settling into the Woody Herman orbit.

His breakthrough was an arrangement. "Four Brothers," written in 1947 on the chord changes of "Jeepers Creepers" for the saxophone section of Herman's Second Herd, was recorded for Columbia on December 27, 1947, by a section of Zoot Sims, Serge Chaloff, Herbie Steward and Stan Getz, and it gave the whole band its nickname. Giuffre later played saxophone in the Herman band himself. He then moved to Los Angeles, studied composition with Wesley LaViolette, learned clarinet and baritone saxophone in earnest, and became a central figure in West Coast jazz. In 1951 he joined Howard Rumsey's Lighthouse All Stars in Hermosa Beach as a full-time member alongside Shorty Rogers and Shelly Manne, co-writing many of the group's charts with Rogers, among them the hit "Big Boy." He left the Lighthouse in September 1953 for Rogers's Giants, and he also recorded in these years with Teddy Charles and with Manne, mostly on tenor and baritone saxophone.

As a leader he recorded for Capitol in 1954 and 1955 and then for Atlantic. Tangents in Jazz, cut at Capitol's Los Angeles studios in June 1955 with trumpeter Jack Sheldon, bassist Ralph Peña and drummer Artie Anton, mixed short pieces and brief interludes. In 1956 he formed the first Jimmy Giuffre 3 with guitarist Jim Hall and Ralph Peña, later replaced on bass by Jim Atlas, a group with no piano and no drums that played what Giuffre called blues-based folk jazz. The trio's "The Train and the River" was featured on the CBS television program The Sound of Jazz on December 8, 1957, where Giuffre was also paired with the clarinetist Pee Wee Russell for a long, loose blues. When Atlas left, Giuffre replaced the bassist with the valve trombonist Bob Brookmeyer, an instrumentation he partly credited to Aaron Copland, and the new trio opened at New York's Village Vanguard and recorded Trav'lin' Light, The Four Brothers Sound (his own tenor saxophone overdubbed four times, with Brookmeyer on piano) and Western Suite. This version of the group can be seen playing "The Train and the River" in the 1959 film Jazz on a Summer's Day, shot at the 1958 Newport Jazz Festival. In 1959 a Hall trio with bassist Buddy Clark shared a Rome concert bill with Gerry Mulligan's band, and that year Giuffre also made a run of Verve albums, among them The Easy Way with Ray Brown on bass and the two-leader sessions Herb Ellis Meets Jimmy Giuffre and Lee Konitz Meets Jimmy Giuffre.

In 1961 he formed a new trio with pianist Paul Bley and bassist Steve Swallow and turned his attention mainly to clarinet. Their Verve albums Fusion and Thesis (both 1961), with pieces by Carla Bley and Paul Bley beside Giuffre's own, and live European radio recordings from Stuttgart and Bremen that year moved steadily toward collective improvisation, in a hushed, chamber-music register unlike the harder-edged free jazz of Albert Ayler or Archie Shepp. The trio's last record, Free Fall (Columbia, 1963), was wholly improvised, several years before the European free-improvisation boom, and it found almost no audience; by one critic's account the group broke up after a night when each player earned 35 cents for a set. Giuffre turned to teaching and, in the early 1970s, to a new trio with bassist Kiyoshi Tokunaga and drummer Randy Kaye in which he added bass flute and soprano saxophone, followed by a later group with Pete Levin on synthesizer and electric bassist Bob Nieske that made three albums for the Italian label Soul Note. He led the jazz ensemble at New York University, taught saxophone and composition privately, and taught improvisation at Manhattanville College.

In the 1980s and 1990s he recorded in Europe with the bass clarinetist André Jaume, among them Eiffel: Live in Paris and Momentum, Willisau 1988, made River Station (1991) with the reed player Joe McPhee, and reunited with Bley and Swallow, who had by then switched to electric bass guitar, for the two The Life of a Trio albums (Saturday and Sunday, 1990), Fly Away Little Bird (1992) and Conversations with a Goose (1996). Through the mid-1990s he taught at the New England Conservatory of Music, where the tenor saxophonist Bill McHenry was among his students. Parkinson's disease ended his playing in his last years, and he passed away from pneumonia in Pittsfield, Massachusetts, on April 24, 2008, two days short of his 87th birthday.

Giuffre made his name as a composer and arranger of cool, pale-toned small-group music, and the voice he preferred was the clarinet, cool-toned and mostly in its low register. His drummerless, pianoless trios gave each player an exposed line of his own, and AllMusic's Thom Jurek heard in Western Suite the ambition of Copland's Billy the Kid inside a form that still swung like West Coast jazz. The folk and blues melodies of the 1950s trio, which he called blues-based folk jazz, gave way in 1961 to open improvisation in which texture, silence and counterpoint carried more weight than solos; some accounts cite his 1959 meeting with Ornette Coleman at the Lenox School of Jazz as an influence on that change. Critics have since placed the Bley and Swallow recordings among the most radical documents of early-1960s jazz, even though they were largely ignored then, and the clarinetist Perry Robinson is placed after Giuffre in the line of free clarinet playing. His drummerless-trio songbooks were reworked by Ted Nash and Steve Cardenas with bassist Ben Allison on Quiet Revolution (2018), and Julian Lage has recorded Giuffre's "Trudgin'."