Serge Chaloff was born November 24, 1923, in Boston into a household where music was the family trade. His father, Julius Chaloff, was a pianist and composer; his mother, Margaret Chaloff, taught piano in Boston under the professional name Madame Chaloff and became one of the city's most sought-after teachers. Chaloff started on piano at six and studied clarinet with Manuel Valerio of the Boston Symphony Orchestra. At twelve he heard Harry Carney, the baritone saxophonist who anchored Duke Ellington's reed section for four decades, and taught himself the horn without a teacher. He later explained the decision to Leonard Feather in blunt terms: "Who could teach me? I couldn't chase [Harry] Carney around the country." From the age of fourteen he was working nights at Izzy Ort's Bar & Grille on Essex Street, underage and without a work permit.

The apprenticeship ran through the big bands. Chaloff played tenor with Tommy Reynolds in 1939, then took the baritone chair with Boyd Raeburn's orchestra in 1944 and 1945, a band that also carried Dizzy Gillespie and Al Cohn, and moved on to Georgie Auld and Jimmy Dorsey through 1945 and 1946. Auld's 52nd Street engagements put him in the middle of the new bebop vocabulary, and by 1946 he was recording small-group dates constantly: Sonny Berman's Big Eight, Bill Harris's Big Eight, the Ralph Burns Quintet, and Red Rodney's Be-Boppers. His first sessions under his own name came fast. On September 21, 1946, he cut a quintet date in Hollywood for Ross Russell's Dial label, and on March 5, 1947, he led a sextet in New York for Savoy with Red Rodney on trumpet, Earl Swope on trombone, George Wallington on piano, Curly Russell on bass, and Tiny Kahn on drums, producing "Pumpernickel," "Serge's Urge," "A Bar A Second," and Kahn's "Gabardine and Serge."

Later in 1947 Chaloff joined Woody Herman's Second Herd and became the bottom voice in the saxophone section that gave the band its name in jazz history. Jimmy Giuffre's chart "Four Brothers," recorded for Columbia on December 27, 1947, set three tenors against one baritone and handed the solos in sequence to Zoot Sims, Chaloff, Herbie Steward, and Stan Getz. The same section carried "Keen and Peachy," "The Goof and I," and "Man, Don't Be Ridiculous." Feather singled him out in Inside Be-Bop in 1949: "Great conception and execution, good taste, clean tone and Bird-like style have made him the No.1 bop exponent of the baritone." Down Beat's readers voted him the year's best baritone saxophonist three times and Metronome's readers five times across the years 1949 to 1953. The band years came at a cost. Chaloff was using heroin by 1947, and Gene Lees described him as the Herd's supplier as well as its heaviest user, dispensing to bandmates behind a blanket hung across the back of the bus. Herman confronted him publicly in Washington, D.C., and Chaloff was widely blamed inside the band for the drug-related death of trumpeter Sonny Berman in January 1947, at twenty-one.

Chaloff went home to Boston in 1950 and spent the next four years there, working the Hi Hat, the Red Fox Cafe, and the Petty Lounge, and traveling for club dates as far as Rock Island, Illinois, where a February 1953 quartet engagement at the Buvette Club was taped and eventually issued. A September 3, 1950, set at the Celebrity Club in Providence, Rhode Island, was broadcast and later released as Boston 1950. He stopped performing almost entirely through 1952 and 1953. Late in 1953 the Boston disc jockey Bob "The Robin" Martin took over as his manager, and in October 1954 Chaloff committed himself to drug rehabilitation at Bridgewater State Hospital, leaving three and a half months later, in February 1955, clean. The Boston musicians who carried him through the comeback were Charlie Mariano and Boots Mussulli on alto, Herb Pomeroy on trumpet, and Dick Twardzik on piano. Mussulli, from Milford, Massachusetts, had held the jazz alto chair in Stan Kenton's orchestra from 1944 to 1947. Two records for George Wein's local Storyville label came out of that circle: a June 1954 quintet date with Mussulli and Russ Freeman, and a nine-piece September 1954 session built around Twardzik's title composition, The Fable of Mabel.

Martin's connections got Chaloff a Capitol contract, and the two albums that resulted are the reason he is still argued about. Boston Blow-Up! was recorded at Capitol's New York studios on April 4 and 5, 1955, with the Boston sextet of Mussulli, Pomeroy, pianist Ray Santisi, bassist Everett Evans, and drummer Jimmy Zitano, most of the writing by Mussulli. Jack Tracy gave it five stars in Down Beat: "Chaloff offers the best display of his talents ever to be put on wax." Eleven months later, on March 14 and 16, 1956, Chaloff walked into Capitol's Los Angeles studio with a rhythm section he had never worked with and no plan. Pianist Sonny Clark was recording for Blue Note, bassist Leroy Vinnegar was the busiest bassist on the Los Angeles session scene, and drummer Philly Joe Jones was holding the chair in Miles Davis's quintet. Chaloff's own account: "this time I was feeling a little more easy-going, and I decided to make a record just to blow. I picked out what I felt was the best rhythm section around and told them just to show up...no rehearsals...no tunes set...and trust to luck and musicianship." The seven quartet performances issued as Blue Serge are called "Chaloff's masterpiece" in Richard Cook and Brian Morton's Penguin Guide to Jazz.

Two months after that session, playing golf in May 1956, Chaloff was hit by back and abdominal pain that left his legs paralyzed. Exploratory surgery at Massachusetts General Hospital found lesions on his spine, and a course of X-ray treatment followed. He kept recording. On June 18, 1956, at Fine Sound in New York, he played baritone on a twenty-minute reading of Charlie Parker's "Billie's Bounce" for Metronome All-Stars 1956, in a band that included Lee Konitz, Al Cohn, Thad Jones, Charles Mingus, and Art Blakey. His last studio date came February 11, 1957, at Webster Hall in New York, reuniting the Four Brothers front line with Sims, Cohn, and Steward for the Vik album The Four Brothers... Together Again!; Charlie O'Kane covered the written baritone section parts so Chaloff could save his strength for the solos. His brother Richard remembered him going to the session in a wheelchair: "when he stood up and played, you never knew he was a sick fellow." Don Gold's Down Beat review called it "a kind of significant farewell in the language he knew best." Chaloff played his final club dates from a wheelchair at the Stable on Huntington Avenue in Boston in May 1957. He was admitted to Massachusetts General on July 15 and passed away the following day, July 16, 1957, at thirty-three. He is buried at Forest Hills Cemetery in Jamaica Plain.

Chaloff took the baritone saxophone, an instrument most players used as a section anchor, and made it move at bebop speed while keeping the full weight of its bottom register in reserve. His starting points were Harry Carney with Duke Ellington and Jack Washington with Count Basie, but what he built on top of them came from Charlie Parker, and Stuart Nicholson has argued that Chaloff absorbed the emotional logic of Parker's playing rather than copying its phrases. His brother Richard described the technical effect from close range: "he could play (baritone) like a tenor sax. The only time you knew it was a baritone was when he took it down low. He played it high....He had finger dexterity, I used to watch him, you couldn't believe the speed he played." The other half of the style was a vibrato and a dynamic range wider than anything the horn had carried before, running from a near-inaudible whisper to a full shout, which is why he is still described as the most openly emotive baritone player in the music's history. That expressive, ballad-centered conception, alongside Gerry Mulligan's lighter and more contrapuntal approach, defined what the baritone was understood to be until Pepper Adams drove the instrument through hard bop's fastest tempos with a rougher tone a decade later.