Joseph Nathan Oliver was born on December 19, 1881, in Aben, Louisiana, near Donaldsonville in Ascension Parish, to Nathan Oliver and Virginia "Jinnie" Jones. He gave 1881 as his birth year on his September 1918 draft registration, but some census records and other sources suggest 1884 or 1885. He moved to New Orleans in his youth, studied trombone first, and then switched to cornet. From 1908 to 1917 he played cornet in the city's brass bands, in dance bands, and in the red-light district known as Storyville, and in the late 1910s a band he co-led with trombonist Kid Ory was considered one of the best and hottest in New Orleans. He married Estelle "Stella" Dominick there in September 1911. He was in demand across economic and racial lines, and he taught and mentored the young Louis Armstrong, giving him his chair in Ory's band when Oliver left for Chicago.

Oliver moved to Chicago in 1918 with his wife and stepdaughter, Ruby Tuesday Oliver. There he found work with fellow New Orleanians including clarinetist Lawrence Duhé, bassist Bill Johnson, trombonist Roy Palmer, and drummer Paul Barbarin, and he took over leadership of Duhé's band at a number of Chicago clubs. In January 1920 he formed a band at the Dreamland that gathered the players who became the core of his Creole Jazz Band, among them clarinetist Johnny Dodds and trombonist Honoré Dutrey. In the summer of 1921 he took a group to the West Coast for engagements in San Francisco and Oakland, where the band worked in the vaudeville tradition.

Back in Chicago in 1922, the group opened at the Lincoln Gardens as King Oliver and His Creole Jazz Band. Oliver played first cornet, with his protege Armstrong on second cornet, Baby Dodds on drums, Johnny Dodds on clarinet, Lil Hardin on piano, Honoré Dutrey on trombone, and Bill Johnson on bass. The band recorded in 1923 for Gennett, Okeh, Paramount, and Columbia. Its first Gennett session, held in Richmond, Indiana, in April 1923, produced nine sides, among them "Canal Street Blues," "Snake Rag," and "Dippermouth Blues," and it was Armstrong's recording debut. Because the acoustic process cut directly from the horn to the disc, Armstrong had to stand in the corner of the room, since his powerful tone could throw the needle off the master. Further sessions followed for Okeh on June 22 and 23, for Columbia on October 15 and 16, and for Gennett again on October 5, 1923. White musicians came to Lincoln Gardens to learn from the band, listening from outside the door because the club was in a Black neighborhood and admitted only Black patrons. A planned tour of the Midwest ended up breaking the band apart in 1924.

In the mid-1920s Oliver enlarged his group to nine pieces as King Oliver and His Dixie Syncopators, with more written arrangements and room for jazz solos, playing the Plantation Café in direct competition with Armstrong's Sunset Stompers at the Sunset Café. The band recorded for Vocalion and Brunswick from March 1926, and those sessions include "Sugar Foot Stomp" and "Wa Wa Wa" (May 29, 1926), "Dead Man Blues" and "Snag It" (September 17, 1926), and "Doctor Jazz" (April 22, 1927). He also played cornet behind blues singers on Okeh, among them Butterbeans and Susie in September 1924 and Sippie Wallace in February 1925. In 1927 the band went to New York, where he disbanded it for freelance work, although Brunswick kept recording a Dixie Syncopators lineup under his name there through 1928. Gum disease was making playing harder, so he began giving solos to younger players, including his nephew Dave Nelson, Louis Metcalf, and Red Allen. He held out for more money than New York's Cotton Club would pay, and Duke Ellington took the engagement instead. Oliver reunited his band in 1928 and recorded for Victor from January 1929, cutting more than thirty sides by September 1930, including "West End Blues," "Call of the Freaks," "Too Late," and "Struggle Buggy." He was still recording for Brunswick in January and February 1931.

The Great Depression ended the good years. A succession of managers had stolen his money, he lost his savings when a Chicago bank failed, and bookings dried up while his periodontal disease progressed; by 1935 he could no longer play the trumpet at all. Stranded in Savannah, Georgia, he pawned his trumpet and his best suits, briefly ran a fruit stall, and then worked as a janitor at Wimberly's Recreation Hall on West Broad Street. He passed away in a Savannah rooming house on or about April 10, 1938, of arteriosclerosis, too poor to pay for treatment. His sister paid to have his body brought to New York, and he was buried at Woodlawn Cemetery in the Bronx, with Armstrong and other musician friends in attendance. In 2007 he was inducted as a charter member of the Gennett Records Walk of Fame in Richmond, Indiana.

Oliver was admired above all for his use of mutes. He pioneered the rubber plumber's plunger, the derby hat, bottles, and cups as ways of altering a horn's voice, and his muted solo on "Dippermouth Blues," for which he favored a small metal mute made by C. G. Conn, was composed in advance and played the same way on both 1923 recordings. His "Wa Wa Wa" with the Dixie Syncopators is credited with giving the name wah-wah to such effects, and his pieces "Eccentric" and "Call of the Freaks" built whole performances on the growls and cries of that "freak" style. His Creole Jazz Band set the model on record for New Orleans collective improvisation, with two cornets weaving around a clarinet and a trombone, and his compositions "Dippermouth Blues," "Canal Street Blues," "Sweet Like This," "Doctor Jazz," and "Snag It" stayed in the traditional jazz repertory for decades. Don Redman reworked "Dippermouth Blues" as "Sugar Foot Stomp" for Fletcher Henderson's orchestra in 1925, and swing bands recorded Oliver's piece too, among them Artie Shaw in 1936 and Benny Goodman in 1937. Oliver named the early cornetist Buddy Bolden as an influence, and in turn he shaped Tommy Ladnier, Paul Mares, Muggsy Spanier, Johnny Wiggs, Frank Guarente, and Louis Panico, whose book The Novelty Cornetist illustrates the mute techniques he learned from Oliver, and above all Armstrong, who called him his idol and wrote that jazz would not be what it is without him. Young white musicians such as Bix Beiderbecke went to hear the Creole Jazz Band in Chicago.