Kermit Hampton Ruffins was born on December 19, 1964, in New Orleans and grew up in the Lower Ninth Ward. He began on trumpet in the eighth grade at Lawless Junior High School, went on to Joseph S. Clark High School in the Tremé, and attended St. Peter Claver Church in the same neighborhood. A strict high-school band teacher gave him some classical grounding, but the music that shaped him came from the streets of his city: the brass bands, and above all the records of Louis Armstrong and Louis Jordan.

In 1983, while still at Clark High, Ruffins co-founded the Rebirth Brass Band with the tuba player Phillip Frazier and the drummer Keith Frazier, taking the Dirty Dozen Brass Band's mix of funk and modern jazz as their model. The group busked in the French Quarter for tips before becoming the house band at the Glass House, and its first record, Here to Stay, was recorded for Arhoolie Records in May 1984. Ruffins contributed two of the band's enduring numbers, "Do Whatcha Wanna" and "Put Your Right Foot Forward." He left Rebirth in 1992, by which time he had added singing to his trumpet playing; he has said that nearly every trumpeter he saw in New Orleans sang, a habit he traced to Armstrong.

The same year he formed the Barbecue Swingers, a traditional-jazz quintet named for his habit of grilling food for fans, which he did from the back of a red pickup truck parked outside his shows. His debut as a leader, World on a String (Justice Records, 1992), brought in the banjoist Danny Barker, the pianist Ellis Marsalis, the bassist Walter Payton, the trombonist Lucien Barbarin and the drummer Shannon Powell. Barker's advice to Ruffins was to play the melody and skip the clever stuff, a lesson he took to heart as the lead voice of his own bands. A second Justice album, The Big Butter & Egg Man (1994), followed, then Hold On Tight (1996), co-produced by the trombonist Delfeayo Marsalis. Ruffins also realized a long-held ambition by leading a 17-piece big band, which made its debut on October 28, 1995, and played the Jazz Tent at the 1996 New Orleans Jazz & Heritage Festival.

In the fall of 1997 the Barbecue Swingers recorded live at Tipitina's, producing The Barbecue Swingers Live, the first release on the new Basin Street Records. He stayed with the label for two decades: Swing This! (1999), 1533 St. Philip Street (2001), Big Easy (2002), Throwback (2005, with the Rebirth Brass Band), Live at Vaughan's (2007), Livin' a Tremé Life (2009), Have a Crazy Cool Christmas! (2009), Happy Talk (2010), We Partyin' Traditional Style! (2013), #imsoneworleans (2015) and A Beautiful World (2017). From 1992 until September 2013, when he gave up the slot to concentrate on other ventures, he played Thursday nights at Vaughan's Lounge in the Bywater neighborhood, a residency that became a fixture for locals and visitors and that Basin Street captured on Live at Vaughan's. In 2003 he won an OffBeat magazine Best of the Beat award for best traditional jazz band or performer.

After Hurricane Katrina in 2005 he became one of the city's best-known musical ambassadors, and in 2007 he played the White House lawn at the Congressional Picnic. He appeared on screen in the documentary Make It Funky! (2005), where he takes part in a trumpet challenge on "Skokiaan" with Irvin Mayfield and Troy Andrews, now known as Trombone Shorty, and played himself as a recurring character on HBO's Tremé (2010 to 2013). In 2011 he signed a lease to reopen the Mother-in-Law Lounge on Claiborne Avenue, the former club of the R&B singer Ernie K-Doe, and in 2012 he opened Kermit's Tremé Speakeasy, a restaurant and bar where he cooks and plays. In 2016 he sang "The Bare Necessities" with Bill Murray on the soundtrack of Disney's remake of The Jungle Book.

Ruffins is first an entertainer and a melodist: Jon Pareles of The New York Times described him as an unabashed entertainer who plays trumpet with a bright, silvery tone and sings with off-the-cuff charm. His idols are Louis Armstrong, whose repertoire runs through his sets, and Louis Jordan, whose jump-band humor shows in his song choices and stage manner. He says the highest note he can hit is a high C, and he credits ten years of playing hard in brass bands, beginning with Rebirth, for the strong lead he brings to the smaller swing and traditional bands he leads. His sets mix New Orleans standards and Armstrong-era favorites with his own tunes, including the comic numbers "Hide the Reefer" and "I'll Drink Ta Dat." The younger New Orleans brass player Trombone Shorty has named him as a mentor. Ruffins sits at the traditional, entertainer end of jazz, with funk and brass-band color from his Rebirth years rather than modern improvisation, and he is as well known in New Orleans for performing at jazz funerals and for his cooking as for his records.