Elvin Ray Jones was born September 9, 1927, in Pontiac, Michigan, to Henry and Olivia Jones, who had migrated from Vicksburg, Mississippi. He was the youngest of the family's musical sons, following pianist Hank Jones, born in 1918, and trumpeter/composer Thad Jones, born in 1923. Jones later said he was drawn to drums from age two, fascinated by marching bands passing the family home, and he built on that as a teenager in his Pontiac high school's marching band. He served in the U.S. Army from 1946 to 1949; on discharge he used his mustering-out pay, topped up with $35 borrowed from a sister, to buy his first drum kit.

Jones turned professional in Detroit in 1949, working the Grand River Street club circuit with saxophonists Billy Mitchell and Wardell Gray as the city's postwar bebop scene took shape. He was part of the same young Detroit cohort as pianist Barry Harris and baritone saxophonist Pepper Adams, who was a teenager newly arrived from Rochester in the same years. A failed 1955 audition for Benny Goodman's band pushed Jones toward New York instead. He arrived in time to record Blue Moods (1955, Debut Records) alongside Miles Davis and Charles Mingus, then spent the late 1950s as a first-call sideman, joining Sonny Rollins's trio for part of the November 1957 Village Vanguard engagement documented on A Night at the "Village Vanguard." Detroit connections followed him east: he drummed on Pepper Adams's The Cool Sound of Pepper Adams (recorded November 1957, with Hank Jones on piano), the Pepper Adams-Donald Byrd Quintet's live album 10 to 4 at the 5 Spot (April 1958), and The Pepper-Knepper Quintet (March 1958).

In September 1960 Jones joined tenor saxophonist John Coltrane's new quartet, completing a rhythm section with pianist McCoy Tyner and bassist Jimmy Garrison that would record most of Coltrane's landmark Atlantic and Impulse! albums: My Favorite Things (1961), Africa/Brass (1961), Live at the Village Vanguard (1961), Ballads (1963), Crescent (1964), and A Love Supreme (recorded December 1964). Jones's rolling, three-against-four polyrhythms and his ability to sustain a churning, non-metronomic pulse behind Coltrane's extended improvisations became one of the defining sounds of 1960s jazz. He stayed through the group's turn toward collective free playing on Ascension (1965), but left in early 1966, unsettled by Coltrane's decision to add a second drummer, Rashied Ali, whose freer, non-pulse approach Jones found impossible to mesh with: "I couldn't hear what was going on," he said later, "I felt I just couldn't contribute."

Jones led his own dates throughout the same period and after, starting with Elvin! (recorded 1961-62, Riverside), which reunited him with both his brothers, Hank Jones on piano and Thad Jones on cornet, alongside Frank Foster and Frank Wess. He recorded the Coltrane tribute Dear John C. for Impulse! in February 1965 and the duo date Heavy Sounds with bassist Richard Davis in 1967. His most productive run as a leader came on Blue Note from 1968 into the mid-1970s, first with a trio featuring saxophonist Joe Farrell and bassist Jimmy Garrison on Puttin' It Together (April 1968) and The Ultimate (1969), then with expanded quintets on Poly-Currents (recorded September 1969), Genesis and Merry-Go-Round (both drawing on February and December 1971 sessions), Live at the Lighthouse (recorded September 1972), and Mr. Jones (sessions from 1969 and 1972). Those Blue Note dates brought in a rotating cast of saxophonists including Steve Grossman, Dave Liebman, George Coleman, and Frank Foster, and Pepper Adams returned the favor of Jones's earlier sideman work by playing baritone sax on both Poly-Currents and Merry-Go-Round; Thad Jones guested on flugelhorn on Mr. Jones.

Jones moved to Vanguard Records for New Agenda (1975) and Time Capsule (1977), then recorded in Japan for Trio Records (Very R.A.R.E., 1979, with Art Pepper and Roland Hanna) and on the American Palo Alto and Quicksilver labels (Earth Jones, recorded 1982; the Coltrane tribute Brother John, recorded October 1982). Through the 1980s and 1990s he led the Elvin Jones Jazz Machine, a touring outfit that at various points featured saxophonists Sonny Fortune and Ravi Coltrane (John Coltrane's son), documented on the live album In Europe (recorded June 1991). On Youngblood (1992, Enja), recorded with bassist George Mraz, Jones fronted a younger generation of players, tenor saxophonists Joshua Redman and Javon Jackson and trumpeter Nicholas Payton. He also reunited with brother Hank Jones in the co-op group the Great Jazz Trio. Jones kept teaching and touring into his late seventies, received an honorary doctorate from Berklee College of Music in 2001, and made his final recording, The Truth: Heard Live at the Blue Note, at a September 1999 engagement (released posthumously in 2004) with a band including trombonist Robin Eubanks and guest saxophonist Michael Brecker. Elvin Jones passed away from heart failure on May 18, 2004, in Englewood, New Jersey, at age 76.

Jones is widely credited with freeing the jazz drummer from strict timekeeping, replacing the ride-cymbal ping of bebop drumming with a rolling, polyrhythmic wash in which triplet-based patterns across the whole kit implied multiple pulses at once, most audibly behind Coltrane's modal and free playing from 1960 to 1966. Albert Goldman called him "the world's greatest rhythm drummer" in a 1970 Life magazine profile, and historian Ted Gioia has described him as one of the most influential drummers in jazz history. His impact reached well beyond jazz: rock drummers including Mitch Mitchell (whom Jimi Hendrix nicknamed "my Elvin Jones"), Ginger Baker, Bill Bruford, and John Densmore of the Doors all cited him directly, and Jones was inducted into the Modern Drummer Hall of Fame in 1995 and ranked among Rolling Stone's 100 Greatest Drummers of All Time. He was also honored as an NEA Jazz Master.