Mitchell Herbert Ellis was born August 4, 1921, on a farm outside Farmersville, Texas. He took up harmonica at three and banjo at six; hearing the Light Crust Doughboys on the radio and, later, guitarist George Barnes pointed him toward the guitar instead. He enrolled as a music major at North Texas State University but studied string bass there, since the school had no formal guitar program at the time. After a brief stint with a University of Kansas dance band, he joined Glen Gray's Casa Loma Orchestra (1943-1945) and then Jimmy Dorsey's touring band, where he cut some of his first recorded solos.

Around 1947, during a six-week gap in Dorsey's schedule, Ellis fell in with pianist Lou Carter and bassist/violinist John Frigo at Buffalo's Hotel Stuyvesant and formed a vocal-and-instrumental trio, the Soft Winds, modeled on the Nat King Cole Trio. The group wrote and recorded original material, including the standards "Detour Ahead" and "I Told Ya I Love Ya, Now Get Out," and stayed together until around 1952.

In 1953, Ellis replaced guitarist Barney Kessel in pianist Oscar Peterson's trio alongside bassist Ray Brown, forming what critic Scott Yanow called one of the most memorable piano-guitar-bass units in jazz history. The trio became the de facto house rhythm section for Norman Granz's Verve Records and a mainstay of Granz's Jazz at the Philharmonic package tours, backing Ben Webster, Stan Getz, Dizzy Gillespie, and Roy Eldridge on record while touring the United States, Europe, and Asia. One stop on that circuit, a 1953 JATP date in Japan, led directly to Ellis's presence on pianist Toshiko Akiyoshi's 1954 debut LP Toshiko's Piano (Norgran): after Peterson heard Akiyoshi play in Tokyo and urged Granz to record her, Granz did so using Peterson's own touring rhythm section, Ray Brown on bass, Ellis on guitar, and J.C. Heard on drums, giving her first record a trio of American players who had no reason to treat her as a novelty. Ellis made his own leader debut in this period with Ellis in Wonderland (1956, Norgran), cut in Hollywood with Peterson, Ray Brown, Charlie Mariano, Jimmy Giuffre, and Harry "Sweets" Edison. He left the Peterson trio in November 1958, the point at which the group's format shifted permanently to piano, bass, and drums with Ed Thigpen replacing him; for the whole of his five years in the chair he had also been, by his own and others' account, the only white member of a working group touring the segregated South.

Ellis's second Verve leader date, Nothing but the Blues (1957), reunited him with Roy Eldridge and Stan Getz plus Ray Brown and drummer Stan Levey, and drew some of the strongest reviews of his career: Nat Hentoff called it his best record yet, and DownBeat's John Tynan named it one of the best jazz albums of the year. He toured as Ella Fitzgerald's guitarist from 1957 to 1960, kept recording as a leader (Herb Ellis Meets Jimmy Giuffre, 1959; Softly...But With That Feeling, 1961, with vibraphonist Victor Feldman and bassist Leroy Vinnegar), and in 1962 cut The Midnight Roll and Gravy Waltz for Epic with Roy Eldridge, tenor saxophonist Buddy Tate, and pianist Ray Bryant, produced by John Hammond. 4 To Go! (1963, Columbia) paired him with pianist André Previn, Ray Brown, and drummer Shelly Manne. Through the 1960s and into the 1970s he settled into Hollywood studio work, spending long stretches in the house bands of Steve Allen's and Merv Griffin's television shows (six years on Griffin's alone) alongside guitarist Jim Hall and, again, Ray Brown; a 1969 reunion date with his old trio boss, Hello Herbie (MPS, 1970), brought Peterson, Sam Jones, and drummer Bobby Durham together in the studio.

The next chapter of Ellis's career began almost by accident. Performing at one of promoter Carl Jefferson's early Concord Summer Festivals in California, Ellis and guitarist Joe Pass asked Jefferson to fund and produce a record; the result, Jazz/Concord (1973, cut at Wally Heider Studios in Los Angeles with Pass, Ray Brown, and drummer Jake Hanna), became the first release on Jefferson's new label, Concord Jazz, which grew into one of the defining mainstream jazz imprints of the following two decades. Ellis and Pass followed it within a year with the live Seven, Come Eleven and the duo studio set Herb Ellis & Ray Brown's Soft Shoe, while Two for the Road (1974), another duo record with Pass, appeared on Norman Granz's Pablo label. In 1974 Ellis also co-founded the guitar trio Great Guitars with Barney Kessel and Charlie Byrd (later expanded to a quartet with Tal Farlow), a group that toured and recorded steadily into the 1980s (Great Guitars, 1975 and 1976; Straight Tracks, 1978; Great Guitars at the Winery, 1980; Great Guitars at Charlie's Georgetown, 1983). Ellis became one of Concord's most reliable artists through the rest of the decade: Windflower (1978) paired him with guitarist Remo Palmier and bassist George Duvivier; Soft & Mellow (1979) reunited him with pianist Ross Tompkins and drummer Jake Hanna from the Griffin show band; and a long-running trio with Ray Brown and pianist Monty Alexander produced Trio (1981), Interplay (1981, with guitarist Cal Collins sitting in), and the Triple Treat series (1982, 1988, 1989), the latter two albums bringing his old Soft Winds partner John Frigo back into the fold as a guest on violin. Doggin' Around (1989), a duo record with bassist Red Mitchell, carries cover art and liner notes by cartoonist Gary Larson, who had taken guitar lessons from Ellis and repaid him with the artwork.

Ellis relocated to Arkansas in 1987 and was inducted into the Arkansas Jazz Hall of Fame in 1994; the University of North Texas College of Music, whose guitar program hadn't existed when he was a student there, awarded him an honorary doctorate on November 15, 1997. His later leader dates continued to circle back to old partners: Roll Call (1991, Justice Records) reunited him on record with Frigo and drummer Jake Hanna, Down Home (recorded 1991, released 1996, Justice) paired him with a younger Texas rhythm section, and Conversations in Swing Guitar (1999, Stony Plain), a two-guitar session with Duke Robillard, was among his last studio recordings. Herb Ellis died of Alzheimer's disease in Los Angeles on March 28, 2010, at age 88.

Ellis played a bop-fluent, blues-rooted style with a country twang traceable to his Texas upbringing and his early exposure to the Light Crust Doughboys, and he was equally comfortable as a soloist and as a supportive, harmonically alert accompanist, the skill that made him indispensable inside the Oscar Peterson Trio's tightly wound arrangements. Along with Ray Brown, he pushed Peterson hard within that group's competitive three-way interplay, a dynamic critics still cite as one of the defining sounds of 1950s piano-trio jazz. His 1953 Gibson ES-175, which he played for decades, became closely associated with him as an instrument. Beyond the trio, Ellis's long-running duo and small-group partnerships, first with Joe Pass, later with Barney Kessel and Charlie Byrd in Great Guitars, and finally with Monty Alexander, Ray Brown, and John Frigo in the Triple Treat series, built a second reputation for him as one of jazz guitar's most versatile and durable collaborators, a player whose Texas swing and blues instincts blended naturally into whatever bebop or mainstream-jazz format he was placed in.