James Harrell McGriff was born on April 3, 1936, in the Germantown section of Philadelphia. Both of his parents played piano, and he began on the instrument at five. By his teens he had added vibraphone, alto saxophone, drums and upright bass, and bass was what he played in his first group, a piano trio. The Guardian says he sometimes played organ in his parents' Baptist church, and AllMusic lists the saxophonist Benny Golson and the soul singer Harold Melvin among his cousins. He served in the U.S. Army as a military policeman during the Korean War, stationed in Korea according to his obituary in the Los Angeles Times, and then spent about two years on the Philadelphia police force while he kept playing bass behind singers including Big Maybelle and Carmen McRae.

The organ took over in 1956. He was drawn to the sound of the instrument when Richard "Groove" Holmes played at his sister's wedding, and Holmes became his teacher and friend. McGriff bought a Hammond B-3 that year and spent six months learning it, storing it for a time at the home of Archie Shepp. He studied at Philadelphia's Combe College of Music and at New York's Juilliard School, and privately with Milt Buckner, Jimmy Smith and Sonny Gatewood. He named Buckner's energy and the poise of Count Basie's band as influences, along with local organists Howard "The Demon" Whaley and Austin Mitchell. His early combo around Philadelphia often included the tenor saxophonist Charles Earland, who soon switched to organ and became one of its best-known players, and the group accompanied touring performers such as Don Gardner, Arthur Prysock, Candido and Carmen McRae.

In 1961 Joe Lederman's small Jell label recorded his trio in an instrumental version of Ray Charles's hit "I've Got a Woman." Local radio play was strong enough that Juggy Murray's Sue label picked it up and recorded a full album, released in 1962, which also produced a second hit in McGriff's own "All About My Girl." Sue issued a run of albums through 1965, among them One of Mine, the live At the Apollo, Christmas with McGriff, At the Organ and Topkapi, ending with Blues for Mister Jimmy. In 1966 the producer Sonny Lester, starting his Solid State label, made McGriff its lead attraction. The first album, The Big Band (1966), put him in front of a studio big band that included Count Basie alumni such as Frank Wess, Frank Foster and Joe Newman, and was later reissued as A Tribute to Basie. Solid State went on to release A Bag Full of Soul, Honey, A Thing to Come By and the singles "The Worm" and "Step One," and in 1970 he made the funk-leaning Electric Funk for Blue Note.

McGriff settled in Newark, New Jersey, where he opened a supper club, the Golden Slipper, and recorded the live albums Black Pearl and, with the blues singer Junior Parker, Chicken Fried Soul in 1971. From 1969 he also played regularly in Buddy Rich's band, though the two were recorded together only once, on The Last Blues Album Volume 1 in 1974. He announced a retirement in 1972 to start a horse farm in Connecticut, but Lester's Groove Merchant label kept issuing his records three or four times a year, and by 1973 he was touring and recording again. That year he made two albums with his old teacher Holmes, Giants of the Organ Come Together and the live Giants of the Organ in Concert. With disco and fusion taking hold he added electric keyboards to his organ work, and among his strongest records of these years are Stump Juice (1975), The Mean Machine and Red Beans (both 1976) and Outside Looking In (1978).

By 1980 he had left Lester and begun working with the producer Bob Porter and the engineer Rudy Van Gelder, and a long run on Milestone Records began with Countdown (1983). Those sessions teamed him with horn players including Rusty Bryant, Al Grey, Red Holloway, David "Fathead" Newman, Frank Wess and Eric Alexander. In 1986 he began a partnership with the alto saxophonist Hank Crawford that produced the Milestone co-led albums Soul Survivors (1986), Steppin' Up (1987), On the Blue Side (1990), Road Tested (1997) and Crunch Time (1998), plus two Telarc dates, Right Turn on Blue (1994) and Blues Groove (1996). Between 1994 and 1998 he also played a Hammond XB-3, an organ with MIDI capability, and he was among the first B-3 players to add MIDI to the upper manual to widen his sound. In the mid-1990s he formed a group called the Dream Team with Newman and the drummer Bernard Purdie, which made The Dream Team (1997), Straight Up (1998), McGriff's House Party (2000, with Dr. Lonnie Smith and Alexander), Feelin' It (2001) and McGriff Avenue (2002). He also played on other leaders' records, among them the gospel singer Tramaine Hawkins's 1990 live album and the tenor saxophonist Illinois Jacquet's Loot to Boot (1991).

On March 29, 2008, the organist Bill Dilks and the drummer Grant MacAvoy gave McGriff a private concert in Voorhees Township, New Jersey, at the Genesis Health Care Center, with his wife Margaret and guests present. McGriff passed away in Voorhees Township on May 24, 2008, at the age of 72, of complications of multiple sclerosis.

McGriff was known first as a blues organist. AllMusic calls him the bluesiest of the major soul-jazz organists, with a sound built on gospel and blues feeling, and his obituaries note that he often described himself as a bluesman rather than a jazz player. The Guardian observed that, unlike his childhood friend and teacher Jimmy Smith, he rarely laid bebop-derived improvisation over funky swing, favoring short riff-based themes and compact, jukebox-ready singles that made him popular with R&B audiences; "I've Got a Woman," "All About My Girl," "Kiko" and "The Worm" all reached the R&B charts. He cited the church, Milt Buckner's energy and Count Basie's band as sources, and The Big Band is his tribute to the Basie sound. In the 1970s he followed fusion and disco trends with electric keyboards and a smoother style, then returned in the 1980s to the organ-trio and small-band format that critics and collectors were rediscovering. The albums with Hank Crawford and the later Dream Team records paired him with veteran soul-jazz and R&B players and younger post-bop saxophonists such as Eric Alexander. Charles Earland, who learned the organ from him, is among the players who carried the style forward.