James Francis Dorsey was born on February 29, 1904, in Shenandoah, Pennsylvania, the first son of Thomas Francis Dorsey, a coal miner who became a music teacher and marching-band director, and Theresa (Langton) Dorsey. By the age of seven he was playing in his father's band, and at nine he made his first public appearance on trumpet with J. Carson McGee's King Trumpeters in New York in 1913. He switched to alto saxophone in 1915 and then took up the clarinet, choosing an Albert-system instrument rather than the Boehm system used by most of his contemporaries, including Benny Goodman and Artie Shaw. With his younger brother Tommy Dorsey on trombone he formed Dorsey's Novelty Six, later Dorsey's Wild Canaries, one of the first jazz bands to be heard on the radio.

In the 1920s he moved through the leading dance and jazz bands of the East Coast: the Scranton Sirens, the California Ramblers (from around 1924), Jean Goldkette, Red Nichols, Ben Pollack, Frankie Trumbauer, and Paul Whiteman, doing a great deal of freelance radio and studio work in New York between engagements. He took the clarinet solo on "Singin' the Blues," recorded on February 4, 1927, by Trumbauer's orchestra with Bix Beiderbecke and guitarist Eddie Lang, a record later placed in the Grammy Hall of Fame. He joined Ted Lewis's band in 1930 and toured Europe with it, and on September 15, 1930, he played clarinet and alto saxophone on the first recording of Hoagy Carmichael's "Georgia on My Mind," alongside Bix, Lang, Joe Venuti, Jack Teagarden, and Bud Freeman. His composition "Tailspin," written with Trumbauer, was recorded by Whiteman in April 1934 and by the Dorsey Brothers in February 1935.

From 1927 the brothers also recorded as the Dorsey Brothers Orchestra, first as a studio-only group for OKeh, where Glenn Miller wrote arrangements and played trombone on early sessions that included "The Spell of the Blues," "Let's Do It," and "My Kinda Love" with Bing Crosby singing. Their first chart entry was "Coquette" in June 1928. In 1934 they formed a permanent touring band and signed with Decca, and in 1935 they had two number-one records, "Lullaby of Broadway" and "Chasing Shadows." The partnership ended on the bandstand in May 1935 when Tommy walked out in an argument over the tempo of "I'll Never Say Never Again Again." Jimmy kept the band, billed it for a few months as the Dorsey Brothers Orchestra in the hope that his brother would return, and from September 1935 led it as Jimmy Dorsey and His Orchestra on Decca. The orchestra backed Crosby on the radio program Kraft Music Hall from December 1935 to July 1937, appearing on 73 broadcasts and on Crosby's Decca records, among them the 1936 hit "Pennies from Heaven."

The early band was regarded as more jazz-oriented than Tommy's, and it recorded instrumentals such as "Dorsey Stomp," "Tap Dancer's Nightmare," "Parade of the Milk Bottle Caps," and "John Silver," with Bobby Byrne, Ray McKinley, Don Matteson, and Skeets Herfurt in the ranks alongside the singers Bob Eberly and Kay Weber. After he left the Crosby show in 1937 and began to tour on his own, the hits came quickly: "Is It True What They Say About Dixie?" and, in 1938, "Change Partners," both number one. Helen O'Connell joined in 1939, and her duets with Eberly, built in a three-part form in which Eberly sang a slow ballad chorus, the band and Dorsey's saxophone took the middle, and O'Connell finished at a faster tempo, sometimes in Spanish, produced the band's biggest records, among them "Amapola," which was number one for ten weeks in 1941, "María Elena," "Green Eyes," and "Blue Champagne." Almost every record the band released from 1939 to 1943 was a hit. Tutti Camarata arranged many of the Latin-flavored numbers, and pianist and arranger Joe Lipman, who joined in 1939, contributed heavily to the repertoire. "Tangerine" followed in 1942, and on July 14, 1942, Dorsey was the first to record "Brazil" (Ary Barroso's "Aquarela do Brasil") for Decca, with Eberly and O'Connell singing Bob Russell's English lyrics. In 1944 "Bésame Mucho," with Eberly and Kitty Kallen, was another number one, and Wikipedia counts eleven number-one hits for the orchestra in all. The band also kept a jazz side: "Contrasts" was its signature instrumental, Dizzy Gillespie wrote arrangements for Dorsey and shares the credit on "Grand Central Getaway," and the guitarist Herb Ellis joined the band after leaving Glen Gray's Casa Loma Orchestra and stayed through 1947, recording some of his first solos there.

Dorsey and his brother combined their orchestras to record a V-Disc on March 15, 1945, and in 1947 he signed with MGM Records and the brothers played themselves in the film The Fabulous Dorseys, which included a cameo by Art Tatum. After the commercial decline of the late 1940s he recorded Dixieland-style sides for Columbia in 1950, which led to his first LP, and then disbanded his own orchestra. In 1953 he joined the Tommy Dorsey Orchestra as co-leader and featured soloist, and the group was billed as the Dorsey Brothers Orchestra. After an appearance on Jackie Gleason's CBS program on December 26, 1953, the brothers hosted the CBS series Stage Show from 1954 to 1956; in January 1956 it carried the first national television appearances of Elvis Presley. Jimmy was diagnosed with throat cancer in 1956. When Tommy passed away on November 26, 1956, Jimmy took over the orchestra. His last recording session, for Fraternity Records, produced "So Rare," in an arrangement influenced by the rhythm-and-blues saxophonist Earl Bostic; it reached number two on the Billboard chart and earned a gold record, the highest placing for a big band in the first decade of rock and roll. Dorsey died of throat cancer in New York City on June 12, 1957, at the age of 53. The U.S. Postal Service issued a stamp honoring Jimmy and Tommy Dorsey in 1996, and the Recording Academy added his 1942 recording of "Brazil" to the Grammy Hall of Fame in 2009.

Dorsey is considered one of the most important and influential alto saxophonists of the big band and swing eras. English Wikipedia says Charlie Parker and Lester Young both acknowledged him as an influence on their styles, German Wikipedia adds Ornette Coleman, and the profile of James Moody on JazzProfiles.com lists Dorsey among that saxophonist's earliest models. His orchestra was above all a hit-making dance band, with eleven number-one records, most of them vocal duets by Bob Eberly and Helen O'Connell on ballads and Latin-American songs, and for years he was commercially overshadowed by Benny Goodman's band and then by his brother's. AllMusic's William Ruhlmann argues that amid those vocal hits it is easy to forget that the Dorsey band was also a serious jazz outfit whose members stretched out when they could: its instrumentals included "Contrasts" and "John Silver," "Sorghum Switch" (1942) used a surprisingly modern electric guitar sound, and Dorsey was willing to work elements of bebop into the book. The jazz case for him rests on that, on his 1920s and early 1930s work with Bix Beiderbecke, Frankie Trumbauer, Joe Venuti, and Eddie Lang, and on the Dixieland sides of 1950, and it is the same swing-era bandleader footing as his brother's, with a pop-chart output that is larger than his jazz one.