Jan Johansson was born on September 16, 1931, in Söderhamn, in the Swedish province of Hälsingland, and grew up in the nearby parish of Söderala. His father, Valter August Johansson, was a district manager and his mother, Ingeborg Persson, a primary school teacher. He began piano lessons at eleven, studied classical piano as a child, and in his teens turned to swing and bebop, playing with local dance and swing bands around Söderhamn, including a sextet led by the clarinetist Gunnar Hammarlund. After finishing secondary school in 1951 he enrolled in electrical engineering at Chalmers University of Technology in Gothenburg, where he was pianist and musical director for several of the university's student revues and played with local jazz groups. He left the degree to work as a musician.

In Gothenburg in the mid-1950s he worked in the orchestra of Kenneth Fagerlund (1954 to 1955) and then, from 1956 to 1959, in the quintet of the bassist Gunnar Johnson, touring Sweden and making radio and record sessions; his first records under his own name date from this period, and he married Else Bergström in 1957. The American tenor saxophonist Stan Getz, then living in Scandinavia, took to his melodic touch. Johansson is the pianist on three tracks of Getz's Imported from Europe, recorded in Stockholm on August 26, 1958, and in 1959 Getz invited him to Copenhagen's Jazzhus Montmartre. He stayed in Denmark through the 1959-60 season, recording with Getz (Stan Getz at Large), with the bassist and cellist Oscar Pettiford on My Little Cello, Pettiford's last studio sessions, and backing the singer Helen Merrill at the club in 1960. That year he also toured Europe with Norman Granz's Jazz at the Philharmonic, described as the first European musician to take part in one of its tours.

In 1961 Johansson moved to Stockholm and joined the orchestra of the alto saxophonist and clarinetist Arne Domnérus, an association that lasted until his death. He settled with his family in Upplands Väsby in 1962, led trios with the bassist Georg Riedel and drummer Egil Johansen and later with Riedel and the guitarist Rune Gustafsson, played in Harry Arnold's radio big band until 1965, and became a leading member of Radiojazzgruppen, the Swedish Radio jazz ensemble, for which he wrote and arranged much of the repertoire, often with Riedel. His trio albums 8 bitar Johansson (1961) and Innertrio (1962) came out on the Megafon label, and the first was named best Swedish jazz record of its year.

His best-known work grew out of his interest in folk music. Jazz på svenska (1964), a duo with Riedel built on Swedish folk melodies, became the best-selling jazz album in Swedish history, with more than 400,000 copies sold, and has stayed in print ever since. He followed it with Jazz på ryska (1967), built on Russian songs, and Jazz på ungerska (1968), recorded with the violinist Svend Asmussen. Other records of the 1960s include In Pleno (1964), Barnkammarmusik with the trombonist Bengt-Arne Wallin (1966), a live set from the 1966 Tallinn jazz festival, and the multi-album Musik genom fyra sekler. He also wrote for film, television, ballet and the stage, including music for Bo Widerberg's film Barnvagnen (1963), the score for Skrållan, Ruskprick och Knorrhane (1967), dance works with the choreographer Lia Schubert, and the theme tune of the television series Pippi Långstrump, with lyrics by Astrid Lindgren.

Johansson was killed in a car accident near Sollentuna on November 9, 1968, on his way from home to Stockholm Central Station to meet colleagues travelling to a concert in Jönköping. He was 37. He was buried on December 4, 1968, at Skogskyrkogården in Stockholm, and a memorial concert that month raised money for a scholarship fund in his name, which the Swedish Musicians' Union still administers; the bassist Palle Danielsson was among its later winners. Several Grammis awards went to albums released after his death, he received the Django d'Or "Legend of Jazz" award in 2000, and he was inducted into the Swedish Music Hall of Fame in 2014. His sons Anders and Jens Johansson, a drummer and a keyboardist best known in rock and metal, founded Heptagon Records in 1994, which has kept his recordings in print, and a biography by Erik Kjellberg appeared in 1998. In 2016 the pianist Jan Lundgren recorded a live tribute album, The Ystad Concert.

Johansson's playing joined a clear, singing melodic line to a classical sense of touch, and he is best remembered for treating Swedish folk tunes as jazz material without decorating them: on Jazz på svenska the whole record is his piano against Georg Riedel's bass, and on tunes such as "Visa från Utanmyra" a reviewer hears a more classical sound in his playing, set against Riedel's tough, almost funky acoustic bass. A review of the later In Hamburg collection hears the same folk influence in his big-band writing and calls it the thing that gave his work its own sound. Stan Getz valued the lyrical side of his playing, and his early records are in the swing and bebop language he learned as a teenager, but through the 1960s he also moved toward modal post-bop, third stream writing and freer forms, and wrote steadily for radio, television and film. AllMusic ranks him among Sweden's top jazz pianists of the 1950s and 1960s, noting that he was little known in the United States. His folk-jazz records have had a long afterlife in Sweden through the Heptagon reissues and tributes such as Jan Lundgren's 2016 concert recording.