Rafi Malkiel was born and raised in Jerusalem, the child of parents who had immigrated to Israel from the Moroccan city of Meknes. He grew up hearing Egyptian and Lebanese songs alongside the Middle Eastern devotional music of his neighborhood, a mix he would draw on directly decades later as a composer. He trained first as a classical musician before discovering jazz, and then Latin music, which he has said he "liked a lot." Between 1985 and 1997 he held a series of music scholarships from the American-Israeli Cultural Foundation. He moved to New York in the mid-1990s to study music and never left.

In New York, Malkiel earned a Bachelor of Fine Arts from The New School, a Master of Music from the Manhattan School of Music, and later a doctorate from Stony Brook University's Department of Music. He built his career simultaneously in two overlapping New York scenes: straight-ahead and Latin jazz. He became, in the words of one Colombian newspaper, "el virtuoso judío del bombardino" ("the Jewish virtuoso of the euphonium"), and worked as a sideman with salsa bandleader Willie Colón, pianist Jason Lindner's Big Band, trombonist Ray Anderson, and Grammy-nominated Colombian singer Toto la Momposina, and took part in bassist Reggie Workman's live restaging of John Coltrane's Africa/Brass. He also worked steadily in New York's salsa scene, recording and performing over the 2000s and 2010s with Wayne Gorbea's Salsa Picante, the Williamsburg Salsa Orchestra, and pianist Hector Martignon's Banda Grande, and played in the Broadway pit orchestras of The Color Purple, A Chorus Line, and the Tony Award-winning Urinetown. In Israel, he is credited on the soundtrack of Souvenirs, the documentary that won first place at the Doc Aviv festival and the Israeli Academy Award (Ophir) for Best Documentary in 2006.

Malkiel formed the Rafi Malkiel Ensemble/Quintet in 2003 to play his own compositions and his arrangements of Jazz and Latin American standards for Afro-Caribbean instrumentation, and the group has performed at venues including the 92nd Street Y, the Kimmel Center in Philadelphia, and the Corcoran Gallery of Art in Washington, D.C., as well as at the JVC Jazz Festival, Lincoln Center Out of Doors, SummerStage, the Sydney Festival, WOMAD festivals in Greece, New Zealand, and Australia, and the Red Sea International Jazz Festival in Israel. His debut album as a leader, My Island (2007, Raftone), featured bassist Andy González, percussionist Anthony Carrillo, tuba player Howard Johnson, and clarinetist Anat Cohen. His second, Water (2010), was made at John Zorn's invitation for Zorn's Tzadik label, in the Radical Jewish Culture series; Malkiel later said he initially balked at the assignment ("I don't know how to do Jewish music"), then built the record instead around water as a universal theme running through Jewish and other cultures alike, complete with an instrument of his own invention, the "aguaphonium" (a garden hose fitted with a mouthpiece and a funnel, blown into a tub of water). Water brought together players from Malkiel's overlapping worlds, including his wife, bassoonist Gili Sharett, saxophonist and bass clarinetist Chris Karlic, flutist Itai Kriss, pianist Pablo Mayor, drummer Daniel Freedman, clarinetist Anat Cohen, and trumpeter Avishai Cohen (trumpeter), with Zorn as executive producer.

Since the mid-2010s, Malkiel has been a regular member of pianist Arturo O'Farrill's Grammy-winning Afro Latin Jazz Orchestra, appearing on a run of the group's albums for Motéma, ZOHO, and Blue Note. In February 2023 he shared in the orchestra's Grammy Award for Best Latin Jazz Album for Fandango at the Wall in New York (2022), recorded with the orchestra and the Colectivo de Son Jarocho Conga Patria, an award Stony Brook University's music department publicly recognized him for as an alumnus. He continues to divide his time between performing, composing, and teaching younger musicians.

Malkiel's music runs Latin jazz, straight-ahead jazz, and the Middle Eastern and North African sounds of his Jerusalem childhood through a single, carefully arranged voice rather than keeping the idioms separate. On Water, a single track can move from an Egyptian-derived, oud-inflected melody to a klezmer-tinged shuffle to an Afro-Cuban groove without the transitions sounding forced; critic Alexander Gelfand, writing in Tablet Magazine, described the album as tied together throughout by "a guiding musical intelligence" expressed in sophisticated arranging and "catchy melodies that never seem to resolve where you expect." Malkiel has framed the underlying logic himself: "Jewish people live all over, and everywhere they live, their culture is mixed with the [local] tradition," which he has taken as license to treat jazz, Latin American, classical, and Jewish diasporic material as one continuous vocabulary rather than a set of separate boxes to check. As a trombonist and euphonium player he is comfortable both as a blending big-band section voice, in Arturo O'Farrill's Afro Latin Jazz Orchestra, and as a soloist and bandleader building his own small-group arrangements around Afro-Caribbean rhythm sections.