Dopo aver esordito nei Flatlanders con Butch Hancock e Jimmy Gilmore, Ely debutto` con un album, Joe Ely (MCA, 1977), di country-rock innocente. Le canzoni migliori sono in realta` composte Butch Hancock, ma Ely le canta in un timbro e fraseggio aspri. Cinque brani sono invece originali.

Proprio quel canto anomalo e` la grande novita` dell'ambizioso e rivoluzionario (per la country music) Honky Tonk Masquerade (1978), dove i suoi accorati lamenti (soprattutto Because Of The Wind e quello liricissimo che da` titolo all'opera) si sposano a fisarmonica e fiati e producono un sound che e` molto piu` aggressivo, avendo assorbito rock and roll, honky-tonk, boogie e rhythm'n'blues (Fingernails, Cornbread Moon).

Ely sembra proporsi come il nuovo Gram Parsons, ma gli album Down On The Drag (1979) e Musta Notta Gotta Lotta (1981), soprattutto Crazy Lemon sul primo e Musta Notta Gotta Lotta e I Keep Gettin' Paid The Same sul secondo, esagerano la componente rock'n'roll al punto da lambire il boogie elettrico del southern-rock.

A better band-leader and rocker than songwriter or composer, Ely always had to rely on the "sound". However, the electronic arrangements and the heavy-metal guitars of High-Res (1984) pushed the idea a little too far, and little survived the wreck (What's Shaking Tonight, Cool Rockin' Loretta, She Gotta Get The Gettin', Letter To Laredo).

The humbler Lord of the Highway (Hightone, 1987) contained its dose of frenzy (Everybody Got Hammered, Are You Listenin' Lucky) but also the profound meditation of Letter to LA.

All the songs on Dig All Night (1988) were written by Ely, but this proved quite the opposite of what he meant: that Ely without the Hancock songs is not quite the same Ely (Settle For Love).

The relatively pensive Love And Danger (MCA, 1993) still contains the wild romp of Highways and Heartaches. The concept Letter To Laredo (MCA, 1995) features a flamenco guitarist and a calmer, more exotic sound. Twistin' In The Wind (MCA, 1998) uses that idea to concoct an atmospheric, evocative collection.

Ely also contributed to Los Super Seven (RCA, 1999), a supergroup effort featuring Freddy Fender, Flaco Jimenez, David Hidalgo and Cesar Rosas of Los Lobos, etc.

Streets Of Sin (Rounder, 2003), the first studio album in five years, is a moody and reflective work, despite Fightin' for My Life and 95 South. And so the likes of Wind's Gonna Blow You Away and Flood on Our Hands lack the energy to stand up to his best work.

Hancock, Joe Ely and Jimmy Gilmore reformed the Flatlanders for Hills And Valleys (2009).

Happy Songs From Rattlesnake Gulch (2007) and Satisfied at Last (2011) added very little to his achievements.