Before his NFL career, Greene had an outstanding college football career at North Texas State University (1966-1968), assisting the team to a 23-5-1 record during his 3 seasons with them. In 1984, he was enshrined in the College Football Hall of Fame.

In 1969, he was selected by the Pittsburgh Steelers with the 4th pick of the NFL draft and spent his entire career with them until his retirement in 1981. Greene was a cornerstone of the Steel Curtain defense that won four Super Bowls in the 1970's. His best championship performance was in Super Bowl IX, when he became the first player ever to record an interception, a forced fumble, and a fumble recovery in a single Super Bowl. Greene was twice named NFL Defensive Player of the Year(1972, 1974), played in 10 Pro Bowls, and was inducted into the Pro Football Hall of Fame in 1987.

His end stats were 181 games, 66 sacks(unofficially, since sacks were not an official statistic until 1982), 16 fumble recoveries, 1 helmet smashed, & 1 football thrown into the stands in disgust.

After retiring from the NFL, Greene became an assistant coach under Steelers' head coach Chuck Noll in 1987. He spent the next 16 years as an assistant coach with the Pittsburgh Steelers, Miami Dolphins, and Arizona Cardinals. In 2004, he was named the special assistant for player personnel for the Steelers.

Greene has also appeared in commercials, the most famous of which is the one in which a child gives him a Coke, prompting "Mean" Joe to smile and give the kid his game jersey. The commercial was listed as one of the top ten commercials of all time by TV Guide magazine. This advertisement was paid homage to in 2006, in a TV ad promoting asthma awareness. In the 2006 ad, a child with asthma tosses his Jerome Bettis Steelers jersey to Jerome himself, who is also an asthma sufferer. In 1981, "The Steeler and the Pittsburgh Kid" expanded the Coke commercial into a TV movie with Greene playing himself and the kid played by Henry Thomas, who soon after starred as Elliot in E.T. the Extra-Terrestrial.