Vitt played four seasons as a linebacker for Towson State (now known as Towson University) near Baltimore, even though he was an undersized 5'10" and measly 190 pounds. He made his first NFL appearance with the Baltimore Colts in 1979, as their strength coach.
Vitt was the Seattle Seahawks' strength coach when Chuck Knox came to be head coach in 1983. He quickly promoted Vitt to defensive backs coach. Vitt moved with Knox to the Los Angeles Rams, where he worked, along with Mike Martz, on his staff from 1992 to 1994.
Vitt has also been an assistant for the Philadelphia Eagles and Green Bay Packers. He served under former St. Louis Rams coach Dick Vermeil for the Kansas City Chiefs for three years until Martz brought him to St. Louis as the assistant head coach and linebackers coach; it marked Vitt's eighth time in the National Football League, and the second with the franchise. During the 2005 NFL season, Vitt served as the interim Head Coach of the Rams while Martz was out due to a bacterial heart infection. He coached the team from Week 5 until the end of the regular season; soon afterward, Martz was fired. Vitt had a record of 4-7, and was replaced by Scott Linehan in the off-season.
The New York Jets reportedly had interest in bringing in Vitt as their head coach after Herman Edwards signed with the Kansas City Chiefs before hiring Eric Mangini. Instead, Vitt was hired by the New Orleans Saints on January 27, 2006 to serve as their assistant head coach/linebackers coach.