The non compete holds up as outlined in the sales agreement, if it's in there than absolutely he can enforce it. He better hope it's laid out clearly however or then, yes, it's endless lawyer fees and time.
We are currently in a 3yr covenant not to compete and it's laid out very clearly exactly what that means (what we cannot do) and exactly what happens if we break the non compete. If we broke the terms I believe it would be very cut and dry, we would owe our buyer X as lined out in the contract which they could very easily enforce in the appropriate court. Anyone can drag out the process of enforcement (or anything legal for that matter) but at some point it's going to wind back around to the contract.
That guy sounds like a total dirtbag inkman.