It depends on what town.
The town I live in isn't really bad but it is horrible for new people because there are a lot of one way streets. And everyone drives super crazy here.
Then St. Louis is horrible to drive in because there are so many drivers and there are parts that you really don't want to drive through because you are likely to be shot in and I always end up in them.

Plus I just hate having to pay attention to stop signs, people walking, etc.
Driving on back roads or highways/interstates you don't have to bother with them.