The rotating mass is a big difference for accelleration and braking. Not insignificant at all.
Cornering will be much worse. The car will not stick the corners as well with no sidewall to flex. It will be much more difficult to keep the 'patch' in contact with the road.
As you enter a corner, the car will try to make that contact patch on each wheel 'skid'. The more the sidewall can flex, the less likely you'll slip in a turn. This will result in an overly 'tight' condition.
Uneven roads are the least of your concerns. And that'* not willwren'* car pictured. Willwren'* cars are in the post BELOW that.