OK, this isn't really a fix, but for those of you that still have trim in good shape that just needs to be cleaned up, I have found a pretty good process.
Below is a picture of what I've got to deal with. Most of the trim is like this with a lot of contamination. It almost looks like white overspray.
This one is above the window and shows some of the marring that I have in spots. This particular area was from when I bought the car last year and I got some wax on the trim and it got marred taking it off. On the left end by the water spots you can see the marring. That was when I found how easy this trim scratches.
On to the process. Last weekend I did a test spot with a clay bar to remove the contamination on the trim. It worked great, but obviously marred the trim. Then I coated it with some liquid wax. It looked much better, but the marring was pretty evident. Then tonight I had an idea, and I rubbed the area good with a microfiber and some XMT Glaze. Glaze is like a very very light polish with fillers as well as a wax (though with very little durability), so I'm guessing it may have polished the finish some as well as filled in the remaining scratches as well as left a nice shine. It'* not showroom new, but looks 100% better.