The aluminum was anodized. This will describe the process
Anodizing - Wikipedia
What happens is over the years, the anodized coating just gets dull and you can't bring the finish back, as you have found out.
In an ideal world, you would strip and re-anodize, but it's not as practical or easy as it sounds, so the pragmatic solution for our cars is to strip the anodize off and buff the aluminum. The only issue with that is now you have to wax the aluminum to give it some protection so it doesn't dull as there is nothing to protect it.
You can use oven cleaner to remove the anodize coating, but you have to use the real nasty stuff with lye in it. Not just any oven cleaner. I haven't done it in a few years now, so I can't tell you what to buy now, but the best stuff then was the Dollar store cheap "not safe to breath" type cleaner.
Then you have to buff the aluminum.... And that's really something that would be extremely hard to do by hand. I've buffed aluminum with a wheel and buffing compound and there's a bit of investment in a buffer (Harbor Freight has a decent one) and then there's the wheels and compounds. I suggest researching that first before you strip the anodizing. I buy my stuff from
Polishing - Caswell Inc but Eastwood has some nice, but slightly overpriced kits, like this one.
Eastwood Buffing Shop Kit