You say "is turned off" but what do you mean? You can do this when the car is off? When it's on? Both?
On multi-gear transmissions, you need to make the next set of gears (what you're changing into) match the speed of the output shaft, whether you're going up or down. Autos use the fluid coupling of the torque converter to do this. Most manual transmissions use something called synchromesh along with the clutch. These "synchros" drag a little bit going into gear each time and speed up the shafts to make them match. They are designed to be used with the clutch during shifts. If you match the revs really well, they don't come into play, and you don't need clutch or drag on the synchros to make a shift.
However, if you don't get it exactly right, it will still go into gear, but will cause extra drag and wear on the synchros. Over time, worn synchros will show themselves with a "graunch" going into gear.
So you can do it. I've done it. Lots of people do it. But on synchromesh transmissions (all road-going manual transmissions in Chryslers that we're talking about here), you risk accelerated wear.
For the sake of completeness, note that non-synchro boxes ("dog" boxes), the rules are different. Primarily for racing, they can wear faster if you use the clutch too much.
