"instead of" contains "of", which should be followed by a noun. so, "instead of" should be followed by a noun (or by something that acts as a noun, such as some -ing forms).

on the other hand, "rather than" can connect just about any two similarly structured things.
nouns: I decided to purchase a truck rather than a car.
adjectives: I wish this transmission were manual, rather than automatic.
etc.
also, it can connect things that aren't technically the same grammatically but that play the same grammatical role. e.g.
it's best to read the passages quickly, rather than with precise attention to every little detail.
--> here, both the adverb "quickly" and the modifier "with precise attention..." modify the action of reading the passages. so, all good.

there's also a very slight difference in meaning. I'll bet you 99.99% that gmac will never actually test this, but ...
... "X instead of Y" means that "Y" was some sort of default / assumed/preferred outcome, but that X has been substituted for it.
... "X rather than Y" indicates no such preference; it just means that there were options X and Y, and, this time, X won.

e.g.
He paid with a check rather than cash.
--> He had two options, and chose the check.

He paid with a check instead of cash.
--> He was supposed to pay in cash, but he paid with a check.

again, the likelihood that this will ever actually be tested is pretty much nil, but, there it is.