>Can anyone help me figuring out why my robots.txt file did not work? I submitted it to alltheweb SE and all of my pages had been indexed.
You might need to add the name of the spider for AllTheWeb - not sure exactly what it's called, but here are names for some of the other search engines.... Maybe try FAST - under the impression that they are connected with AllTheWeb in some way???!!!
AltaVista - Scooter
Hotbot/AOL/Goto/MSN/Iwon/Looksmart - Slurp
Excite - Architext Spider
Google - Googlebot
Fast - Fast