I got really worried when you brought this up, because I didn't check even think to check. However, I have verified that both Firefox and IE will NOT cache images between servers.
That's a good idea, but I'm just using these icons along with some HTML/CSS art for a web app. I'm not shooting for the most accurate results; I just need to know which server can request/respond quicker (it's an AJAX app). If anyone wants the code, though, I may make it available with small, medium, and large images.
My VSP only has 128 kB of RAM. Does anyone know if this would greatly affect network speeds? My PHP code executes fine, but I'm guessing that having 50 socket connections established one-after-the-other might slow things down. If not, maybe I should consider switching hosts like you said, but I just had a breakdown this week after migrating to the new one!