Here’s three different reasons.
One reason for why you shouldn’t dress “better” or “worse” is because when you dress better, you’re doing it to impress other people. If you’re dressing to impress, then you’re spending time trying to change yourself in order to seek the approval of other people. And this is generally a bad life idea – shouldn’t people value you for your personality, intellect, and skills, instead something as superficial as the way that you look? As a result, you should dress in a way which does not seek the approval, nor disapproval of other people. The latter is included because contrarians go so predictably against the grain that they have conformist tastes. You should thus dress carelessly but appropriate. Wear clothes, for one, where other people do: do not, for example, wear your shorts as a shirt, nor should you wear your shoes on your head, so on and so forth. Other than that, sort your shirts and pants in a random order such that you have as little control or responsibility for your dress as possible. In this way would you maximally avoid dressing for others.
But there is another reason. There are times when you do dress up, not primarily to seek the approval of other people, but in order to pay your “respects”. For example, at funerals, people don’t wear wear casual clothes or even flashy clothes. And when I was growing up, when I went to church on Sunday, I would always dress in a shirt and dress pants. It was not because there was anyone special at mass, but because you show your best to God. Although now that I think about it, this too is an argumentum ad populem because I guess you’re dressing to impress God, and in a room of other people who believe the same thing and thus dress well too, you are still doing it to impress others.
Are there thus times when you should dress up even if nobody will notice you? If answered in the affirmative, then one finds a reason to dress up that is (99.9%) free from social pressures. And I think I found one. You dress up if you respect yourself and what you do while you’re dressed up. I’ve found, for example, that if you dress up then you take whatever it is more seriously, even if nobody else is around, just because you feel like you need to “live up to” the clothes you are wearing. You need to earn the right to wear good clothes, and in so doing, you show that you respect yourself. Because while clothes show others what your social status is, they also show people that you’re serious. Would you hire an accountant who looked homeless? Personally I would, because money doesn’t care about what an account looks like, but I think it would show to me that you care more about the subject at hand. That’s simply what I see when others dress up.
I might also add that by “dress better” I mean dressing more formally. This is because “fashion” as it passes for people my age looks childish, unkempt, and sloppy.
So as a result, I think that you should dress better primarily if you take whatever it is that you’re doing seriously. This doesn’t mean that you can’t be serious without dressing better, or that dressing better will make you do things more seriously (though it does help you do just this), or that you should judge a man by how formal his clothes are. But it does mean that your clothes will always reflect the character of the person wearing them. Wearing no clothes definitely shows that you’re some type of character, wearing clothes randomly shows you’re incoordinated and ignorable, but wearing clothing you’ve paid a sliver of attention to shows that you’re someone who cares about who you are and the image you see for yourself. Wearing socially-independent clothing is an impossibility.