I think that the reason is because inexperienced developer usually looks for just a solution. When he thinks he found it, he usually rushes to implement it so he can think about a solution for the next problem, and so on. Well, then he realises that such a way of developing software is just not very productive. Then he reads a lot of books and articles and finally understands that he has to design something before implement.
Okay, so any good developer usually finds it out on his own. So why am I talking about it here then? Because there is actually much more to the thinking before implementing than just design. Here is what am I talking about. Suppose you have to develop some kind of network application. Inexperienced developer will probably rush off to write code right away and end up with big unmaintainable monster with screwed up network protocol. More experienced developer would think about how and what exactly the application should do and implement it only then. So he has some global idea about what is going on, so his application tend to be a much better one than in the first case.
Now get to the point. For some reason, "a much better one" is often far from "the best" and it tends to become worse with the time. There may be a lot of reasons, actually, such as a lack of time, developer's stupidity and such. But one of those reasons is often because of the misunderstanding this "think before implement" concept. Well, I would even say "over-complicating", because it is actually simpler than it seems. When one reads a book about "software design", he often thinks that the book contains some truths that he must know in order to design good software. Well, in some way it is true, but what people often forget is that no amount of knowledge can replace a good thinking! Enough of theorising, however. Let us go back to the example.
Suppose we have this application designed well on the global level. We know what and how it must do. And we start to implement it. But we do not live in ideal word, so ideas often collide with harsh reality. So we may have, for example, two C++ classes called "NetworkApplication" and "ClientPart". The first is a general control class that is supposed to control everything. The second is a client protocol implementation. Both are well designed and well implemented. But then we start to connect them together and find some collisions. For example, different parts of NetworkApplication (for example, "beginSession" and "resumeSession" functions) need slightly different implementations of the same operation in the ClientPart, say, "connect". But developer already designed and implemented both classes, so when he faces this problem he thinks of it as minor design problem and carelessly makes some dirty patch. For example, he implements a some kind of wrapper over ClientPart's "connect" in the NetworkApplication, called "resumeConnect". Or he implements it in the ClientPart, it does not matter. What matters here that he forgets to think before doing so.
What happens next usually is that code tends to become more and more messy over time and people start to blame wrong design, stupid developer or whatever. The main point here is that they are right in some way about "stupid developer" - I mean that everyone are the stupidest beings in the world when they do not even try to think! It is actually not about software development only - it happens everywhere. But software development seems to be one of the areas when the smartest people tend to make this mistake too often. What could be done to prevent the code from turning into a big mess over time? Well, here is what should happen:
1. Developer finds a problem. It may not seem to be a big problem, but he must nevertheless understand that it is a problem. A good way to achieve it is to notice yourself thinking "there is something has to be done about this" - if you are thinking this way, you have encountered a problem. It may be a little, but you still have to solve it a correct way, not "just solve".
2. Once he identified a problem he should stop doing anything and start to think what exactly has to be done and how it has to be done. And be careful to not pick up the first solution, but to think a lot and decide what solution is the best. The funny thing is that the easier the problem, the harder it is to do - because people tend to think less of smaller problems or to not think at all. This is just a good way to make small problem into big one.
3. Once the best solution has been found, developer implements it. If it encounters something he did not thought of, he must go back to thinking until he is sure that this solution is still the best one, or a new best solution is found.
Remember dwarven king Loghaire Thunder Stone from the Arcanum said "Humans act first, think later and feel last of all". The correct way in the software development is first to feel that something is wrong (no matter how insignificant it may seem), then to think carefully what to do about it, and only then do it. No excuses! Even if you are thinking about such a simple thing as the name for your local variable or about whether to declare it local or field.
The important point here is to find the best solution possible. Another reason why developers tend to fix things quick and dirty is the programmers' natural laziness. Well, programmer have to be lazy to be a good programmer, but any good programmer knows that the best thing he can do being lazy is not to find a solution quickly, but to find a solution that will save him from doing a lot of boring stuff later. So better to find a good solution now, perhaps by redesigning the whole model, than to get confused by a lot of "connect" implementations later.
Another important point is to be able to find the best solution. That is what books like "The Art of Unix Programming" for.
Also, I would like to say it one more time: no amount of knowledge can replace a good thinking! One may say, "hey, that developer just did not know about thin glue rule!". Well, his fault it is, right. And it is true as well that if he knew about it, he would quickly identify his problem connecting NetworkApplication and ClientPart classes as the glue layer problem and would did his best to avoid glue bloating, thus going the right way. But the main point of this post is that no knowledge can provide the best solution to any problem. So even if he is the most knowledgeable developer in the world, he would still encounter situations when he does not have enough knowledge to solve another problem.
As I already said, it applies not only to software development. People often tend to find solutions by habit, by asking another people, but not by thinking. And the smaller the problem, the more often it happens. Why it is not so important in other areas is because small problems do not usually tend to turn into big ones in other areas. "My computer keeps crashing! - Go buy a new one!" - okay, that is a solution. If he thought about it, maybe he would find a reason why it keeps crashing and solve it without spending a few hundred dollars on a new one, by reinstalling OS, for example. But that is all! Old computer gets thrown away and it will not turn the rest of his life into nightmare by trying to take a revenge for it. Badly developed software probably will. At least until the problem is fixed the right way or until the software gets thrown away (in the worst case).
It is a really simple thing: feel, think, do. Nothing more. No need to believe in God or Satan. No need to buy expensive proprietary tools that promise you to create finest products without thinking about anything at all. You still have to know a lot of things to be able to find the best solutions, though, especially if the problem is not actually very simple one, but that is a really obvious thing - one can get away without thinking and end up with badly developed software, but there is no way one can develop something in any way if he does not know how!