But why was the pugs project abandoned?

Because it wasn't working out as a project: it wasn't maintainable and it went unmaintained. Other projects were easier/simpler/faster/more rewarding to work on.

Why did Rakudo developers start from the scratch when something like pugs was already available to build upon?

Pugs wasn't the first Perl 6 implementation by any stretch of the imagination either. There had already been two implementations on Parrot at that point. Ultimately they proved to take the wrong approach as well.

The tricky thing about implementing Perl 6 that no one predicted (and I suspect that no one could have predicted) in advance is that you have to get the grammar engine correct to have any hope of implementing it well. How you do that is an interesting question. The Rakudo approach is to write just enough of a self-hosting grammar engine to bootstrap everything else. To make that work, you need enough underneath the bootstrapping level to provide sufficient bootstrapping power. Another approach is to build a simple, self-hosting metaobject system (like Cola or Squeak) and describe the grammar engine in terms of itself from there.

Parrot's biggest failing in its first five years was that it provided almost no facility to get a bare-bones Perl 6 running. Pugs succeeded in that Audrey et al built enough of a Perl 6 implementation that people could run specification tests. If that had been possible with Parrot in 2002 or 2003, things would have turned out differently.

Pugs failed as a project on its own because it required heroics from Audrey to evolve and change. If it had been more approachable, or (towards the end of its useful life) faster, or more maintainable, or (as has been the big catalyst in Rakudo in the past several months) mostly bootstrappable in Perl 6 itself, things would have turned out differently.

No one has ever written a compiler for Perl 6 before, and if you look at the specification changes especially over the past two years, you'll see that most of the changes reflect questions and requests from implementors about the specification. Look at how long it's taken projects like IronPython and IronRuby and JRuby to implement languages already implemented elsewhere (and with actual funding and full-time developers). Implementing a language is not an easy or a fast process. Python 3000 took eight and a half years from conception to its 3.0 release, and its changes were relatively minor. (Microsoft still hasn't implemented C99 in Visual Studio, though I suspect that's not a question of resources at this point.)

Sometimes it takes a few false starts to find the right design. Certainly Parrot has iterated through a few designs on the way to find better and better approaches. Some of the ideas I had several years ago turned out to be big mistakes. Some of the ideas I had several years ago worked pretty well, but then we all learned something that countered our basic assumptions. If we had known then what we know now... well, you don't often get the luxury of being able to be right about everything or even most things up front.

Fortunately, we do have the ability to make changes. It's software, after all. It's malleable. It's flexible. You can let your users use last month's release for a month or two until you make some big changes to the internals and get next month's release to the point where it does what they need it to do and you can improve it further. You work in that cycle, where you don't have to make dramatic progress every day. Sometimes you do get a thousand tests passing or you fix a very satisfying bug or you delete a hundred lines of code that's suddenly unnecessary or you find a quick fix and get a 5% performance improvement. Usually you don't have those dramatic experiences, but as long as you make progress month after month toward a very real, very achievable goal, you're doing okay.

Meanwhile, along the way, some people get sick. Some people lose their jobs. Some people have families. Some people lose families. Volunteers do what volunteers do, and the best you can do is help them be most effective when they're available and try to keep the work fun or rewarding or at least manageable. Some of them want to work on Pugs. Some don't. Some of them want to work on Rakudo. Some don't. Some of them report a bug and that's all you ever hear from them and that's fine too, because sometimes you find someone brilliant with some spare time who can take on a larger task and it's very, very satisfying to help someone else realize that you don't have to be a genius or a famous programmer or tall or handsome or male or female to contribute to a project bigger than you could do by yourself. You just need to be a little bit patient, a little bit stubborn, a lot interested, and humble enough to be willing to learn and to experiment.

If the price for all of this is that a couple of million people think you're incompetent or diabolical or arrogant or foolish for working so hard on a project that's taken so long, well, I suppose you focus on the other six and a half billion people in the world that won't care that the project took longer than you ever thought if it makes their lives a little bit easier somehow. That, and you realize that a significant portion of the handful of people who think you're the worst person in the world still benefit from things you helped invent and achieve in Perl 5.