Leaving aside issues of language design and the emergence of other languages, it's interesting to think about other reasons why Perl lost popularity. Some of you know this history better than I do, but I think that it's now unknown to most HN readers.
The enormous reason that I see is the insistence, from Larry Wall and others, on a bottom-up "community" transition from Perl 5 to Perl 6. The design process for Perl 6 was announced at a Perl conference in 2000 [1]; 15 years later, almost every Perl user was still using Perl 5. The inability of the Perl community to push forward collectively in a timely way should be taken by every other language community as a cautionary tale.
Tim O'Reilly made a secondary point that may also be important. For a long time, Perl books were O'Reilly's biggest sellers. But the authors of those titles didn't act on his suggestion that they write a "Perl for the Web" book (really a Perl-for-CGI book). Books like that eventually came, but the refusal of leading authors to write such a book may have made it easier for PHP to get a foothold.
[1] https://en.wikipedia.org/wiki/Raku_(programming_language)#Hi...
I had lots of experience writing Perl5 before the company switched to Python3.
> The inability of the Perl community to push forward collectively in a timely way should be taken by every other language community as a cautionary tale.
I think this is a good point that I hadn't considered before.
I think Perl stopped being able to attract new users. There is always going to be users leaving. If they aren't replaced, you will slowly shrink.
I think the point you raised is part of why they couldn't attract new users. I also think people asked themselves "why chose perl now, if I know I need to re-write when Perl6 comes?" and decided Perl5 was bad choice. I also think the fact Perl had this reputation for being ugly, difficult, and "write only line noise" kept people from even considering it, even if that reputation didn't match production codebases.
I just had to check if it even existed because I was sure that I had a CGI book that focused on Perl from O'Reilly in the late 90's, and sure enough, the book I had was published in 1996 (with a second edition released in 2000).
Not saying your anecdote is inaccurate, but my perception around that time was that "Learn PHP in 24 Hours" was a lot hotter than O'Reilly's Perl books - so it may have just been luck, marketing, a flashier title, or even just that PHP was better suited for what people wanted to learn and do.
perl5 -> php3 here. I'd say PHP won for the following reasons: 1. Easier to set up. 2. More homogeneous environment which meant easier deployment across the hosts of the day. 3. More secure (a bit). 4. Unicode actually worked. 5. Far more logical syntax: did not have to man perldsc[0] every time you wanted a data structure of greater depth than a monodimensional hash or array. 6. Language features like scalars which made programming easier. 7. Often less files... it was designed to be embedded inside of HTML instead of generating it.
I think Perl died from a combination of three factors:
1. The rise of other web languages that did what Perl was good at but better.
Perl was probably a fine language for sysadmins doing some text munging. Then it became one of the first languages of the web thanks to Apache and mod_perl, but it was arguably never great at that. Once PHP, Python, Ruby, and (eventually) JavaScript showed up, Perl's significant deficiencies for large maintainable codebases made it very hard to compete.
In many cases, a new, better language can't outcompete an old entrenched one. The old language has an ecosystem and users really don't like rewriting programs, so that gives it a significant competitive advantage.
But during the early rise of the web, there was so much new code being written that that advantage evoporated. During the dot com boom, there were thousands of startups and millions of lines of brand new code being written. In that rare greenfield environment, newer languages had a more even playing field.
2. Perl 6 leaving its users behind.
As a language maintainer, I feel in my bones how intense the desire is to break with the past and Do Things Right This Time. And I'm maintaining a language (Dart) that is relatively new and wart-free compared to Perl. So I can't entirely blame Wall for treating Perl 6 as a blank check to try out every new idea under the sun.
But the problem is that the more changes you make to the language, the farther you pull away from your users and their programs. If they can't stay with you, you both die. They lose an active maintainer for the core tools they rely on. And you lose all of their labor building and maintaining the ecosystem of packages everyone relies on.
A racecar might go a lot faster if it jettisons all the weight of its fuel tank, but it's not going to faster for long.
I think the Perl 6 / Raku folks go so excited to make a new language that they forgot to bring their users with them. They ran ahead and left them behind.
3. A wildly dynamic language.
If you want to evolve a language and keep the ecosystem with you while you do it, then all of that code needs to be constantly migrated to the new language's syntax and semantics. Hand-migrating is nightmarishly costly. Look at Python 3.
It's much more tractable if you can do most of that migration automatically using tools. In order to do that, the tools need to be able to reason in detail about the semantics of a program just using static analysis. You can't rely on dynamic analysis (i.e. running the code and seeing what it does) because it's just not thorough enough to be safe to rely on for large-scale changes to source code.
Obviously, static types help a lot there. Perl 5 not only doesn't have those, but you can't even parse a Perl program without running Perl code. It is a fiendishly hard language to statically analyze and understand the semantics of.
So even if the Perl 6 folks wanted to bring the Perl 5 ecosystem with them, doing so would have been extremely challenging.
I would say this is a case study in a programming language tragedy, but I honestly don't even know if it's a bad thing. It may be that programming languages should have a life cycle that ends in them eventually being replaced entirely by different languages. Perhaps Perl's time had simply come.
I am grateful for all of the innovative work folks have done on Perl 6 and Raku. It's a cornucopia of interesting programming language ideas that other languages will be nibbling on for decades.
Perl was the perfect language when the majority of people wanting to use it already knew shell, sed, awk, and C.
It succeeded because it was a beautiful and horrible combination of those tools/languages. If you know those things, Perl is really easy to bang together and generate a webpage or use to automate administrative task.
Given Larry Wall's education, perhaps we shouldn't be surprised that Perl underwent a linguistic evolution from pidgin[1] to creole[2].
I came to Perl a bit late, not being a super adept user of Unix and only having written C in school. I'd put myself in the category of Perl developer that was part of the second "creole language" phase of Perl's development. I learned a ton of Unix and got better at C by learning Perl.
While Perl's mixed nature made it successful, when the world of web development expanded to include more people without the necessary background to benefit from the admixture, it went from asset to hinderance. All that syntax went from instantly familiar to bizarre. A perfect example of this are the file test operators[3].
The Perl 6 struggles definitely added to the difficulties posed by the changing nature of the web dev community. They created enough uncertainty that tons of people asked themselves "Why should I learn Perl 5 when Perl 6 is just around the corner?". That slowed adoption from in the 2001-2010 timeframe while Python and Ruby grew rapidly.
Rakulang is kind of magical. Writing it is like using a language given to us by aliens. I wish it was getting more uptake because it is fun and truly mind expanding to write.
In the end, I think the shifting nature of the community was a larger factor in the decline of Perl than the slow Perl 6 rollout and failure of messaging around it.
I still love writing Perl 5 and I wish I got to do more of it.
[1] https://en.wikipedia.org/wiki/Pidgin [2] https://en.wikipedia.org/wiki/Creole_language [3] https://perldoc.perl.org/functions/-X