How Forking Perl 5 Could Work


Stevan Little (the man behind Moose) gave a talk at the Orlando Perl Workshop called Perl is not Dead, it is a Dead End. The talk culminated with an announcement of an experimental reimplementation of the useful parts of Perl 5 in Scala, a project called Moe.

This is not the first fork or pseudo-fork of Perl 5. The Topaz project eventually begat Perl 6, which begat Parrot (the way I understand Parrot today is that it's what you get if you try to write a virtual machine in C to run Perl 5.006 effectively, and then check in a lot of suboptimal code hoping that magical elves will somehow coalesce out of the luminescent aether to fix your mess). Later on came Kurila which threw out some of the awkward parts of Perl 5 (and some of the useful parts) in an attempt to gain more speed. Lately, something called Perl 11 looks like an attempt funded by CPanel to add optional optimization levels to a pre-5.10 Perls. (I suspect the people behind Perl 11 will object to this characterization, but I find the lack of specificity frustrating.)

Now comes Moe.

Will it succeed? I don't know. Before anyone can address that question responsibly, you must understand what a fork can or should fix.

What's Wrong with Perl 5?

Perl 5 has two main problems: its implementation and its implementation. Any time you think you've found another problem, look deeper and you'll discover it's one of those two problems (and very likely "its implementation" is the culprit").

The Perl 5 VM is a big wad of accreted code that, in some places, traces its lineage back to Perl 1. It's exactly what you'd expect from code written in the era where "make it fast" meant "Write it in C, come hell or high water" and it shows. Where a Smalltalk might be implemented in something like Slang or PyPy in RPython, Perl 5 doesn't do that.

That choice almost certainly made sense in 1993. By the time of Topaz in 1999, C made less sense. By the time of Parrot in 2001, C made little sense. In 2013, C makes almost no sense.

Bear in mind that my argument is "Writing the whole thing in C makes little sense for a language larger than Lua".

Why moe Might Fail

"Almost as good as Perl 5" isn't that compelling. If I wanted to use a faster and better language with worse deployment and fewer libraries, I'd write more Haskell.

Perl 5 has no language designer with singular vision and the time and taste and experience to shape the language in a cohesive gestalt. (That's probably the worst part about Perl 6—it took Larry's attention away from a working product to something which so far has not delivered anything usable in and of itself.)

Technical reasons, like "Wow, the JVM isn't really a good platform for a Perl!" or "The subset of Perl 5 that's practical to implement is basically Groovy and that already exists." I'm not predicting these specific cases, mind you. I offer them as examples of technical reasons which may exist. (Even though I suspect the JVM really isn't a good platform for a Perl.)

Social reasons, like "This is more work than we thought, and it depends entirely on volunteer labor." (That excuse worked for Perl 6 for a while between the time TPF stopped giving $50k grants and the Ian Hague "Get this in a usable state in the next couple of years to attract more grant money!" grants didn't achieve their goals.)

Can moe Deliver?

It's possible moe can work. It has to avoid two traps:

  • Falling into a local maxima because of the limitations of the underlying technology. (It would be mean of me to call this the Rakudo-on-other-VMs Trap, so I won't.) For example, it is so exceedingly difficult to implement a language with decent performance when the semantics of how you use memory and where you get that memory and how you release it and when you release it are different from the assumptions that the VM and its optimizer and any JIT and tooling and extensions expect that you would have to be the combination of Michael Abrash, Mike Pall, John Carmack, Cliff Click, and quite possibly Pablo Picasso to make it work well across multiple VMs.
  • Ossifying into something that can't change before it produces sufficient utility. (It would be mean of me to call this the Parrot Support Policy trap, but at someone who argued both sides of that support policy at various times, it's one of the most important reasons why Parrot and Rakudo locked into their whirlpool of mutually irrelevant destruction.) The best general use projects I've seen have found themselves extracted from specific situations only at the point where the specific project can support the necessarily generalization and where there's enough external knowledge about the needs of the extracted process that such generalization is possible.

In other words, it's a mistake to commit to supporting internal details until you're certain that those internal details will remain in place without reducing or removing your ability to make necessary changes for future improvements. Both Perl 5 and Parrot fell into this trap.

What moe Could Produce

If I were to implement a language now, I'd write a very minimal core suitable for bootstrapping. (Yes, I know that in theory this is what NQP or whatever it's called these days in Rakudo is supposed to provide, but unless NQP has changed dramatically, that's not what it is.) Think of a handful of ops. Think very low level. (Think something a little higher than the universal Turing machine and the lambda calculus and maybe a little bit more VMmy than a good Forth implementation, and you have it.)

If you've come up with something that can replace XS, stop. You're there. Do not continue. That's what you need.

Then I'd figure out some sort of intermediate tree- or graph-based structure suitable to represent the language as I'd like to implement it. (Rakudo has a decent implementation here. If it had gone into Parrot five or six years ago, the world would be a different place.)

Then I'd produce a metaprogramming system, something of a cross between Moose's MOP and the Perl 6 metamodel. (Rakudo gets this very right. Credit where it's due. If Parrot had managed to adopt this in 2011... well, that's a long rant for another time.)

With those three things, I believe it's possible to build all of the language you need. If you're clever—if you're careful—you can even produce a system where it's possible to create and modify a sublanguage through metaprogramming but limit the scope of those sublanguages to specific lexical scopes in your system. (That idea is probably the idea that Perl 6 the language gets most correct. It's also very Lispy, in the sense that it's tractable in Lisp, but fortunately for the rest of us programmers, Perl actually has syntax.)

Figuring out a bytecode system is mostly irrelevant. (Freezing bytecode too early cost Parrot a lot of time and energy.) Figuring out a replacement for XS is essential (everyone says that Perl 5's parser is the biggest barrier to alternate implementations, but the utter dependence of the CPAN on XS and the haunted Jenga horrors of XS and the Perl 5 internals is the biggest barrier to adoption of alternate implementations).

Breaking the dependence of the CPAN on the Perl 5 internals—even in a small way—while allowing the evolution of the language and the annealing of the implementation over the specification toward completeness (annealing in the AI sense, not necessarily metallurgy) may be a viable path to producing a Perl 5.20 which allows optional backwards compatibility if you want it and usable new features if you need them.

Notice that this plan ties the implementation of an alternate Perl 5 to no one specific backend. I suspect that RPython will demonstrate that it's workable, while I'm tempted to suggest that LuaJIT has possibilities. (Again, I think it's 70% likely that the JVM and the CLR will prove themselves workable for the first half of implementation and completely bizarro-land useless for the second 80%, but Rakudo will demonstrate that soon enough.) I don't know about JavaScript VMs.

Is this worth doing?

Hey, I've paid my dues trying to implement a modern VM and modern Perl implementation.

The real question is whether an alternate implementation of Perl 5 can demonstrate its value for real programs before's money runs out keeping Perl 5 on life support. It's clear that the current Perl 5 implementation will never get the kind of attention it needs to introduce it to the 21st century, and it's pretty clear that no Perl 6 implementation right now is anything other than years away from being useful, let alone of interoperating with a Perl 5 implementation.

I don't think Perl 5 is flirting with irrelevance. I do think that every year that goes by with Perl 5 ossifying further in implementation makes it less likely that the necessary changes will happen. (The code doesn't get much cleaner, the likely implementers get busier and less interested, and the fashion-driven Silicon Valley marketing machine keeps vomiting out new trends you absolutely must adopt right now, you creeping dinosaur, which is a distraction of sorts.)

If the CPAN has proven anything, it's that one size doesn't always fit every program. Maybe the p5p model of trying to please everyone (and generally only pleasing sysadmins with Perl 4 code they reluctantly last updated in 1993 only because they started it in 1987) doesn't fit all either, and maybe an alternate implementation of Perl 5 will produce a viable model to reinvent Perl 5's internals.


Could you explain in more detail how is the way nqp/rakudo/Perl 6 manages memory different from how JVM expects you to do it?

Maybe instead of writing a new language and then port our libraries to it we should choose some existing language and start porting CPAN to it. I personally don't care that much about the syntax - it is the people and the surrounding infrastructure that is the biggest value of Perl5.

On a second thought - yeah - that will not work.

That's a good idea. I'll work on an entry about the relationship between speed, optimization, garbage collection, and language design.

I have a question. Is there a Perl 6 specification? Has the "apocalypse" / "exegesis" stuff run it's course? I look at Python, which has a really nice specification, and see what came of it. There are multiple implementations of Python including ones written in python itself (9 according to wikipedia)! I am just curious where that part of the process is. Their webpage states:

"Many parts of the language are rather mature, for example the object system and large parts of the syntax. Other parts, like concurrency, are still in flux, and wait for an implementation to try them out."

I understand that Parrot/Pugs/Rakudo/NQP are all quite complicated things, but the spec itself "should" be something that can be finished, right?

I wish the specification just came out with the requirements, instead of waiting to see what is possible from potential implementations. It seems like Perl6 could have a much stronger statement toward completion if they could say "here is a complete spec, and here is a test suite, now go build some implementations".

So, I started writing a response, and it went far too long. I've posted it here as a longer explanation about my opinions on the JVM's suitability for hosting a language as dynamic and flexible as Perl. I draw from my experience with Clojure.

Moe certainly has a mountain range to hurdle, but I believe that using the JVM will be an asset, not a liability.

That is, Perl 6 is highly experimental in some areas, but they don't want to end up a research language or hobbled by a bad initial specification.

They have taken the sensible (yet maddening) approach of not specifying portions of the spec that are likely to change after people start using them and learning the problems and edge cases.

This seems to be their way to fight eventual irrelevancy through bad implementation. Unfortunately, it does lead to some level of *present day* irrelevancy through extended development periods and not being "done".

In the end, if they end up with a well defined language that gets a good amount of use, I doubt anyone will complain too loudly about how long it took, but the path to that is hard.

P.S. Rakudo is just testing out some concurrency designs. They have green threads working now, and underlying support ready for OS threads, and now they get to test out their ideas of concurrency, autothreading, etc. I've been waiting for this point for a long time. Here's some info for someone doing a bit of work on threading in Perl 6:

The idea of implementing a core with something fast and then accreting features with something higher-level seems optimized for language experimentation, not for user experience. Isn't that what happened to Perl 6? Larry optimized it for his job: Language design.

The small core idea seems to have worked out well for the Squeak user experience. Then again, it's the same idea behind Dis and Inferno in Plan 9 where it didn't end up as popular.

I'm overstating things a little bit, but in one sense XS is the implementation of Perl 5.

Haskell has done well adding features w/o usually changing the underlying computing model; in fact, not changing the computing model is one of their guides for acceptable new language features. It's not a hard rule, tho.

OTOH their core model was created by and for mathematicians, so its orthogonality puts even Python to shame. With that foundation, adding syntactic sugar is reasonably most of your activity.

I'm drawing a stricter line between the computing model of a bootstrapping VM and the computing model of a language built on that. If anything blurs that model, Perl 5 is an example.

That line is, in practice, much blurrier than I make it sound--you're absolutely right about that, but I do think that Squeak/Slang demonstrate that it's possible.

Well, you're obviously right in one sense with Perl 5 insofar as modules have gone beyond what perlapi and the language offer out of the box, thus giving us things that p5p shied away from (sometimes with good reason), and without having to recompile the core.

The bad thing is when the core has to change for other reasons, and those modules break. Method::Signatures? PadWalker? EndOfScope? Coro? The core supports what these people wanted -- not always well, but enough -- and they made it happen. And now they're effectively part of the language. I wouldn't want Perl without M::S and Coro and AnyEvent and a dozen other modules that really are effectively Perl 5+.

But this comes at a price. The failure modes are second class citizen status, bit rot (when the core devs don't care about breaking the modules), and fossilized internals (when they do).

Modern Perl: The Book

cover image for Modern Perl: the book

The best Perl Programmers read Modern Perl: The Book.

sponsored by the How to Make a Smoothie guide



About this Entry

This page contains a single entry by chromatic published on January 13, 2013 2:29 PM.

The Incessant Language Propaganda Horse Race was the previous entry in this blog.

Wrapping Existing Code without Anonymous Functions is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

Powered by the Perl programming language

what is programming?