Vision and the Perl 5 Ecosystem


Now that Dave Mitchell has released Perl 5.10.1 the bikeshedding may begin again. David Golden asked What do you want Perl 5 to be? I've suggested that people define their own visions for the future of Perl 5.

I believe this is important for a few reasons:

  • The Perl 5 community has never agreed collectively in any sense on one or two strategic visions for the language and its ecosystem.
  • Discussions about practical concerns always flow from ideas -- often implicit -- about the vision of the language and its ecosystem.
  • Everyone who has a stake in the language and its ecosystem has particular uses in mind which deserve fair hearings.
  • A project without a strong sense of vision will make progress only on the whims of people dedicated enough to perform work and get it committed.

I admit some bitterness in the last point; I have little desire to participate in arguments on p5p over proposed changes. (I'm not thrilled about excoriation for a personal Git fork of Perl 5 away from p5p.)

Yet I still maintain that all of this debate and discussion and even some of the acrimony comes from conflicting ideas of what Perl 5 is and should be.

A Digression about Multiple Implementations

I watched with some interest a project to port Perl 5 to the JVM. Unfortunately, P5 on the JVM has stalled; the developers believe the problem is currently intractable.

I won't discuss the concerns the developers have raised; that's a discussion for another context.

Yet doesn't it seem strange that Perl 5 is one of the major languages without a port to the JVM or the CLR? How about to Parrot?

Part of the problem is a lack of people knowledgeable about Perl 5 internals with both the time and interest to work on such a project. Hopefully projects such as corehackers will improve things, especially now that the 5.10.1 release has stopped consuming all of the resources of p5p.

Another part of the problem is the internals themselves, but I've mentioned that before.

The final part of the problem is perception. For all that Perl has been a great (some might say promiscuous) glue language for a couple of decades, is it only my perception that the language ecosystem has grown insular? Perhaps that's part of the external perception problem: the Perl 5 world is its own world, with its CPAN and its XS and its quirks, and it leaves the rest of the world to its own devices.

Maybe that's part of the reason Perl 5 bindings are sometimes so difficult to coax out of companies and organizations. Maybe that's why Perl 5 support in new compilers and environments and services is so slow to appear, if it ever appears.

The Vision and the Ecosystem

I wonder if we're falling afoul of a perfectionist/completionist culture, where the next stable Perl 5 release will surely be the point at which we can stop this madness of patches and changes and releases.

Yes, that's a deliberate strawman.

Tell me it's wrong, though, especially in the face of distributions which don't expect people to write modern Perl 5 with the Perl 5 installed by default. (Don't take this as a personal complaint about Red Hat; Apple has had a similar flaw for much, much longer.)

Is it possible that some people see Perl 5 as an appliance, as plumbing? It's the strange contraption under your sink that sometimes makes a knocking noise if you leave the hot water on too long and you certainly hope it doesn't break because you know it'll make a fantastic mess and you might have trouble finding someone who can fix it.

I'm not suggesting that that's true. It might not be accurate.

Yet I wonder if the disconnect between the people who want to upgrade Perl 5 on their boxes once every ten years on a Saturday afternoon between 3:50 and 4:05 pm and the people like me who want to make Perl 5 easier to learn, easier to use correctly, and more powerful in terms of abstraction and portability and optimization is pervasive throughout the community.

Answer me these questions

  • Is a purpose of CPAN for experiments in language extensions?
  • Do they converge into proposals for additions to Perl 5 itself? If so, how?
  • Does the CPAN help or hinder the desire for stability of the once-a-decade upgraders?
  • Whose needs does the CPAN meet?
  • Are those needs complementary or contradictory to the goals of various groups?
  • How does the rest of the CPAN ecosystem support or discourage these goals?

I have my own answers to those questions, but the conversation is more important than me dictating what I think you should think.

However, if you thought Perl 5 version numbers are a mess, you haven't seen the real horror. Consider ratings of the Module::Build distribution, where it's taken nine years to reach the point where some people are satisfied that a pure-Perl 5 installation system can replace a broken system which relies on your ability to use regular expressions to rewrite cross-platform shell scripts (which often need to call back into Perl 5 to emulate missing POSIX utilities) or whether people should use a DSL wrapper around MakeMaker which relies on distribution authors to copy and paste code which may have bugs (and in fact has had a couple, requiring those authors to release new versions of those distributions merely to make them installable).

If the Perl 5 community gets its once-a-decade chance to upgrade a bunch of machines next year, wouldn't it be nice to sneak in one or two Better is Better moments for a change?


"A project without a strong sense of a vision"... vision is a dangerous thing. Clarity of vision got us Lisp, Forth, and Java -- all wonderful languages in their own way, but each also severely confined. It's human nature to know, or to feel like we know. We want to know what problems we're facing, konw what solutions there are, and know which is the best. But we can't. We don't know what the real problems are -- only hindsight will reveal that. We especially don't know what the problems are going to be a year or two down the road. The best solution won't occur to us until later. And there probably isn't even a single best but instead a complex web of compromises which would remind us what we have now and where we are now but in a temporally updated form. I'm not saying that lack of vision is a good thing; but lack of a single, unified vision certainly may well be. Perhaps the odd assortment of visions, most of them small, incremental ones, are the best thing. Compared to Java, Python and other language efforts, Perl probably makes its programmers feel the least like they're being drug along with someone else's "vision", though I think Ruby is doing a good job there too.

#corehackers is brilliant. I wish I could take advantage of that right now but I'm dealing with some crap. Actually, this reminds me of when Nickolas Clark was briefly working at the same company I was -- he was micromanaged and his skills squandered. The company was obsessed with the extremely short term as they had been for ten years. Part of my recent job ranting is based on this story. Not to compare myself to Nickolas Clark, but I think a lot of us Perl programmers are stuck in a vicious cycle -- we're over-worked and micromanaged. Even if our companies wanted to invest back into Perl, they wouldn't do it by being at all generous with our time. They're not even generous with our work/life balance. They're employing Perl programmers because they work hard for what they're paid and do what they're told. This makes it hard for anyone who wants to give back to Perl to do so. Too many people in the Perl community also spend too much time between jobs. Besides being disruptive, that's unnerving and it makes it hard to concentrate on Perl itself when you do have work. We're all going to find our selves in a position pretty soon here where we're working on Java by day and Perl by night. I think this is a case of a problem got bad before people were aware of it -- the community needed the image overhaul long ago and now we're getting the bulk of the punishment for the lack of it now.

But I think there is shared vision -- p5p, XS programmers, and old fashioned lovers of Perl 5 want the core cleaned up and refactored. As for what will be done with the newly cleaned up core no one can predict, but I have faith in the distributed instinct of smart people.

You've been beating a drum to try to motivate people. You remind me of myself when I was in charge of Phoenix Perl Mongers. My message of "come to meetings, present, actually present when you say you're going to, c'mon, we can do this but I can't do it myself" sounded good on the surface but people only got the subtext of "I'm tired of people checking meetings out and I really only want regulars; there's a lack of presentations and it's your fault; I don't like giving presentations; the whole thing kind of sucks anyway". You, like (I think) myself are a lot stronger on the technical elements than the social elements. I think, despite our intentions, we're going to step on a lot of toes unless we can find more constructive ways to beat this drum. Perhaps there are other ways that involve less blood, sweat and tears, but being an example and just being helpful to others is the best approach I've seen actually work.

Yes, the community has grown insular. I've been saying that for a long time. Friends have independently come to me, in one case after having spent time in the Ruby camp, and said the same thing to me. The reaction to typesafety then later autobox was an eye-opener here -- people kept commenting that "we don't need that!" and "that's not the Perl way!". Some times typesafety is a godsend -- when you're working on a large project where interfaces have to change often you get built-in tests that are more thorough than manually written ones. Yet the community has grown better at arguing against having features that they don't have than at evaluating ideas for their actual merits.

All I can say is that Perl's hope is just a seed right now, if it doesn't want to become, for 99% of people out there, a badly conceptualized stereotype overriding any realities and scaring off any fresh blood once and forever.

Any language that has been around for a while goes through terrible growing pains. Python lost a lot of people with py3k and in the load road leading up to it. Ruby's inability to get a fast and backwards compat new version out has damaged the perception of Ruby as being worthy to deploy for any real purpose. Java's hacknayed attempts to bolt on autoboxing and generics have tarnished its lustre as a simple, clean C-like language. It's remarkable -- nay, amazing -- that Perl has had so few growing pains and has held up so well for so long. It's over due for maturing, no question, but I think it's important to give it credit and especially not to be condescending about what it didn't do well enough in getting to where it's at. This remarkable accomplishment has to be weighed against the damage done by the messiness of core and the over-reaching backwards compatibility. Remember that even if Perl weren't "messy", the langauge would still have a stimga just because of the code written in it. There's now a lot of really bad Java in the world due to colleges teaching it as a first language and the radical growth of its use in India where novices are learning it cold and going out into the field and programming it. Any language that makes a lot of money for people experiences this influx of ineptitude. Ruby is winning over Java in the our-programmers-are-smart-and-write-clean-code department. Perceptions of bad code in Perl are only partially the fault of Perl's impressive backwards compat but I suspect more largely the fault of Perl's success in bringing in new programmers. Trying too hard to "clean up Perl" won't have the desired result.

All of this stuff is complex, non-linear, involved, and we lack the perspective on it to make the judgment calls. All we can do is effectively deal in large quantities of uncertainty. Part of that is not beating any drum too loudly, embracing the chaos, and encouraging and helping every programmer interested in core without politics about directions people want to take perl getting involved.


"Better is Better"... yes, but there are problems with putting things in core. Not everyone thinks Module::Install is a good idea. The fragility and lack of maintainership are overriding problems and it might be that core cleanup has to happen before user-visible goodness can be deployed. Yes, CPAN is a good place to experiment with language extensions and I love it when those are able to make it into core, eventually. They converse into proposals for extensions to Perl 5 mostly by debate on p5p about whether the feature is interesting enough that tired bodies can be bothered to risk having to take over maintaining the module.

"Does the CPAN help or hinder the desire for stability of the once-a-decade upgraders?" I could argue both ways on this but I think I'd be missing the point that PHP ate our lunch here. PHP's victory in the "install it and it works" space demonstrates Worse is Better better than anything else I can think of. We need PPI or something similar to it to be adopted as the way to distribute Perl apps. MakeMaker optimizes for the worst case -- running on an Apollo or something. That has merit and I don't want to throw out that baby with its bathwater but how can we make it so that people can install perl, download a script, run the script, and have it work, and preferably have the answer be less doomed in the long term than PHP's?

CPAN meets the needs of Perl programmers. Perl programmers seldom write code for anyone other than their employer or other Perl programmers. This is a problem. I've written about this before. Ruby programmers write code for *people*. PHP programmers almost never write code for each other. Code re-use is good, but it's a tool and it needs to be evaluated critically.

Not wanting to upgrade all of the CPAN modules and risk breaking something -- more likely than perl itself breaking -- keeps people from upgrading perl, I think. Very few people know about autobundle. I've had embarrassing outages due to perl upgrades and module problems.

I wrote before about Motorola's hundreds of installs various versions of perl in people's home directories on the server... that kind of follows what Flash does with upgrades. Each version of the player ships with each previous version of the player, it seems like. Perhaps it's time to fork Perl 5, so to speak, with deprecations. People aren't afraid to run old versions; they are afraid to upgrade. Capitalize on this.

Sorry, long winded because "I lack the time to make it short".

The various groups are at odds; the clash between the camps of "code re-use is good" and "I don't want to upgrade because things might break/installing Perl programs is hard!" stands out the most.

So far, CPAN authors have been doing the only thing they know how to make CPAN play nice with upgrades, and that's to be backwards compat with their APIs or else bump the major version number. That's besides fielding bug and smoke reports, specifying which version of perl the thing works with, and so on. The simple fact is that the more code you change, the greater the chance something is going to break, and an autobundle update and a new version of perl changes a heck of a lot of code. People are right to be nervous.

The whole CPAN problem really isn't my bag. I probably don't have any real insights for lack of thought. My knee jerk reaction is for module authors to please be more aware of what actually happens when lay people try to use their modules in the case where their module becomes a popular dep. In that case, "code re-use is good!" becomes more of a gray area. Also, we out-clever ourselves. I keep finding that I have to convert modules to MakeMaker before they're install because someone is using some new fangled build system that doesn't bloody work.

It may be that we have to decide that CPAN is for programmers and make something else for Perl app bundles. Perhaps the bundles need to get smoked. Perhaps it should smoke bundles that get posted other places if they're popular enough -- such as the blog code that's floating around, and AppEngine, and those things. Perhaps there should be some under the hood interaction between the two but from the user's point of view, they should know if upgrading perl is going to break any application in the registry of Perl apps.


I find this post confusing. The bullet points about CPAN near the end appear to come from out of nowhere. I don't see the connection between them and the alleged insularity problem you discuss earlier in the post.

Modern Perl: The Book

cover image for Modern Perl: the book

The best Perl Programmers read Modern Perl: The Book.

sponsored by the How to Make a Smoothie guide



About this Entry

This page contains a single entry by chromatic published on August 24, 2009 12:01 PM.

The Problems with Indirect Object Notation was the previous entry in this blog.

Who Benefits from the CPAN? is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

Powered by the Perl programming language

what is programming?