Drupal Core's Future and Current Problems

Recently, there has a been a LOT of discussion about the future of drupal core and some of its rather "legacy" modules. This is a good conversation to be having, however I think we are spending our efforts in the wrong place, or perhaps better, I think we have or priorities misaligned.

What do I mean by this?

As great as removing the cruft from core, and lessening the burden on core developers is, I think there's a more important conversation to be having here, and these points shed a completely different light on this conversation. The issue we really need to be discussing is one of packaging management, not bikeshedding about what modules should and should not be maintained by core any longer.

Why is this significant?

Packaging is a core concern for a number of reasons, the two I want to immediately highlight are the aforementioned "lessening" of core contributors work load, as well as supporting other drupal based distributions which already have to depend on packaging as a factor of what they do.

Lessening Core Workload

Core has a number of items that, in any other distribution, would be considered "feature" modules. These module are VERY important as, just like any other distribution, they characterize WHAT the standard install profile IS. I bring this up because, just as Open Atrium would be a completely different product without case tracker, or documentation, so drupal core is a different product without a module that specifically handles blogging. This is why removing blog from drupal core is something I'm not thrilled about. It's not that we want to deliver a different product (or if we do, that's definitely NOT the conversation we're actually having) it's that we want to remove the "product features" from the plate of the core developers, and this brings us to the first "value" I think we need to be tracking.

Value #1:
It's not what we deliver, it's how we deliver it.

The tl:dr version of this is pretty simple. Just because we remove blog module from the list of things that core devs need to worry about doesn't mean that it can't or shouldn't be delivered in the standard install profile. The very real practicality is that we have hundreds of thousands of existing drupal installs out there that need the ability to upgrade their core-provided blog module at some point. I propose that this is best handled by a blog module packaged within the standard install profile.

Supporting Drupal Distributions

Currently, standard and minimal are the "blessed" install profiles. This is actually a good thing as Drupal needs to be SOMETHING when you install it. What's bad about them is that they don't have to follow the same packaging rules as the rest of the drupal based distributions in the current ecosystem. This of course means that certain points of pain for building a distribution are largely not felt by core, and thus will never actually be fixed (or at least are significantly less likely to). This also means that what is currently distributed as "drupal core" is both slow and difficult to change. If core were implementing the same sort of distribution strategy as all other distributions, new profiles could be added to core once deemed strong enough and popular enough to do so. In the same way "standard" product features would still be packaged with core, but get the benefit of a faster potential improvement cycle by being in contrib. This brings us to values number 2 and 3.

Value #2:
All install profiles should be packaged the same way.

Value #3:
Asynchronous development cycles are good for both product and kernel.

Back to the Original Point

We're spinning our wheels bikeshedding WHAT should be removed from core when I think the real question is: "What should be packaged into our product." In truth, Drupal's not really all that well defined of a product. There are a bunch of modules that are there that no one ever turns on and various components of modules that are commonly enabled that no one makes use of. But the fact that Drupal is a rather badly defined product doesn't mean that it can't still be packaged effectively, and this is why I'm concerned about the current conversations within the community. I know my values are not necessarily the same as everyone else, but to summarize my value statements again:

Proposed Values:

1.) It's not what we deliver, it's how we deliver it.
2.) All install profiles should be packaged the same way.
3.) Asynchronous development cycles are good for both product and kernel.

These values are NOT really addressed by bikeshedding what we're going to move out of core. Understand, I DO believe we need to be moving product features out of core, however I also believe that people will be more likely to agree on what is a "product feature" if they have assurances that that code will continue to be distributed, it just won't require the maintenance of core contributors to do so. I am very concerned that we'll spend months debating the various components we want to remove, and then even more months figuring out how to handle their removal in an upgrade path for Drupal 8. I've been told packaging is not a concern until Drupal 9+, and as much as I admire and respect the various individuals I've had private conversations with concerning this topic, I cannot shake the feeling that packaging is the far more important problem, and that ignoring it in favor of removing code completely from what we distribute is a horrible horrible misstep. This approach brings up many other questions, but they're important questions to answer, and I hope that this post can facilitate a larger conversation on this topic.

Kris "EclipseGc" Vanderwater

Jeff Eaton (not verified)

5 September 2011

I agree wholeheartedly with points #2 and #3, but am on exactly the other side of #1. ;-) I think our delivery/packaging mechanism is an implementation detail that has to follow our decision on purpose and focus. We've always demonstrated an ability, as a community, to take tactical implementation steps. We haven't demonstrated an ability to make the tough calls about long term vision. Packaging approaches are tactical, and thus essential, but they aren't an actual decision.

Finally, I don't think that "Packaging" is a Drupal 9 issue. It's a "Whenever we do it" issue, because the biggest roadblocks are process and infrastructure oriented -- not Drupal release cycle oriented. There's nothing to prevent us from splitting out modules with no dependencies for the next version of Drupal *7* if we want to do the work to make our packaging system work that way, and are willing to take the time to set up the repos for individual modules.

I spent a long time arguing for the approach you describe -- years, in fact. Finally, I came to the conclusion that our community is not willing to invest in this kind of infrastructure work unless there is a compelling vision for what benefits lie on the other side of the work. For better or worse, the discussions about core focus, target audiences, and so on are throwing light on what an improved packaging system could help accomplish. In that way, I believe that the difficult discussions about where we're headed and what we need to accomplish are essential investments in our future -- the very opposite of bikeshed discussions.

I suspect that we're not that far from each other -- it's just a matter of what we feel the primary blockers are for accomplishing the Big Picture Goals. Thankfully, there's plenty of room for different people and different groups working on different aspects of the problem at the same time. Getting our packaging infra structure to the point that this approach can work, and getting core refactored to reduce cruft and interdependencies, both benefit each other nicely.

5 September 2011

@Jeff Eaton,

In an effort to have a simple one line value, it is possible that I state value #1 both too strongly and too vaguely. I'll respond to your points, and try to redefine value #1 a bit more clearly (if perhaps, not as concisely).

I think our delivery/packaging mechanism is an implementation detail that has to follow our decision on purpose and focus.

This is actually part of the problem. "Purpose and Focus" have nothing to do with packaging. Dozens of non-trivial drupal distributions are packaged in the exact same manner currently, and they all have a completely different purpose and focus. So I argue that packaging has nothing to do with core's purpose and focus.

I spent a long time arguing for the approach you describe -- years, in fact. Finally, I came to the conclusion that our community is not willing to invest in this kind of infrastructure work unless there is a compelling vision for what benefits lie on the other side of the work. For better or worse, the discussions about core focus, target audiences, and so on are throwing light on what an improved packaging system could help accomplish. In that way, I believe that the difficult discussions about where we're headed and what we need to accomplish are essential investments in our future -- the very opposite of bikeshed discussions.

This whole paragraph actually could be an argument in favor of my proposal here:

  1. There are lots of compelling "vision" reasons to do this. I'd be happy to outline them but I'd rather stay focused on the topic.
  2. The infrastructure is largely already in place... the main point left is to actually force core into using the existing infrastructure.
  3. Core focus and target audience is a conversation YEARS in the making now. I'd prefer this not wait longer.
  4. Understand, I'm not trying to say that these conversations (which are essential to our future) are unimportant. Just that they have a bikeshed-esque quality to them in that it will require a LOT of back and forth to come to an agreement for what is a relatively modest payoff. I propose that having packaging in place before those conversations happen will expedite them because people will be more willing to agree on what is and what is NOT a product feature knowing that those features will continue to be distributed.

So if I can explain value #1:
We value HOW we distribute over WHAT we distribute because:

  1. WHAT we distribute has a lot of legacy attached to it that will require significant discussion before it can be resolved (this is ABSOLUTELY necessary, I'm not saying we shouldn't do it).
  2. HOW we distribute has also largely been decided, it's just that core does not participate in this methodology, thus the decision should be easy. Granted there are discussions like "how ex-core modules will be managed on an ongoing basis", but the value proposition is that that discussion will hopefully yield more fruit more quickly than the "I want X out of core" discussions that will require some pretty serious discussion for each value of X.

As a good first step to #2 here the core Standard install profile could be broken out, ALL BY ITSELF into a repo and core could be migrated to us drush_make like any other profile on d.o. I would celebrate like a mad man if that happened, it would mean squat to WHAT we distribute, and would be a HUGE win to HOW we distribute. It would open the door to discussing how we maintain these separate repos, what it all means, and future of product feature migration. Again, if we continue to deliver the same product (for the moment) then my bet here is that we'll be more willing, as a community, to define our product features AS product features and not shy away on an individual feature basis for various unforeseeable reasons.

I hope that explains my logic and value #1 a bit more.

5 September 2011

Thanks for writing this clarification.

I think there has been a lot of confusion about the actual, underlying problem space - especially for people who are not so deeply involved in Drupal core work.

In fact, I have the impression that most people agree with the outlined vision already. However, there are a couple of concerns and worries with regard to the infrastructure, organization, and maintenance of the envisioned "the Drupal product" and its extensions, which we badly need to start to collect and address, in order to make any kind of substantial progress on the topic soon.

So far, many of these have been raised in various comments, chats, and phone calls - some of them clearly, some others only vaguely. Additionally, there seems to be some confusion that we'd need to solve technical problems with installation profiles on drupal.org first (which is actually not the case for the simple transition from "modules bundled in core" to "modules bundled in a Standard installation profile"). But yes, these concerns and thoughts pile up and lead to general fear, doubt, and uncertainty.

5 September 2011

I think this post addresses the fundamental disconnect in our vision of what Drupal is. Both Drupal 6 and Drupal 7 have been released under false pretenses. The real releases happened with a delay of six months until Views was available for the given version. Sure you can build sites with just Drupal core but it's a core of modules that is what makes it powerful. Without them Drupal is rather mediocre.

So I think the solution should be a small core that is no longer equated with Drupal. It's just its core but Drupal actually ships with actual site building features - ideally with several profiles that make Drupal into a blog, community or a publishing website. And that Drupal should be released annually or even every six months with an LTS (long-term support) release every three years that contains core changes.

I know this may be unrealistic given the architectural issues of Drupal but it works for Wordpress, Ubuntu, or CiviCRM so perhaps Drupal should learn from these successful models.

In reply to by chx (not verified)

5 September 2011

I'm sorry you see it that way chx. That being said, the code/gold is already written for this and deployed to d.o. I'm simply proposing core make use of it.

5 September 2011

I agree, Kris, and want to add that "what we deliver" has as much to do with drupal.org as it has to do with Drupal itself. It has to be obvious that Drupal's packaging model is distros, and that Drupal is designed to be extended using modules; and it has to be easy to find and install modules and distros. Reorganizing core isn't enough by itself, especially if we're taking out the product features.

Anonymous (not verified)

6 September 2011
the code/gold is already written for this and deployed to d.o. I'm simply proposing core make use of it.
Based on Eaton's comment's here and the snowman thread chx linked to. The code now deployed on d.o. isn't robust enough to do what's being asked for. Unless I'm reading incorrectly.
6 September 2011

The code I'm discussing is the existing drush_make distribution stuff on d.o. Existing drush_make can do most (if not all) of what would be required to do this (depending on specific requirements). Unless the d.o version is significantly different, chances are very good that core should be package-able from the existing script. I'll admit I can't speak to that directly so most of my arguments are based on existing drush_make capabilities.

Add new comment

The content of this field is kept private and will not be shown publicly.