Rethinking Drupal’s Theme/Render Layer Drupal 7 and the Arrays of Doom

By John, 18 April, 2012

This Friday through Sunday, I’m going to San Francisco to attend a sprint to “rebuild the theme layer” in Drupal 8. This is the next exciting step in a journey I started nearly 3 years ago.

I’ll hope you’ll join me at the sprints as we combat Drupal 7 and the Arrays of Doom!

A Drupalcon 2012 film: Drupal 7 and the Arrays of Doom
Modified slightly from a tweet by @SGreenwellUT

Scene 1. [Day break. Our themer is hunched over his keyboard writing awesome themes in Drupal 6, but he’s troubled.]

The loose thread that unraveled our shirts

While using Drupal 6, there were often times themers would come across a template variable that they needed to alter, but, in drupal 6, all variables were PHP strings that had already been rendered into HTML. In order to alter the HTML they didn’t want, themers had to resort to using awkward PHP string manipulation functions or trying to retrieve the original raw data and starting over. While contributing to Drupal 7, I had a simple idea to make it easier to modify some template variables before they were rendered into a PHP string:

  1. Load the raw data in a template_preprocess_HOOK function,
  2. allows themes to alter it in their THEME_preprocess_HOOK function and
  3. then render it in template_process_HOOK (though I wanted to call it template_render_HOOK.)

You can see the results of that work in the $classes_array and $classes variables in Drupal 7.

theme_hook_process_fail()

I wanted to extend that idea to other template variables. But almost immediately after the process hooks were committed, I discovered another group of developers had been working on solution to the same problem. Their solution was to take Drupal’s Form API and extend it be able to take raw data (in “render arrays”) and render it inside the theme system’s template files. I opposed this change as I thought it was way too complicated for beginning themers, but I wasn’t able to build consensus as the “render API” was already about 80% completed.

So, once again, a module developer and a themer were working cross-purposes at the same problem. Each of us should have reached out to the other before doing the work in earnest. hook_process_FAIL(). I ended up working with those developers to improve the render API as my only other choice was to throw my hands up and give up. Mmmmm… Lemonade. :-)

An obvious product placement in the middle of our movie

Front cover of Drupal 7 Module Development
Drupal 7 Module Development

After the Drupal 7 code freeze/slush date, I got the chance to help write the “Drupal 7 Module Development” book with some great Palantiri (we all worked there when we started the book.) I wrote the theming chapters and I tackled documenting the Render API for module developers.

It was a big challenge because I had previously only thought about how to explain the theme system to themers. But I’m proud of the results. Chapter 3 and 4 of that book probably have the best, fullest explanation of the Render API available.

Uh-oh.

But after completing those chapters, I started thinking about how to explain the Render API and the rest of the Drupal 7 Theme System to theme developers. One day in IRC, chx asked me a question about the render API and it took me about 10 days to figure out the answer. And, when I did find the answer, I realized that I had written the code in Drupal core that answered his question! It had gotten complicated enough that even the developers who wrote it had a hard time understanding it.

A mess of arrows pointing at 17 different parts of the theme layer
A simplified wiring guide to the Theme Layer

I knew we were in trouble so I submitted a session to the very first Core Conversations at Drupalcon San Francisco 2010. I showed this infamous slide and boasted I would try to explain the entire theme system in under 60 seconds. I almost made it. I shocked core developers with my presentation, but there were very few themers there. And it sparked almost no follow-up conversation.

I was way ahead of the pack. Drupal 7.0 was still 9 months away and I was probably the only theme developer using D7 at the time. I kept my fingers crossed that it wouldn’t be as bad as I feared.

Oh, crap.

It turned out, it was probably a little worse than I originally feared. We discovered its impossible to determine if a render element is empty. You have to change the render element to a string before you can tell if it generates any markup. This makes it really quite awkward when you want to add some wrapping HTML, but only if the variable has any markup in it. That bug is unfixed and unfixable without radically altering a common template pattern.

And the more I thought about how to explain the render API to themers the more I realized that, for themers, the render API is undocumentable.

Why? Why, for the love of god, why?

  1. The theme functions we inherited from Drupal 6 were not flexible enough to allow re-use within the render API. For example, theme_links only accepts text or rendered HTML in its list. Putting rendered HTML in a renderable array is nonsensical. Each module developer would have to solve this problem their own way since core lacked this ability.
  2. Each render element is unique. The first item in this list naturally leads to this item. The sad truth is you have to have a custom solution to build each variable in the theme system.
  3. An inconsistent HTML spec for form elements led directly to an inconsistent API. Form API’s job was to render forms, but HTML form elements are the most inconsistently implemented elements in the HTML language. Because of this, there are lots of “exceptions” and special cases built into the Form API to handle the inconsistency. Each of these special cases were given a “generalized” solution to fix with the form HTML problems, but that meant we had several generalized solutions to markup problems. Render API, as the sucessor of Form API, inherited every single one of these special cases. And, unfortunately, there are no hard-and-fast rules for when you should use #pre_render or #theme_wrapper or #prefix or #post_render or #suffix or #kill_me_now.
  4. Render elements modify themselves as they go through the system. To make it easier to create a form definition, the Render API has “default element info” and “shorthand formats”. You don't have to create the full, verbose render array that is needed to generate the HTML. All the array #types have default properties that get added to the array; unfortunately, those only get added after the array is passed to render(). So themers examining the render array won’t see any of the default properties, which may be the very ones they want to alter. But there’s a special kind of rage I reserve for #types that add default #pre_render functions. Pre-render functions are executed on the render elements before any of the other code in render() is run. And those functions can alter the render element in any way they want. Dante would be proud of us.
  5. Render elements are undocumented and undocumentable. While the Render API is documentable for module developers, the same cannot be said for the elements that are built with it. Because of item #4 above, the same render element is different depending on whether you are inspecting it from inside the template’s preprocess function or from within the template itself or from within the theme function that is used to render the element. If we document what’s in the render element while in the template file, we have an incomplete picture of how to alter it. If we fully document each element, it would take pages of writing.

A gradual realization

After Drupal 7.0’s release, themers started to complain about the lack of documentation for the render elements. A few people volunteered to document it. Rather than shout “GIVE UP! YOU’RE ALL DOOMED!”, I crossed my fingers that they would prove me wrong. Unfortunately, they did fail. (Incidentally, I ate dinner with one of these brave souls at Drupalcon Denver and he was extremely relieved to hear me call render elements “undocumentable.”)

The fact that some template variables could NOT be printed the same way was confusing and led to calls to remove data structures from templates. But, it turns out its more than a documentation problem, you can actually kill your site by using the wrong print method. For example, don’t try this from within comment-wrapper.tpl.php: Looks harmless enough. But, trying to render $content from within that template will lead to a White Screen of Death due to an infinite loop as it recursively tries to render comment-wrapper.tpl over and over.

As themers started using dsm() to drill into the render elements, their complaints just grew louder. After Drupalcon London 2011, they started swearing like sailors in the issue queue.

And then Jacine started writing up her thoughts on how to design an improved theme system. She shared some of her early thoughts and I promised I would try to summarize the problems highlighted in several lenghty threads on drupal.org and present a Core Conversation at Drupalcon so we could drum up support for actually doing something about it. I started by asking Alex Bronstein to co-present with me and asked Moshe to review the Core Conversation proposal before it was submitted. We themers needed developers on the same page from the get-go.

Catharsis

On the first day of Drupalcon Denver, Jacine posted a blog with her thoughts. On the second day of Drupalcon, we had our core conversation, “Re-thinking the theme/render layers”. The full video is available on the Drupalcon Denver site.

Some people called it surreal. I was just hoping we could drum up support for trying to redesign the theme system for Drupal 9. I only had two slides (including the one from my San Francisco talk) because I wanted it to be conversation rather than a presentation. So five minutes in, we started taking questions and comments. People kept getting more and more animated and excited as they spoke.

About 49 minutes into the presentation, Chx got up to the microphone, outlined his thoughts on the problem, and culminated by saying “I don’t think anything we have currently is salvagable. We need to throw it out.”, “We need to restart absolutely from scratch. Come up with an architecture and implement it.” and ”We cannot release Drupal 8… we just cannot… with this current system where you have these arrays of doom”. He got a standing ovation.

Hitting the ground running

Immediately after the session, right outside the doors to the room, several of us kept talking. Chx was pretty emphatic about getting started right away and wanted to have a sprint at the beginning of April. I pointed out some of us needed to finish our taxes before April 16, so we ended up picking April 20-22.

Jen Lampton at Chapter Three has done a fantastic job organizing the sprint. My flight arrives at SFO at 9am on Friday and I’ll be packing my toothbrush and my bullwhip.

I AM SO EXCITED!

Topics: Drupal, Drupalcon, Drupal theming

Comments16

The content of this field is kept private and will not be shown publicly.

Düde
This weekend will be the most important in many many many years :)

may we all return victorious!

chx (not verified)

11 years 10 months ago

And I am not a themer and the reasoning I had was coming from a completely different angle -- I still think we can't do render arrays and this new OOP system others do for Drupal 8 cos noone would be able to learn it. This has nothing to do with render arrays being bad for themers :) Still. We are going to redo it and this time listen to themers :)

John

11 years 10 months ago

Yep. Our big process fail during Drupal 7’s development was that themers and developers didn’t talk to each other soon enough. Everyone’s intentions were good; it was mostly a communication failure. This was a failing on both sides. I'm glad we’re working together to fix things this time. :-)

I'm also really excited about the sprint! The theme layer is the entry point for everyone new to Drupal (not just theme devs!) because HTML is the common language of the web. Drupal is a really cool and powerful system, and I don't want new people running away because the entry point looks too daunting. I'm interested to see how a simplified theme layer will change adoption rates :)

Jeff Eaton (not verified)

11 years 10 months ago

I still feel bad as one of the earliest proponents of abstracting FormAPI into a generic RenderAPI. While it made certain things easier (reordering node elements, for example), it's accumulated a dizzying array of ugly workarounds and hacks as it pushed lower and lower into our markup chain. Even little things -- like deciding how you should build a list of related links in a renderable array -- is full of questions with no clear answers.

I'm very interested in seeing this work come together; it has the potential to improve things for both themers, developers, and documentation writers...

While I'm not a themer, nor have I ever been extremely deep in Render API, I have explored the deepest throughs of Form API, which is now kind of the same thing. And I have seen Drupal 7 (Commerce) sites' massive, massive, massive render arrays.

Render arrays devour memory. So not only is this about making Drupal's theme system usable (as in understandable) again, it also has the potential to significantly improve's Drupal's performance, or at least memory consumption! :)

I'm very much looking forward to read/hear more about the results of this sprint :)

I'm glad you brought up memory consumption. The other day I started thinking about the amount of buffering that happens while rendering a page. (Every call to theme() returns a string, which usually gets folded into a slightly bigger string, and on and on...) As long as the entire theme system is on the table... I tried to think of a good reason why the page can't simply be rendered in order and sent right down the wire, and I came up empty.

Every call to theme() returns a string, which usually gets folded into a slightly bigger string, and on and on...

The memory consumption of PHP is influenced by a lot of things. Nested function/method calls do affect memory (each nested call has its own in-scope variable stack), but they are inevitable in all programming languages.

The goal is to limit the depth of the nested function/method calls. Recursive calls can cause the depth to increase dramatically; correspondingly, the memory increases.

The data structures can also cause the memory to increase. Bigger data structure == more memory.

Those 2 reasons are why the renderable arrays are so problematic. They are big, fat arrays that need recursive calls to render() in order to make them into strings.

D6/7’s plain-PHP template files are a performance hit too. Though it’s not immediately obvious why. Alex Bronstein was finally able to shed some light on this subject for me over the weekend. The PHP template files have to be loaded into memory and then re-parsed each time its used. 50 comments on a page? It has to load and re-parse comment.tpl.php 50 times. Stopping in the middle of that step each time to recursively call render() just makes it worse.

So it turns out the D6-style of theme() returning strings into variables isn't where the pain is at all.

Tim (not verified)

11 years 10 months ago

Oh goody! Every 2 years we all get to start from scratch and learn Drupal all over again. It's like we get kicked to the bottom of the hill again.

All the buffering may not add up to much memory, but if the html started streaming back to the client as soon as it was available, it would improve the perceived as well as the actual page load time. Also, the browser could start loading images and javascript files as soon as the img and script tags started arriving.

Harry (not verified)

11 years 2 months ago

Your map of the relations between the parts relevant to the render(), is difficult enough to grasp. I seached the internet what is actually rendering, from a developer's perspective. I think I got it a bit more from this post, but I'm not quite there. The reason I began searching the internet about it, was that I learned from debugging that render() works differently at different parts of the code. You said it's undocumentable for themers, and that got me to understand.

As themers started using dsm() to drill into the render elements, their complaints just grew louder. After Drupalcon London 2011, they started swearing like sailors in the issue queue.

I've been reading on drupal.org/projects/ accationally for a couple of years, and I must admit that when you wrote that some core developers started swearing like sailors in the issue queue, and I hovered over the link and saw that the link actually was to a comment, and I clicked it. drupal.org is such a positive and respectfull community, and I just hadn't seen that kind of swearing there before. I had to laugh :-) This render()-related problems is a big big problem isn't it? Nothing to laugh about, really.

So much to say and so little time. Well, I can say this, John... you had me at hello. I'm a second generation Graphic Communication professional and these (Drupal, php, html, css, mysql, actionscript, jQuery, javascript, perl, cgi, etc, etc) are all just tools in my box. Drupal has become the core tool-set of my business since D6 and now D7. I primarily use it as an app with little need to understand it.

Fast forward from my first commodore 64 to November 2012. I was welcomed into a family of developers to assist with themeing a website. Oh lord what did I get myself into. Thank life for giving me all the tools I would need to be this "Theme Dev" < so weird.

Artist > Graphic Art > Graphic Communication > Graphic Design > GUI > UI > UX... woe horsey! KISS

So, anyway... John, and Jen, thank you both for stepping up! It is people like you who keep me interested, devoted and promoting Drupal. You've given me motivation to find a way to get involved with D8 even if it's just using it sooner than later.

I program for so many reasons but at the beginning it was just to show someone my art. (=