Rethinking Drupal’s Theme/Render Layer Drupal 7 and the Arrays of Doom

This Friday through Sunday, I’m going to San Francisco to attend a sprint to “rebuild the theme layer” in Drupal 8. This is the next exciting step in a journey I started nearly 3 years ago.

I’ll hope you’ll join me at the sprints as we combat Drupal 7 and the Arrays of Doom!

Scene 1. [Day break. Our themer is hunched over his keyboard writing awesome themes in Drupal 6, but he’s troubled.]

The loose thread that unraveled our shirts

While using Drupal 6, there were often times themers would come across a template variable that they needed to alter, but, in drupal 6, all variables were PHP strings that had already been rendered into HTML. In order to alter the HTML they didn’t want, themers had to resort to using awkward PHP string manipulation functions or trying to retrieve the original raw data and starting over. While contributing to Drupal 7, I had a simple idea to make it easier to modify some template variables before they were rendered into a PHP string:

1. Load the raw data in a template_preprocess_HOOK function,
2. allows themes to alter it in their THEME_preprocess_HOOK function and
3. then render it in template_process_HOOK (though I wanted to call it template_render_HOOK.)

You can see the results of that work in the $classes_array and $classes variables in Drupal 7.

theme_hook_process_fail()

I wanted to extend that idea to other template variables. But almost immediately after the process hooks were committed, I discovered another group of developers had been working on solution to the same problem. Their solution was to take Drupal’s Form API and extend it be able to take raw data (in “render arrays”) and render it inside the theme system’s template files. I opposed this change as I thought it was way too complicated for beginning themers, but I wasn’t able to build consensus as the “render API” was already about 80% completed.

So, once again, a module developer and a themer were working cross-purposes at the same problem. Each of us should have reached out to the other before doing the work in earnest. hook_process_FAIL(). I ended up working with those developers to improve the render API as my only other choice was to throw my hands up and give up. Mmmmm… Lemonade. :-)

An obvious product placement in the middle of our movie

After the Drupal 7 code freeze/slush date, I got the chance to help write the “Drupal 7 Module Development” book with some great Palantiri (we all worked there when we started the book.) I wrote the theming chapters and I tackled documenting the Render API for module developers.

It was a big challenge because I had previously only thought about how to explain the theme system to themers. But I’m proud of the results. Chapter 3 and 4 of that book probably have the best, fullest explanation of the Render API available.

Uh-oh.

But after completing those chapters, I started thinking about how to explain the Render API and the rest of the Drupal 7 Theme System to theme developers. One day in IRC, chx asked me a question about the render API and it took me about 10 days to figure out the answer. And, when I did find the answer, I realized that I had written the code in Drupal core that answered his question! It had gotten complicated enough that even the developers who wrote it had a hard time understanding it.

I knew we were in trouble so I submitted a session to the very first Core Conversations at Drupalcon San Francisco 2010. I showed this infamous slide and boasted I would try to explain the entire theme system in under 60 seconds. I almost made it. I shocked core developers with my presentation, but there were very few themers there. And it sparked almost no follow-up conversation.

I was way ahead of the pack. Drupal 7.0 was still 9 months away and I was probably the only theme developer using D7 at the time. I kept my fingers crossed that it wouldn’t be as bad as I feared.

Oh, crap.

It turned out, it was probably a little worse than I originally feared. We discovered its impossible to determine if a render element is empty. You have to change the render element to a string before you can tell if it generates any markup. This makes it really quite awkward when you want to add some wrapping HTML, but only if the variable has any markup in it. That bug is unfixed and unfixable without radically altering a common template pattern.

And the more I thought about how to explain the render API to themers the more I realized that, for themers, the render API is undocumentable.

Why? Why, for the love of god, why?

1. The theme functions we inherited from Drupal 6 were not flexible enough to allow re-use within the render API. For example, theme_links only accepts text or rendered HTML in its list. Putting rendered HTML in a renderable array is nonsensical. Each module developer would have to solve this problem their own way since core lacked this ability.
2. Each render element is unique. The first item in this list naturally leads to this item. The sad truth is you have to have a custom solution to build each variable in the theme system.
3. An inconsistent HTML spec for form elements led directly to an inconsistent API. Form API’s job was to render forms, but HTML form elements are the most inconsistently implemented elements in the HTML language. Because of this, there are lots of “exceptions” and special cases built into the Form API to handle the inconsistency. Each of these special cases were given a “generalized” solution to fix with the form HTML problems, but that meant we had several generalized solutions to markup problems. Render API, as the sucessor of Form API, inherited every single one of these special cases. And, unfortunately, there are no hard-and-fast rules for when you should use #pre_render or #theme_wrapper or #prefix or #post_render or #suffix or #kill_me_now.
4. Render elements modify themselves as they go through the system. To make it easier to create a form definition, the Render API has “default element info” and “shorthand formats”. You don't have to create the full, verbose render array that is needed to generate the HTML. All the array #types have default properties that get added to the array; unfortunately, those only get added after the array is passed to render(). So themers examining the render array won’t see any of the default properties, which may be the very ones they want to alter. But there’s a special kind of rage I reserve for #types that add default #pre_render functions. Pre-render functions are executed on the render elements before any of the other code in render() is run. And those functions can alter the render element in any way they want. Dante would be proud of us.
5. Render elements are undocumented and undocumentable. While the Render API is documentable for module developers, the same cannot be said for the elements that are built with it. Because of item #4 above, the same render element is different depending on whether you are inspecting it from inside the template’s preprocess function or from within the template itself or from within the theme function that is used to render the element. If we document what’s in the render element while in the template file, we have an incomplete picture of how to alter it. If we fully document each element, it would take pages of writing.

A gradual realization

After Drupal 7.0’s release, themers started to complain about the lack of documentation for the render elements. A few people volunteered to document it. Rather than shout “GIVE UP! YOU’RE ALL DOOMED!”, I crossed my fingers that they would prove me wrong. Unfortunately, they did fail. (Incidentally, I ate dinner with one of these brave souls at Drupalcon Denver and he was extremely relieved to hear me call render elements “undocumentable.”)

The fact that some template variables could NOT be printed the same way was confusing and led to calls to remove data structures from templates. But, it turns out its more than a documentation problem, you can actually kill your site by using the wrong print method. For example, don’t try this from within comment-wrapper.tpl.php: Looks harmless enough. But, trying to render $content from within that template will lead to a White Screen of Death due to an infinite loop as it recursively tries to render comment-wrapper.tpl over and over.

As themers started using dsm() to drill into the render elements, their complaints just grew louder. After Drupalcon London 2011, they started swearing like sailors in the issue queue.

And then Jacine started writing up her thoughts on how to design an improved theme system. She shared some of her early thoughts and I promised I would try to summarize the problems highlighted in several lenghty threads on drupal.org and present a Core Conversation at Drupalcon so we could drum up support for actually doing something about it. I started by asking Alex Bronstein to co-present with me and asked Moshe to review the Core Conversation proposal before it was submitted. We themers needed developers on the same page from the get-go.

Catharsis

On the first day of Drupalcon Denver, Jacine posted a blog with her thoughts. On the second day of Drupalcon, we had our core conversation, “Re-thinking the theme/render layers”. The full video is available on the Drupalcon Denver site.

Some people called it surreal. I was just hoping we could drum up support for trying to redesign the theme system for Drupal 9. I only had two slides (including the one from my San Francisco talk) because I wanted it to be conversation rather than a presentation. So five minutes in, we started taking questions and comments. People kept getting more and more animated and excited as they spoke.

About 49 minutes into the presentation, Chx got up to the microphone, outlined his thoughts on the problem, and culminated by saying “I don’t think anything we have currently is salvagable. We need to throw it out.”, “We need to restart absolutely from scratch. Come up with an architecture and implement it.” and ”We cannot release Drupal 8… we just cannot… with this current system where you have these arrays of doom”. He got a standing ovation.

Hitting the ground running

Immediately after the session, right outside the doors to the room, several of us kept talking. Chx was pretty emphatic about getting started right away and wanted to have a sprint at the beginning of April. I pointed out some of us needed to finish our taxes before April 16, so we ended up picking April 20-22.

Jen Lampton at Chapter Three has done a fantastic job organizing the sprint. My flight arrives at SFO at 9am on Friday and I’ll be packing my toothbrush and my bullwhip.

I AM SO EXCITED!