fidzu : Drupal

Over the past year we have gone through many iterations of our CI system, 3 major versions to be exact (not counting minor updates). During this time we have focused on stability, speed and documentation. These are the major changes that we have found make a huge difference.

I recently started on a project that involves migrating some data from a legacy app & database into Drupal. The old application is a collection of PHP scripts that basically just generate forms, accept data, insert said data into the database, and output it on a website. Pretty simple stuff - there's not a whole lot going on. It was developed long before many of the popular CMS's and frameworks came to be, and probably before people really started paying attention to the character encoding of their data.

I initially began the data migration using Drupal's Migrate module, setting up my new content types and fields, and running through some test imports. Things seemed to go well, until I started scanning the imported data. I started to see some really strange characters like ’ and é

For the most part, it was pretty clear what these characters were supposed to represent, based on the context they were placed in. I knew right off the bat that it had something to do with character encodings, something that I've never taken the time to truly understand. I made the mistake of thinking "oh this shouldn't take too long to fix", later unraveling my very own "character encoding hell". Do I'll talk more about that towards the bottom of the post, but let's first get an intro character encodings.

So what are character encodings exactly?

When textual data is stored and transmitted, it has to first be converted to binary just like everything else. So how is text converted? There needs to be a lookup table matching characters with binary representations. That's determined by the character encoding that is chosen - which can be one of many.

Whenever data is decoded, the character encoding must be known beforehand. Web browsers do this every time you view a website. How do they know what encoding the text is in? The web server tells it, or the browser has to make an educated guess (not what you want!).

ASCII is a very basic character encoding that many are familiar with. It covers the English language along with common symbols and control characters, using only 7 bits to provide a maximum of 128 characters in the set. ASCII isn't really used anymore on the web because of the small number of characters it supports. There are many encodings that do use a full 8-bit byte for each character, re-claiming that wasted 8th bit from ASCII. In America, the most common single byte encodings are probably Windows-1252 and ISO-8859-1. However, no single byte encodings can possibly hold all of the characters necessary to create one "universal" encoding that can support a language like English and Japanese.

UTF-8 is a Unicode compliant character encoding that has become the dominant encoding on the web because it IS a variable width (ranges from 1 to 6 bytes per character, limiting wasted bits) universal encoding. It supports just about every character in every language you can think of. This is very important for the web. It makes multilingual sites easier to manage since you don't have to worry about any localized character sets for each language. Everything uses the same character set.

Most developers should only be dealing with UTF-8 at this point (or another Unicode encoding) and should understand how character encodings are involved in every part of your website or application.

Where you need to worry about them

Web pages

There are two opportunities to do this - one is the "Content-Type" HTTP response header which is typically set to text/html; charset=utf-8 for standard HTML pages. Your application should set this before delivering the response to the browser. It also includes the MIME type of the response which tells the browser what type of document is being delivered (image, video, document, etc).

The other is a meta tag header <meta charset="utf-8"> or <meta http-equiv="Content-Type" content="text/html; charset=utf-8">. The former is the newer HTML5 version. It's a bit confusing to indicate the character encoding within the data that needs to be decoded, but this is allowed in HTML. Parsers will interpret everything as ASCII until it hits that header (which will work, since the HTML syntax is within ASCII and can be parsed that way) then re-parse the document with the new encoding. The reason it's supported in HTML is to account for any inability to set the HTTP response header which would otherwise provide the same info.

You should be using both methods. Without setting this data, your browser will have to guess, and it may display "garbage" text.

You can actually see what character encoding your browser chose to render the page in, and even force it to render it using a different encoding. It's a handy tool that can help diagnose if a page was meant to be rendered in some other encoding. Chrome, FireFox, and Safari all support this ability (IE probably does as well) in the "View" menu.

Form submissions

When data in input in text boxes or text areas in an HTML form, the browser has to encode it before sending it to the server. What encoding does it use to do this? Again, this is up to you to decide. By default most browsers will just use the same encoding that the page was rendered with. However, you can specify this in the <form> tag: <form action="/process.php" accept-charset="UTF-8">. So, while explicitly indicating the character set to encode the data with is not totally necessary, you should do it anyway just in case. If this is not set you risk the browser encoding the data in some random Encoding that your back-end is not anticipating.

MySQL connections

Something that is often overlooked is that when you communicate with a database server and you send textual data, you need to indicate the character encodings when sending and receiving data between your back-end and the MySQL server. This makes sense once you understand that any time text is transferred from one place to another, you need to indicate what encoding the text is in. The text does not automatically have a way of indicating what encoding it's in (not universally anyway).

It get's pretty tricky here, at least with MySQL. There are specific settings in MySQL that you can set after a connection has been established that dictate how characters are treated between the client and server. In particular, there are three things that are important to look at:

• The encoding of data you send to the server from the client
• What encoding the server should convert to after receiving the data
• The encoding the server should return to the client when queries are run (a conversion if necessary).

Assuming the data you're sending MySQL actually IS in UTF-8 and the data in MySQL is stored as UTF-8, you'll want all three of those things to be UTF-8. No conversions will actually take place, and MySQL will just pass everything along as UTF-8. To set these values, you can simply execute the statement SET NAMES utf8 after making a connection.

If you DON'T set these values, then MySQL will more than likely default to latin1 (Windows-1252) which is just asking for trouble! A knowledgable developer may recognize that they need to set the character encoding for their web page and forms, and even their database fields (see below). However, if they have a backend script that accepts UTF-8 data from a form submission, but it doesn't tell MySQL it's UTF-8, then MySQL will think it's latin1 when it's actually UTF-8.

MySQL text field storage

Text fields in MySQL require you to indicate the character encoding of text fields. Defaults are set up on the server, database, and table level (each inheriting from the former). Since you're telling MySQL a field is a text field, it needs to know how to interpret the raw data it's storing as textual data. Without doing so, you wouldn't be able to query for text or have MySQL compare text fields with one another. You could alternatively just use BLOB fields instead, where the data is stored as binary data and not interpreted in any way. Just keep in mind you won't be able to search on these fields.

From what I can tell (as well as another blog author I reference below), MySQL doesn't change how the data is stored based on the encoding of a field. It just interprets the data differently when reading it. I could be wrong here (if I am, please let me know in the comments). However, you can alter the encoding of an existing field which will re-encode it (actually does change the data) as long as you recognized the limitations outlined in the that article.

For instance, if the string é was encoded as latin1 and stored in MySQL in a latin1 field, it will be stored as hex value E9. You can verify this yourself by running something like SELECT hex(text_field_name) FROM table_name. If you then run ALTER TABLE `table_name` CHANGE `text_field_name` `text` MEDIUMTEXT CHARACTER SET utf8 NULL; MySQL will convert that data from latin1 to UTF-8 for you. If you run the hex query again, you'll get back C3 A9.

In MySQL 5, all the text storage defaults are set to latin1 (Windows-1252) just like they are for the connection settings discussed above. While it's well known at this point that UTF-8 is preferred, I believe that the consensus was that the team behind MySQL didn't want to make the dramatic change to UTF-8 quite yet (though I think this is how it will be in MySQL 6).

Note: the collation of a field has nothing to do with how the data is stored, and instead effects how the data is compared and sorted.

What can go wrong

One of the important things to understand is that UTF-8 is a multibyte encoding. This means that the majority of characters are represented with more than one byte (typically two or three), while a traditional character set just uses one byte per character. To really understand let's look at a practical example. Here's the word "Résumé" in two different common encodings:

Encoded in	Bytes	Interpreted in Windows-1252	Interpreted in UTF-8
Windows-1252	52 E9 73 75 6D E9	Résumé	R�sum�
UTF-8	52 C3 A9 73 75 6D C3 A9	R©sum©	Résumé

The normal letters in Résumé are actually the same in both Windows-1252 and UTF-8 (that's was an intentional design choice when creating UTF-8). However, once we get into the special accented e, it's actually represented with two bytes in UTF-8 and just one in Windows-1252.

If your data is sent from the browser to the server as UTF-8 (this is something you as a web developer have control over), is then stored in your database as UTF-8, and finally pulled out of the database and displayed on your website as UTF-8, then will be all dandy. This is how things should be and you have little to worry about.

But let's say that you accidentally removed the charset meta tag from your page template and your web server and app don't set the "character-encoding" HTTP header. Your app will still be delivering UTF-8 encoded text to the browser, but the browser no longer knows it's UTF-8. So the browser guesses. It's possible it will correctly guess that it's UTF-8, but it's also possibly it will guess Windows-1252 as the encoding. If that happens, Résumé will look like R©sum©. Why? Because the é was sent to the browser as C3 A9, which in Windows-1252 translates to the characters  and ©. If it were correctly interpreted as UTF-8, the browser would correctly translate C3 A9 to é.

Whoops! You catch the error a few hours later and make sure the character set is properly set to UTF-8. No harm done, right? Since the the data didn't actually change at all, there is no data corruption to worry about. The browser now interprets the text data properly as UTF-8. Well, not so fast. Let's say you have a form on your site that accepts comment submissions. Someone submitted the form when the browser rendered the page in Windows-1252. Since most browsers will encode data in form submissions using the same encoding that it used to render the page, the text may be encoded as Windows-1252 and sent along to your backend.

Now you have a problem. Your app is expecting the data to come in as UTF-8 when it's actually encoded as Windows-1252! If the comment contains with the word "Résumé", it will be encoded to 52 E9 73 75 6D E9 instead of 52 C3 A9 73 75 6D C3 A9 like it should be in UTF-8. You may have your database driver setup to indicate that the data you're sending it is UTF-8, and the database field may be set to store it as UTF-8. It's possible that your DB abstraction code throws an exception when it sees that 52 E9 73 75 6D E9 is not valid UTF-8 (which is correct, it's not valid), but it may let it slide and insert it anyway.

After you fixed the website to tell browsers you're sending it UTF-8, let's say someone goes to view the comment that was submitted earlier. Instead of displaying Résumé, the browser will actually display R�sum�. That funny question mark box is the Unicode "replacement character" - which is used when the byte sequence in the text is not supported in UTF-8. E9, the byte that Windows-1252 uses for é, is NOT a valid UTF-8 character. If you're curious why, you need to read more about how UTF-8 encodes text.

You have data corruption now. To fix this, you'd need to find the data entries that were submitted with the wrong encoding and manually convert them to UTF-8. Not a fun task. Yo can't even assume that any text submitted after your bad commit would be encoded as Windows-1252 and is stored incorrectly. Some browsers may have properly send UTF-8 data and it was properly encoded. So if you tried to re-encode all the data after that commit to UTF-8, you may end up double-encoding the valid UTF-8 characters.

Conclusion and further reading

Character encodings are not trivial to deal with and it's not terribly fun trying to figure out why funny characters are being displayed on your app or website. Hopefully after reading this you'll take the subject seriously and really begin to understand what character encodings mean to your website and application. Generally you should be using UTF-8 everywhere!.

The application I'm working on has been operating for years without a character encoding set in the HTML pages or form submissions. I've determined that form submission data has been encoded in UTF-8, Windows-1252, and ISO-8859-1, with no definitive way to figure out what data is what. The application does not set the character encoding for the MySQL connection at all, so it defaults to latin1 (Windows-1252). To top it all off, the database text fields are a mix of UTF-8 and latin1 (and the UTF-8 fields may have been "converted" from latin1). It's been frustrating trying to determine what is right and wrong in the database and how to properly fix it because there's both a mix of encodings on the field storage level as well as the input coming into the app. There's no sure fix for any of it.

If you're interested in learning more, these are some great starting points:

• What Every Programmer Absolutely, Positively Needs To Know About Encodings And Character Sets To Work With Text
• Handling Unicode Front To Back In A Web App
• UTF-8 and Unicode Wikipedia
• Getting Out of MySQL Character Set Hell

Brian from Modules Unraveled and I put together a short screencast for people unfamiliar with Drupal. If you are wondering what Drupal can do, or weighing your options in the world of web frameworks and content management systems, here is a common use case: a simple business website.

As you see, it is fast and easy to build a site with Drupal, especially because of the Drupal community, which Modules Unraveled and Pantheon are grateful to be a part. Enjoy the first installment of "The Site I Built."

About Modules Unraveled Brian Lewis is one of the people making Drupal more accessible. He does a great job of explaining how to make Drupal do whatever you want, even if you don't code. Brian's site is a great resource we often recommend at Pantheon.

About "The Site I Built" In this small Drupal world, we get to know each other pretty quickly - we meet at camps, chat on IRC, and work together on sprints, projects and all kinds of other initiatives. It's a great community, and I am happy to be a part of it. Nevertheless, it is hard not to develop a bit of a drupal-centric worldview.

Here at Pantheon, we are trying to burst this bubble, or at least make it a bit bigger. We know that most people who end up on Pantheon are "in the know." But we also see people using us to kick the Drupal's tires, and we know a vanilla Drupal 7 install is not the most compelling first impression. Nor does it hint at Drupal's flexibility, scalability, and ease of use.

Yes - I said it, Drupal can be easy for new users.

This is our first in a series of efforts to prove it - we are calling it "The Site I Built," (hoping it conjures up images of your grandpa, walking you down an old country road, and... um, building you a website). We want to spread the word about Drupal however we can!

During DrupalCon Prague I had the pleasure of giving a presentation about teaching Drupal 8 as part of the core conversations track. In the presentation I talked a lot about providing a good learning environment for people starting with Drupal 8, and removing the fear and uncertainty that are so counter to learning.

Dan Callahan is part of the Identity Team at Mozilla who are trying to solve some of the problems of privacy and security on the Internet that have been hitting the headlines recently. Dan works on the Mozilla Persona project, a system to both replace passwords with verified identities and put that verification under user control, rather than the control of large corporate entities.

Over the past two years, we've built Drupal 8 into what will be the most flexible, future-proof Drupal version ever. Core developers have contributed thousands of hours of work to expanding Drupal 8's capabilities and modernizing our APIs.

We're several months into Drupal 8's API completion phase, and we're releasing monthly alphas as we nail down key APIs, refine the developer experience, and continue vital work on performance. To finish Drupal 8, we must focus on essentials, so I'd like to ask the Drupal developer community to look ahead to the next big step: the first Drupal 8 beta.

When does alpha become beta?

Earlier in the year, we announced that Drupal 8's first beta would be released once we had a stable data upgrade path from Drupal 7. At DrupalCon Prague, however, the Drupal core developer team made a bold decision: instead of using Drupal's update.php database update script to convert Drupal 7 sites to Drupal 8 on the fly, Drupal 8 core will instead include a robust data migration API (based on the popular migrate module) to migrate data from existing sites into new Drupal 8 installations. This means that Drupal 8 core will provide reliable, extensible migration from Drupal 6 as well as Drupal 7. We believe this to be important for organizations running older versions of Drupal can reliably modernize their sites.

Data migration no longer blocks a beta release

In order to make this important data migration change possible for Drupal 8, the initial Drupal 8.0 release will be primarily intended for building new Drupal sites, and the finished data migration path for existing Drupal 6 or 7 sites may be provided in a later Drupal 8 release, like Drupal 8.1. (For more information on how we might improve the Drupal release cycle after the release of Drupal 8, see the proposal to manage the Drupal 8 release cycle.)

This means that a data upgrade path from Drupal 7 is no longer a prerequisite for releasing Drupal 8.0-beta1. Instead, we will focus on what testers and contributed module authors most need from a Drupal 8 beta: (1) a stable data model and (2) stable critical APIs.

Stable data model

A stable data model means that developers should not need to perform data migrations between beta releases of Drupal 8 (except where necessary to resolve critical issues). The Drupal 8 data model includes database schemas, file-based configuration storage, and storage services like the Entity and State systems.

Stable critical APIs

To provide contributed module developers with a useful milestone for module porting, beta 1 will include stable critical APIs. These are fundamental APIs that most or all contributed modules depend on, including the configuration system, the Entity and Field API, the Plugin API, and the Routing and Menu systems.

Other API changes approved by core maintainers will continue through the end of the API completion phase, but after the first beta, we will shift from away removing deprecated code and instead retain more backward compatibility layers. (Module/theme developers who wish to go through the porting process only once should wait for the first release candidate.)

What issues are blocking beta1?

Drupal core maintainers determine which specific issues must be resolved to meet the criteria above. We have worked with core developers to identify a list of beta-blocking issues. There are currently 48 of these "beta blockers" outstanding. As you can see, there are many difficult problems in this list that need to be solved. We need your help to resolve these issues so that we can release beta1 and expand Drupal 8's reach to new testers and contributors.

It's focus time!

While the end of Drupal 8's development cycle is in sight, there's still a lot of work to do. Now more than ever it's essential to focus on the critical issues that will bring Drupal 8 closer to release. If we don't, we risk pushing Drupal 8's release off for many more months. The sooner we create a beta, the sooner we can release Drupal 8 to the world.

Many people looking forward to Drupal 8's release aren't sure how best to help out. I'd like to ask all sub-system maintainers to watch their sub-system's issue queues closely to help new contributors triage issues and fix bugs, especially for beta-blocking issues. I'd also like to ask everyone to review patches carefully, make only necessary API changes, and document APIs clearly. Or, if you aren't able to work on Drupal 8 issues directly, consider sponsoring core developers for Drupal 8 contribution.

Help us make Drupal 8 the best release of Drupal yet by working on our alpha releases and toward a Drupal 8 beta!

ICS celebrate International Volunteer Day by launching their new blog containing a rather special blog post from Parliamentary Under Secretary of State for International Development, Lynne Featherstone.

ICS celebrate International Volunteer Day by launching their new blog containing a rather special blog post from Parliamentary Under Secretary of State for International Development, Lynne Featherstone.

International Volunteer Day was established by the UN General Assembly in 1985. Over the years it has become an opportunity for organisations of all kinds to take time out to recognise the contribution of their volunteers.

On this day last year, Ban Ki-moon, the Secretary General of the United Nations said:

2013-12-06 10:00 - 10:15 Australia/Brisbane

Online meeting (eg. IRC meeting)

Here is a how-to for beginners: Install more than 1 Drupal Stack on Acquia DAMP

1. Go to Acquia Dev Desktop Control Panel -> Settings… -> Sites. You will arrive at this page below. Instead of clicking "New…," which will give you a new instance of Drupal on your current stack, click the "Import…" to create a new install.

2. On the Import site, for the "Site path:" choose "Browse…" On this pop up, search for the stack of Drupal you wish to install and click open. The one I have selected is drupal-7.x-dev.

4. If you are doing a new install, for the Database, select "Create new database" and make a new DB name.

5. You can enter the "Subdomain:" and "URL path" of your choice.

6. After you click "import," Acquia Dev Desktop will take you to the Select an installation profile page. You will see the page below displayed on your browser. Now you can continue with your regular installation.

Whether you've been handed an existing Drupal site by your IT department or you're starting from scratch, it's important for a Marketer to fully understand how Drupal is put together. Understanding the different systems and layers will help you make better marketing decisions and build content that takes full advantage of Drupal.

This blog post may get a little deep, but don't worry. If you come away with just a few good concepts about Drupal then you're doing fine and you can always refer back to this post as you need to. (bookmarks are your friend)

These systems are the key parts that make up a Drupal site:

• The Theme Layer sits on top of your Drupal site and has final say on how things look to your site visitors.
• The Taxonomy System gives you the ability to categorize and tag your content.
• The Node System determines what a "content" is and how it is structured
• The User System creates separate "accounts" for each logged-in visitor.
• The Path System allows you to create site structure in your URLs

The Theme Layer

The best websites separate the look and feel from the content. That is, in theory, what the theme layer in Drupal does. You or your developer can create a Theme that is made up of CSS and a bit of php (don't worry, it's easy or mostly done for you in the theme) to create a look for your site.

For those who prefer a more visual representation, here's a great graphic Drupal.org created to explain theming architecture:

According to Drupal.org, "All layers can implement a themed representation of the output, but (with a few exceptions) only within the theming layers can overrides occur." So, making a single change to the theme will carry down to every single page of your Drupal site.

If you need a certain landing page to be different, it's very easy to create a separate theme file for just one page. (see "page" section on that link)

Prefer that all your blog posts to have a different font size? You can easily theme content types separately.

Want to hide certain blocks on a form page? You can hide certain blocks on a form page.

Here's a 20 minute video from our friends at Drupalize.me that will help you understand all this a bit more: Introduction to Theming Basics.

If you only understand those few things then you're already well on your way!

The Taxonomy System

Taxonomies are used to classify and categorize website content. In other words, they allow you to create a tags to help organize your content within your selected content type. Tags are essential to keeping your content organized and well-structured.

They are helpful internally to the editorial team writing articles so they have a place to nest their content on the site, or to the sales team when they're looking for a past example to help sell a lead on choosing their company's services. Tags also help website visitors find what they're looking for. After all, what's the point in having content if the consumers of that content can't find it?

Overall tag categories are collected in what are called vocabularies. Under vocabularies there are terms or tags followed by sub-terms/sub-tags. An example of this would be a events business with several different service offerings they cover on their blogs. So, there could be a vocabulary called Weddings or another called Birthdays or a third called Corporate Events. Within those vocabularies, tags could include venues, decoration, or catering with sub tags like indoor or outdoor. This video from BuildAModule has another great example of leveraging taxonomies to create different content categories.

The Node System

You'll hear the words "Content" and "Node" used almost interchangeably in the Drupal world. A node is nothing more than a group of "fields" (or "chunks" if you listen to Jeff Eaton...and I do) that are related to each other.

In other words, a node is a group of fields that all go together. For example, if you put these fields together:

Then you'd have a blog post "Content Type". The Node system has an easy method for defining all the content types you could ever want. You might have different fields that are needed for a Press Release:

• Headline
• Byline
• Body text
• Image
• Breakout quote
• Date
• Contact first name & last name
• Contact email address
• Contact phone number

or a company event:

• Title
• Host
• Location
• Date
• Time
• Description
• RSVP form

See? It's easy to put together exactly the fields you need for almost any purpose. Drupal provides you with a "Basic Page" and a few other types. The Blog module gives you a "Blog" type. And, you can easily create any of the types you want.

The User System

You might be the only person logging into your site, or you may have a whole team that needs access to the site, or perhaps your consumers can log in and create their own accounts. The User system allows you to manage these users so that everyone may log in securely and access what you want them to access. You do this through creating roles such as: Administrator, Author, Developer or Subscriber.

Within each of these user roles, you can set up user access permissions and restrictions to certain content types, admin features such as modules or appearance settings.

The Path System

Drupal, unlike most other CMSs, does not force any kind of organization onto your content. All content is equal to all other content. That's right, every new content that is created lives on the main node level of a Drupal site. For example, node/123 is just as important as node/1 or node/1000001. There is no built in categorization or layers in a Drupal website, Drupal assigns a number to each piece of content chronologically and you must create your own unique URL paths. While this may seem daunting, it creates deep flexibility in how you're able to lay out your website structure. Moz has a great guide to URL structuring that covers best practices that your site structure should follow for SEO and user experience.

Search engines often look at how a website is structured to determine what silos (groups) of content that exist on a site. The bigger or deeper the silo the more likely that the site is an authority on that particular topic - especially if users find the content useful and spend a lot of time on the site, link to it, and share it on social networks.

While understanding the architecture of your Drupal site might be overwhelming at first, understanding and utilizing these systems will help you create marketing content intelligently that creates a powerful impact. Take a look at your own site, use this post for reference, or bookmarkt it for later. Have questions? Hit us up in the comments, we're always happy to help.

An overview of Drupal's architecture and how Marketer's can use it to full benefit

drupal, Planet Drupal, drupal marketing