How we built the BBC blogs v4 application: part one

Content Management Systems (CMS) have become increasingly popular as a way of producing content online. The BBC uses a number of different content management systems.

Eliza Kessler explained the CMS the Internet blog was using in a previous incarnation and earlier this year Nick Reynolds gave a glimpse of the blog's migration from one version of the CMS, which is called iSite, to another.

The Blogs application itself is not a CMS. iSite2 is the relevant CMS and Blogs v4 is a front-end which connects to the API ("application programming interface") to read the data generated from that CMS. However, we were responsible for building forms in iSite for this application.

I am the Web Developer for Blogs and part of the BBC '/programmes' team. In this post I am going to describe the challenges that we faced during the migration of blogs from iSite1 to iSite2 and why we decided to build a new blogging system from scratch.

You may have noticed that this blog (and others) have started to look a bit different. We recently relaunched the blogs platform as version 4 using iSite. It includes these new features:

Features:

1. Responsive and Touch friendly, so audiences can read your blog content from anywhere using any device type. (Tested on iOS, Android, Windows)
2. Modern clean design.
3. Bigger and better Share button, which works on all screens and is more clean in terms of look and feel
4. Automatic latest Tags module, you don't have to update the tags in the sidebar module each time you add a new tag. The system does it for you.
5. Native support for social media (Youtube, Vimeo, Soundcloud etc) embedding.
6. Lazy Loading for images so that page loads faster
7. Better integration with Facebook, Twitter and other social media platforms providing them with necessary meta information.
8. Slick and powerful content authoring in iSite2.
9. Future posts, meaning you can write a post for Christmas in September and it will appear on the front-end automatically in December.
10. Real Preview meaning that it will show the post exactly as it will look like after posting before posting.
11. New pan-BBC responsive comments module was integrated so that you can comment on this blog directly from your mobile devices.

Our main objectives in this migration were:

1. Increase stability, performance and scalability.

2. Make the system responsive.

3. Migrate to iSite2 before the decommissioning deadline of iSite1 (15 Feb 2015).

As simple as that sounds, it most definitely was not a simple process.

So why did we ditch blogs v3?

The technical debt was too high to maintain the old system as it supported a lot of legacy architecture, which was no longer neccessary in the migration to iSite2. It was simpler to rebuild cleanly than trying to retrofit the old model to work with the new data source. 

This also gave us an opportunity to improve user-experience and stability using lessons learned during the support period for Blogs v3 and experiences shared by the past developers.

Challenges

Here are some of the key challenges in setting up the new system and how we solved them:

Challenge # 1: Migration between data sources:

The BBC blogs system has a history of migration. It has moved from Moveable Type(MT) to iSite1 and now from iSite1 to iSite2. Our assumption is that the future deprecation of the current server platform will require the content to migrate again, and we have worked to ensure that this will be achievable with less effort – from everyone concerned - than was required during the migration to Blogs v4. 

Solution

We identified this as a possible future challange during the migration process and used the 'Adapter Design Pattern' so that we only need to make changes in one place to adapt a new data source in future.

Challenge # 2: Identify the most time consuming area and automate it.

To build a mature system in a limited time span and migrate data as iSite1 had a deprecation deadline.

Solution:

From the past experiences it was evident to us that the most time consuming area would be the migration of data. Transforming the data from old formats to new could not have been done without proper automation tools. These tools were written in the Ruby language with some supporting shell scripts.

We ran these scripts overnight to keep the project on time.

Challenge # 3: Data inconsistency: 2 Different sets of data present in iSite1 ( Old MT data + iSite1 data)

Solution:

To overcome the data inconsistency issue, we used the divide and rule procedure in which we divided the data on the basis of its consistency and wrote scripts to make it similar (i.e having the same XML schema) for the transformation process.

Challenge # 4: Each blog has its own theme

Solution:

Rather than recreating the wheel we borrowed the solution from '/programmes'. The solution is known as "The Branding Tool". You can configure sets of colors and metadata in this tool and it will generate CSS automatically. I think is an amazing idea implemented by '/programmes' team. We integrated that in Blogs v4.

The slides below demonstrate how each blog looks very different from one another but they are all still generated from the same codebase.

Challenge 5 : Deliver the product with maximum quality in "just enough" available time

As the blogs platform is a BBC product, it is expected to have a certain level of quality. So rather than re-creating the wheel we cheated. To minimize time, we cherry-picked classes from the codebase of '/programmes' which was already mature and tested them multiple times. The two codebases, however, remain entirely separate. We still had to write many bits ourselves and in this work the testing team played a very vital role. 

Ritwik Mangal, our main tester who was working in coordination with Shashikant Jagtap, played a vital role in the upbringing of a mature product. His attention to detail on this project paid off when we released the blogs v4. A new product always has lots of bugs and new issues but Blogs v4 had very few due to all previous testing and amendments.

This was a team effort which has succeeded in overcoming geographical differences (as some co-workers were based in other parts of the world) while following an Agile process. The product is fully developed onsite in the UK while testing was done in full in India. David Marland played a vital role in helping with architectural decisions. The whole blogs team worked together to deliver the high quality product that you see before you now.



Offcourse automated cucumber tests were used for the testing; manual testing procedure was also used during the releases. 

I have split this post into 2 parts. In this part I have only talked about the major 5 challenges that we faced. In my next post, I will highlight the next set of challenges so that hopefully you can learn from our experience.

Please comment and tell me what you think.