Archiving XML with Symphony

27^th November 2008, 24 comments , tagged with XML, symphonycmsSymphony is an incredibly powerful CMS, but it’s ability to cache dynamic XML feeds is lacking. In this article I explain a simple method of parsing a third-party XML feed and archiving its contents in native Symphony sections.

Over the last six months or so, Symphony content management system has become my framework of choice. The reasons are numerous and deserve their own blog post entirely; but the foremost reason is that Symphony allows a developer to define their own data structures, query it, and have the output served as well structured XML to be formatted into XHTML using XSLT views.

With such a focus on XML, it made sense to provide a “Dynamic XML” data source type that pulls an XML feed directly into the CMS. While this native functionality is useful for grabbing a feed of most recent Flickr photos or Twitter posts, I have found it insufficient in several respects:

• if the XML feed is unavailable or invalid at the time of caching, the data source will fail
• content in the feed is not archived — when content is removed from the remote feed, we can no longer access it

In order to solve these problems (as alluded to on the Overture forums) I have used a method of saving items from the remote XML feeds into native Symphony Sections, so that they are archived and can be queried as native entries.

In the following example I’ll explain how to archive your Twitter timeline into Symphony.

Section

The data is to be stored in a Section, so begin by creating a Section with the same data structure as the feed. In my example I have created a “Tweets” section with the following fields:

• ID (Textfield)
• Message (Textarea)
• Date (Date)

Figure 1. Tweets section

Event

Create the event “Save Tweets” from the Tweets Section just created. Make sure the “Allow Multiple” option is selected, so we can save more than one tweet with a single request.

Figure 2. Save Tweets events

Data Source

When saving tweets to the Tweets section, we need to be able to check if the tweet already exists in our cache. Thankfully the Twitter API assigns a unique ID to each tweet that we can use. If an RSS feed is being used the permalink or guid elements are suitable alternatives.

Once the Twitter feed has been loaded, we need to compare the XML against existing entries in Symphony. Therefore create the data source “Cached Tweets” to select the most recent tweets. We don’t need the full entry, just the ID element.

Figure 3. Cached Tweets data source

Page

Create the page “save-tweets” and attach the Save Tweets event and Cached Tweets data source to it. We need a smidgem of XSLT to output the data source as XML. This should do it:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:template match="data">
    <xsl:copy-of select="cached-tweets" />  
</xsl:template>

</xsl:stylesheet>

Figure 4. Save Tweets page

The end result is a page that outputs the latest tweets as XML, and that also acts as a form handler to accept POST requests to add new tweets. All that remains is the brains of the operation — some custom PHP goodness.

The script is self-explanatory, but here’s the outline:

1. Fetch the XML feed of cached tweets from Symphony (created above)
2. Fetch the latest tweets from the Twitter API
3. Iterate through status update in the Twitter feed
4. Check whether an entry containing the same tweet ID exists in the Symphony XML
5. If no entry exists, build multi-dimensional post variables to send to our page

cron.save-tweets.php

<?php

$page = "http://yourdomain.com/save-tweets/"; // path to Symphony page to output latest cached tweets and has the Save Tweet event attached
$twitter_username = "username"; // screen name (e.g. twitter.com/username)
$new_tweets = array();

// Get the most recent cached tweets from Symphony
$symphony_feed = DOMDocument::load($page);
$symphony_tweets = $symphony_feed->getElementsByTagName("entry");

// Get the most recent tweets from Twitter
$twitter_tweets = DOMDocument::load("http://twitter.com/statuses/user_timeline.xml?id=$twitter_username");

// Loop through the Twitter tweets
foreach ($twitter_tweets->getElementsByTagName("status") as $tweet) {

    try {

        // Get tweet information from the XML, format date
        $tweet_id = $tweet->getElementsByTagName("id")->item(0)->nodeValue;
        $tweet_text = $tweet->getElementsByTagName("text")->item(0)->nodeValue;
        $tweet_date = date("d F Y H:i", strtotime($tweet->getElementsByTagName("created_at")->item(0)->nodeValue)); // Mon Aug 18 09:38:05 +0000 2008

        // Query the cached tweets XML using the tweet-id field to see whether the entry already exists in Symphony
        if ($symphony_tweets->length == 0) {
            $cached_tweet = 0;
        } else {
            $xpath = new DomXPath($symphony_feed);
            $cached_tweet = $xpath->evaluate("count(//entry[id='" . $tweet_id . "'])");
        }

        // If the count() was zero, it is a new tweet so add to the new tweets array
        if ($cached_tweet == 0) {
            array_push($new_tweets, array($tweet_id, $tweet_text, $tweet_date));
        }

    } catch (Exception $ex) {
        var_dump($ex);
    }
}

// Set up initial POST variables
$post = "MAX_FILE_SIZE=104857600&action[save-tweets]=Submit";

// Build post variables for each new tweet
for($i=0; $i < count($new_tweets); $i++) {
    $post .= "&fields[$i][id]=" . $new_tweets[$i][0];
    $post .= "&fields[$i][message]=" . urlencode($new_tweets[$i][1]);
    $post .= "&fields[$i][date]=" . $new_tweets[$i][2];
}

// if there are new tweets, send a single POST request to the page
if (count($new_tweets) > 0) {
    $save_tweets = curl_init();   
    curl_setopt($save_tweets, CURLOPT_URL, $page);
    curl_setopt($save_tweets, CURLOPT_POST, 1);
    curl_setopt($save_tweets, CURLOPT_POSTFIELDS, $post);
    curl_setopt($save_tweets, CURLOPT_RETURNTRANSFER, TRUE);
    curl_exec ($save_tweets);
    curl_close ($save_tweets);  
}

?>

What is left to add is that this PHP script should be scheduled to run as a cron task on your server (the frequency of which depends on how much of a Twitter-aholic you are!).

There you have it — a relatively simple method to archive data feeds into Symphony. Imagine the possibilities of archiving references to all of your Flickr photos, or the ability to mine years of Last.fm plays. Thanks to Symphony’s entirely flexible structure, you can recreate almost any data schema and with some rudimentary PHP to perform basic XML DOM manipulation, you end up with a pretty powerful system.