Blog Posts

Import serendipity entries into drupal

I had to relaunch a web site using Serendipity with a new Drupal installation. The old site had about 100 blog posts and lots of comments we did not want to lose. After some googling, i found this excellent blog post on creating drupal nodes programmatically and this useful resource on drupal node fields. i knocked up the following script that does the job. Just copy this to s9y-import.php and place it in your web root and call it in your web browser. Please back up your database in case something goes wrong (believe me, if you have no backup, it will go wrong!). You might want to read the code comments to know about the current limitations.

Read the full post to see the source code.

updated on 14.7.2009

<?php

/**
 * Import s9y blog posts along with comments into drupal
 *
 * Simply put this into your drupal installation root and call it with a web browser
 *
 * Known limitations:
 *   - You need to be logged into drupal as the admin user (id = 1) and all entries will appear to come from that user
 *   - Assumes that both s9y and drupal use the same encoding.
 *   - Requires s9y and drupal to share one database. If this is not the case, just copy the serendipity_entries and serendipity_comments tables to the drupal database
*   - If your s9y tables do not start with serendipity_ or your drupal tables not with drupal_ you will need to adjust the SQL statements
 *   - Does not take care of contained images. You either have to keep the <s9y>/uploads/ directory online or you copy the contents of that folder to the drupal path and adjust the src attributes using SQL.
 *   - Will make your whole body the teaser, meaning the complete entry is shown in the overview.
          The reason is that s9y handles teaser and full entry differently from drupal.
          You can edit over-long posts and just save them to have an automatic short teaser.
 *
 * Written by <david.buchmann@liip.ch> on 18.6.2009
 * Inspired by this blog post:
     http://vhata.net/blog/2007/03/27/converting-from-serendipity-to-drupal
 */

/*** configuration     ***/
/** map s9y user id to drupal id.
 * the format is s9yID => drupalID
 * entries not found in this map are assigned to uid 1, which is the admin user.
 */
$usermap = array(1=>1);

/*** end configuration ***/

require_once './includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);
error_reporting(E_ALL);


$mappings = ''; //hold url mapping info
$q = mysql_query("select id, title, timestamp, last_modified, CONCAT(body,extended) as body, authorid, comments from serendipity_entries order by timestamp asc");
while($f = mysql_fetch_assoc($q)) {
    $userid = isset($usermap[$f['authorid']]) ? $usermap[$f['authorid']] : 1;
    $newent = array('created' => $f['timestamp'], 'changed' => $f['last_modified'], 'title' => $f["title"], 'body' => $f["body"],
        'teaser' => $f["body"], 'format' => 2, 'uid' => $userid, 'type' => 'blog', 'status' => 1, "comment" => 2,
        'promote' => 1, 'sticky' => 0);
    $newento = (object)$newent;

    node_save($newento);
    //we can not directly get the new node id :-(
    //we hope there is only one at that exact moment
    $qid = mysql_query("select nid from drupal_node where created = ".$f['timestamp']);
    $newid = mysql_fetch_assoc($qid);
    if(! $newid) {
        echo 'Could not retrieve newly inserted node<br />';
        continue;
    }
    $newid = $newid['nid'];

    //url for rewrite
    $qurl = mysql_query("select dst from drupal_url_alias where src='node/$newid'"); //if you use more than 1 language, you will need to consider language column
    $newurl = mysql_fetch_assoc($qurl);
    $newurl = $newurl['dst'];

    $qurl = mysql_query("select permalink from serendipity_permalinks where type='entry' and entry_id=".$f['id']);
    $oldurl = mysql_fetch_assoc($qurl);
    $oldurl = $oldurl['permalink'];

    $mappings .= "$oldurl\t$newurl\n";

    if ($f['comments'] > 0) {

        $c = mysql_query('select id, parent_id, timestamp, title, author, email, url, ip, body FROM serendipity_comments WHERE entry_id = '.$f['id'].' ORDER BY parent_id');
        $map = array(); //index: old parent id, value: new parent id
        while($com = mysql_fetch_assoc($c)) {
            $parent = 0;
            if ($com['parent_id'] != 0) $parent = $map[$com['parent_id']];
            $newcom = array('pid' => $parent,
                            'nid' => $newid,
                            'uid' => 0,
                            'subject' => $com['title'],
                            'comment' => $com['body'],
                            'hostname' => $com['ip'],
                            'timestamp' => $com['timestamp'],
                            'status' => COMMENT_PUBLISHED,
                            'format' => 1,
                            //thread field is determined by comment_save function
                            'name' => $com['author'],
                            'mail' => $com['email'],
                            'homepage' => $com['url']);
            //funny: comment_save returns the id, whereas node_save does not
            $newcid = comment_save($newcom);
            $map[$com['id']] = $newcid;
        }
    }
}
?>
Copy the following into your apache config or .htaccess file and adjust paths
<pre>
RewriteEngine on
RewriteMap urlmap "txt:/path/to/file/urlmap.txt"
RewriteRule ^/blog/(.*) /site/${urlmap:$1} [L,NC,QSA,R]
</pre>
And copy this into urlmap.txt
<pre>
<?php echo $mappings ?>
</pre>

This script only does what I needed at the moment, it is not the perfect importer. It took about 10 seconds to import the 100 blog posts. Then it took me maybe 2 hours going through the blog and fixing links between blog posts and editing the long entries to enter the teaser border.

If you improved the script, please share your improvements here.

Comments [23]

Tom, 25.06.2009 14:17 CEST

It's "Lose", not "Loose".

david, 25.06.2009 14:25 CEST

oh my god, a typo in the wild of the internet! well, thanks for helping me improve my english...

was the script useful for you?

david, 14.07.2009 14:01 CEST

edited the code to
* allow for mapping user ids from s9y to drupal
* output mapping of s9y urls to new drupal ones for mod_rewrite

Jeff, 29.10.2009 09:22 CEST

"utf8_encode($f['body'])" in the generated array is usefully if your content is cut before the first umlaut.

dmitry, 04.05.2010 10:51 CEST

Thank you for this post!
Really helpfull!

deminy, 11.06.2010 05:20 CEST

I'm in the process of migrating my sites to Drupal. Thanks for sharing your code.

chrisbond, 23.11.2010 09:36 CEST

Really very nice tutorial.

Seraphyn, 02.01.2011 16:17 CEST

Does this work under Drupal 7 too?
Greetings and happy new Year
Seraphyn

david, 03.01.2011 10:08 CEST

hi seraphyn,

just try it out with a test installation of drupal 7 :-) i did not try it out, as my migration is finished.
while drupal often changes its api over major version changes, i assume that such fundamental things did not change for 7. would be nice to hear if you manage to use the script or what needs to be changed in order to make it work.
and a happy new year to you too! david

Seraphyn, 03.01.2011 11:55 CEST

Okay, setup a testing Drupal 7 after Release on my server to import it with that script.
Think if something to remanage in your php I'll send you a diff-file.

Greetings
Chris

david, 03.01.2011 12:04 CEST

great, good luck and hope to hear from you :-)

Seraphyn, 12.01.2011 15:01 CEST

Does not work in Drupal 7:
Notice: Use of undefined constant DRUPAL_ROOT - assumed 'DRUPAL_ROOT' in /DELETEDSECURITYREASONS/includes/bootstrap.inc on line 2094

Warning: require_once(DRUPAL_ROOT/includes/errors.inc) [function.require-once]: failed to open stream: No such file or directory in /includes/bootstrap.inc on line 2094

Fatal error: require_once() [function.require]: Failed opening required 'DRUPAL_ROOT/includes/errors.inc' (include_path='.:/usr/local/lib/php/') in /includes/bootstrap.inc on line 2094

Could this be an SQL-Statement alone?
So it is possible to write an Module for this like the WP one?

Greetings

david, 13.01.2011 09:14 CEST

It seems something changed with Drupal 7. a quick web search brought up this post where somebody has similar problems with DRUPAL_ROOT and gets the explanation that he has to define it himself now, like define('DRUPAL_ROOT', getcwd());.

If you can't debug the script, a simple solution could be to use this script import your s9y data into a empty Drupal 6 installation and then migrate from there to Drupal 7. I am sure there are scripts to migrate data from Drupal 6 to 7.

Seraphyn, 13.01.2011 17:20 CEST

We are step closer:
* Warning: mysql_query() [function.mysql-query]: Access denied for user 'HUMPTIDUMPTI'@'localhost' (using password: NO) in main() (line 40 of /usr/www/users/teikoa/teiko/s9y-import.php).
* Warning: mysql_query() [function.mysql-query]: A link to the server could not be established in main() (line 40 of /BLABLABLA/teiko/s9y-import.php).
* Warning: mysql_fetch_assoc(): supplied argument is not a valid MySQL result resource in main() (line 41 of BLABLABLA/teiko/s9y-import.php).

Sometimes I think I need to learn more sql.
And 6 to 7 is not a real option, I would like to Fix this, because it would help many others out there... ;)

david, 13.01.2011 17:27 CEST

the using password: NO means probably that the sql connection is not initialized. which probably means drupal 7 moved to a different db driver, like mysqli.

Seraphyn, 13.01.2011 17:33 CEST

I used a test.php:
<?php
/* Verbindung aufbauen, auswählen einer Datenbank */
$link = mysql_connect("IP", "USER", "FASCINATINGPASSWORD")
or die("No Connection: " . mysql_error());
echo "PERFECT!!!";
mysql_select_db("DATABSENAME") or die("Wrong DB-Name in DBMS");
?>
Works.
Question is,what do i need to change in this case (mysqli) , or isn't possible to let the code directly roll up in phpmyadmin/sql-cli?
Greetings

Seraphyn, 13.01.2011 17:44 CEST

Set in a myysql_connect, but i've get now only this Error:
Warning: mysql_fetch_assoc(): supplied argument is not a valid MySQL result resource in main() (line 43 of BLABLLALBA/teiko/s9y-import.php).

Try to Fix it ;)
Greets

Seraphyn, 13.01.2011 17:55 CEST

$usermap = array(1=>1);

define('DRUPAL_ROOT', getcwd());

mysql_connect("SERVERIP","USER","PASS");
mysql_select_db ("DBNAME");

So far, so good.
Imported title, time and user, but
- no text in node
- no tags
Gimme Errors:
Could not retrieve newly inserted node
Thats all.
After switchin to Drupal 7 and Refresh
Have a lot of Errors.
Try to find out whats next.
Tell you wht's goin on when finished... ;)

Greetings

Seraphyn, 13.01.2011 18:19 CEST

Found out that the Body of the node is stored in field_data_body "Data storage for field 2 (body)"
with entity_id The entity id this data is attached to
and entity_type The entity type this data is attached to.
That's the reason why the Nodebody has no text.

Rashed, 08.02.2011 12:12 CEST

Thanks a million for this

Lifesize, 09.02.2011 15:57 CEST

Im new to PHP (and programing in general) so I had a few problems with it to start with but I got it working eventually. Thanks David. How long did it take you do make this???

david, 09.02.2011 16:11 CEST

hi lifesize

glad to hear you got it working.
as to how long it took, i don't remember exactly. i had to read some about the drupal api, which is documented ok but not all self-explaining. figuring out the s9y database layout was not very difficult as its straightforward. the logic is rather simple.
i'd say something half a day to a day probably.

cheers,david

Nimlok, 16.02.2011 13:58 CEST

thanks for the code, just what I needed.

Add a comment

Your email adress will never be published. Comment spam will be deleted!