September last year I met up with an old room mate. She told me about a student project called Making Commitments Matter (MCM). They had created a database for UN resolutions. Even though the team was quite large, spanning all across Europe, they didn't have anyone in the group who had a sufficient level of IT know how to bring it all online. Once again it shows that Universities usually are unable to bring together different departments. So I had no choice but to help them out!

Step one defining a database schema for the excel files they currently used to store the information. Contrary to the document systems of the various UN organizations which scatter the various PDF's across different sites, the aim of resolutionfinder.org (previously called UN-informed.org) is to bring all information to one database. Furthermore resolutionfinder.org would also show the entire history of documents and each of the clauses there in. This again is a huge leap forward, since right now its essentially impossible to determine if a given document or clause is actually the most current version and how things changed over time. Especially this history was a big challenge in the definition of the schema.

Aside from this the two years that MCM spend interviewing key stakeholders like NGO's, government officials and researchers have lead to a fairly complex set of additional attributes that also need to be stored for each document and clause. The fact that each clause is stored separately is also a key improvement since many documents contain decisions on many different topics. With this database it will be possible to search for just the clauses for a given area as both documents and individual clauses are tagged.

Using the admin generator of the symfony PHP framework I started building up an interface to the database in order to be able to get feedback from the MCM team. By way of a call for help on my blog Dennis Riedel, living in Barcelona, joined the effort and began working on an import tool to move the data from the excel files into the database schema. At a project meeting in Helsinki in early December 2009, we used this prototype to define the final specifications for the database.

The goal was to have a working frontend ready for the conference MCM was organizing on the topic of "UN information management" end of May 2010. No small undertaking for a small team of two guys in their spare time, neither of whom being real experts in frontend development. So I turned to Liip for help. At Liip we have budgets to support technically or socially relevant efforts. The process is basically pitching the idea to the entire team to get development resources. In the end resolutionfinder.org was allocated 12 man days development as well as 1 man day for server administration and a server to run the application on!

Christoph and Jordi both stepped up on short notice to help me build up a frontend that actually exceeded by far the hopes of the MCM team. The site leverages Apache Solr for the search with features boolean operators and facette filtering. We display the documents, clauses as well as their relation and history. Its also possible to register to store bookmarks to documents and clauses as well as comment them. Most amazingly we still have a few days of development left that we will use to further optimize the UX on the search interface. I should also thank Stefan Sicher who delivered a very sleek and clean design just days before the conference.

Obviously there is still lots left to do. As mentioned before we want to further improve the UX on the search. We are also considering adding alternative search approaches, for example some people might want to search chronologically, others organization etc. We also want to add data mining capabilities so that we can automatically import voting records and maybe eventually even entire documents. Supporting more languages both for the interface and the content also is high on our list. We might also provide the content via an RDF database to enable others to come up with queries of their own. The team is obviously full of ideas and is actively looking for companies or developers willing to donate resources. The source code of the entire application is of course open source: patches welcome!

Here is an example search for "security council" documents, for regional education topics outside of africa. Here is an example document and example clause with a history on the topic of malaria. Additional information on the technology can be found in the case studies publised as Lucid Imagination and symfony. I also published a series of posts on my personal blog. Overall it was a very fun and rewarding experience and obviously I will continue to work on the project in my spare time. For Liip I can already see how several of the lessons learned during the project have paid off on client projects as well. So overall I guess its a win for all!

 Recently Fabien announced an online conference to present the current state of Symfony 2. I quickly signed up and told Jordi who was just about to sign up himself. I figured there surely are more guys at Liip that want to join, so I send a mail to our internal developer list. Next someone suggested opening the entire thing up to the world. So I fired off an email to Fabien and we agreed that this would be an awesome way to get the social aspect into this online event. A few mails later we decided that it would be fair to ask such hub's to buy 5 tickets (not that anyone will check this, but hey lets play fair if we want stuff like this to happen again). Fellow Liiper Pierre wrote up a blog post that Fabien then published as the official Symfony Live Hub announcement.

So what was missing? Yeah, I didn't do my job to announce our Zurich hub! Here it is: We decided to go with the evening of the 23rd. The fun officially starts at 17:00, but feel free to start dropping by any time after 16:30. I managed to get a small budget to provide some food and drinks.

Feel free to leave a comment if you plan to come so that we know how many chairs to setup.

See you there!

This is an idea I had mind in since last year, and finally was convinced to put online these days. Hoping you'll like it, Hannes

If anything about current web development can be called «glamorous»[1], it’s creating applications for en vogue platforms such as the iPad or the latest line of Android phones. After all, they're not only super sexy, but the rate of progress we see is staggering as well - a pretty tempting combination.

At the same time, web technology is constantly taking up more and more speed, progressing on many levels at an exceptional pace. Standards such as HTML5 and CSS3 as well as their support in transformative tools such as Webkit have been helping web user experience improve dramatically and at a much greater rate than the progress we have seen on any "desktop" environment - which almost stagnated for a decade.[2] And let's not forget: seeing the speed of JavaScript engines improving by orders of magnitudes clearly means: something important is happening here.

This blog entry is trying to deal with the great deal of stuff that's happening when these two ongoing developments -amazing devices and accelerating web tech- come together. That «stuff» we see out there has such as breadth, depth and scale that we shall be trying to get from stuff to patterns, and then give them names. Names that help us actually have clearer conversations about those things as they develop, like folks did with «AJAX», «RIA» or «Web 2.0». The name we're suggesting is NIWEA - Native Interoperable Web Apps. Let's see what this means:

«Native», the N in NIWEA, is a word that has been bended and stretched for years. We'll use it in just two simple ways:

• «Made for this thing» - native apps just do feel right - on your desktop, your mobile, your tablet, your TV.
• «No crutches» - web native apps use use the tech and specs of the web - bred through open standards, no plugins, no add-ons.

 «Native» also means access to the capabilities of the underlying platform: sensors, geolocation, graphics, storage, background computations, audio, video - you name it.

In the example sections there's an whole array of examples illustrating this point, runtime environments such as PhoneGap, systems like Jolicloud, native user experience enablers like jQTouch, Google's Store for Installable Web Apps are certainly amongst those that must be mentioned.

«Interoperable» can mean a lot as well, but it's a strong notion, and one that keeps getting more and more important. The «I» in NIWEA means two things:

• «Works with my sites» - NIWEA can be standalone, but mostly they'll be talking, synchronously or asynchronously, to web APIs. APIs based on web standards that have interoperability built in, enabling mashups
• «Works with my gadgets» - another kind of interop, the "write once, run anywhere" idea. Indeed, web technology is the only platform I see that has the potential to overcome the terrible fragmentation we see in mobile platforms these days. Being the only platform without an owner certainly helps.

Interoperability between devices and consistent communications between applications - browser engines, REST & Co. can do it.

NIWEAs are built using web technology, and nothing else - but «web» doesn't mean exclusively «browser» any more. And «web» doesn't mean «online only» any more either. In fact we're going more and more towards what you might want to call an «ubiquitous web», with the browser engine becoming the universal runtime.

The are many ways NIWEA bring application functionality to you, be it packaged Web OS, Chrome Store or HTML-based Adobe AIR apps, desktop «gadgets» or dashboard «widgets» - it's all native-feeling, interoperable web tech.

Why care?

So why is this relevant at all? Aren't those iPhone and iPad apps out there just so superior that there's no need to even bother about that old-school web stuff? Well, I think this is about the long term, about what we can do to foster "sustainable" development practices that are truly open and e.g sufficiently safe against lock-in. In the short term, proprietary platforms might be lucrative and spectacular, but the benefits of having web tech "everywhere" might very well be far greater: the web is where the developers are, and enabling them to do NIWEAs could very well be pretty disruptive in many areas. And: we should never ever underestimate the web.

And now?

Having a word for all that stuff happening these days could help a lot, I think. So please do let me know: what do you think about it? Do you like it? Is it stupid enough? Or way too stupid?

Would you tag your bookmarks with "NIWEA"? Could we have a "NIWEA Conference"? Would you sign an "I support NIWEA" list? Have any links that just have to be on the list below?

Already thinking about funny logos? :)

Examples & Links

• Mobile Examples
  
  • Phonegap
  • jQTouch
  • Palm webOS
  • Nokia Web Runtime
  • JIL & BONDI
• Desktop Examples
  • Jolicloud
  • Chrome OS & Chrome Web Store
  • brevity
  • Apple Widgets & Windows Gadgets
  • Adobe AIR with Webkit
  • Mozilla Prism
  • iTunes Extras
• Miscellaneous Links
  • http://www.quirksmode.org/blog/archives/2010/03/html5_apps.html
  • http://www.quirksmode.org/blog/archives/2010/04/mobile_web_or_o.html
  • http://www.zeldman.com/2010/02/01/flash-ipad-standards/
  • http://www.toolness.com/wp/?p=821
  • http://radar.oreilly.com/2009/05/google-bets-big-on-html-5.html
  • http://radar.oreilly.com/2010/02/a-prism-for-jolicloud-web-cent.html
  • http://ajaxian.com/?s=native
  • http://humanapi.org

Footnotes

[1] For a note on glamor also see the original Ajax article
[2] See also Google I/O 2009 keynote for web vs. desktop UX - (not overly scientific) slide, video

A few months ago the current development branch for the next major release of PHP was dropped. It was in development for several years and was intended to bring Unicode support to the core of the language. The approach taken was not seen as feasible for many so it was never completed. Its existance however slowed down development and made release management difficult. For example PHP 5.3 was basically created to get some features from this new Unicode branch released. Obviously having to back port lots of features to 5.x that had piled up over the years proved to be a very difficult task, which took over one year.

The obviously question to ask is what will happen now? The German t3n magazine recently interviewed me on this topic for the 20th edition of their print magazine.

t3n: Derzeit ist nicht klar, ob die nächste PHP-Version 5.4 oder 6 sein wird. Woran liegt das?

Lukas: PHP 6.0 war als der „Unicode-Release“ auserkoren. Dazu wollten die Mitglieder des PHP-Projekts das Thema Unicode und den grundlegenden Ansatz noch mal überdenken. Der daraus resultierende Thread auf der Mailingliste ist vorerst im Sand verlaufen, ohne dass irgendwas dokumentiert wurde. Somit stellt sich die Frage, ob der nächste große Wurf wirklich mit Unicode zu tun haben wird.

t3n: Welche spannenden Neuerungen sind denn schon im neuen Trunk oder werden vermutlich bald einfließen?

Lukas: Das aus meiner Sicht interessanteste Feature ist „Horizontal Reuse“. Dabei geht es um einen alternativen Multiple-Inheritance- Ansatz. Die Wahl ist hier vorerst auf Traits gefallen. Ein möglicher neuer Aufhänger für PHP 6 sind Zends Performance-Patches. Mit FPM ist außerdem bereits ein alternativer Ansatz zur Nutzung von FastCGI in den Trunk eingeflossen. Schließlich gibt es noch die The- men „Large File Support“ und „Large/Infinite Integer Support“. Hierfür gibt es jedoch noch keinen Patch.

t3n: Facebook hat vor wenigen Monaten das Projekt „Hip Hop“ veröffentlicht, um PHP zu beschleunigen. Was halten Sie davon?

Lukas: Allgemein bin ich recht unbeeindruckt. Die Zugewinne von „bis zu 200 Prozent“ halte ich für extrem wenig, soviel würde ich von einem PHP-zu-C-Extension-Compiler erwarten. Der größte Vorteil ist die Speicherreduzierung dank Multi-Threading. Von daher habe ich das Gefühl, dass Facebook langfristig sowieso lieber seine eigene, wenn auch an PHP angelehnte, Sprache haben will. Meine Hauptsorge ist, dass Facebook langfristig keine Ressourcen mehr in die Entwicklung des APC-Cache steckt.

t3n: Immer mehr Entwickler entdecken, wie komfortabel schlanke und moderne Sprachen wie Ruby oder Phyton sind. Warum sollten Entwickler trotzdem auf PHP setzen?

Lukas: Das Killer-Argument für PHP ist und bleibt die große Verbreitung. Es gibt viele Entwickler, preiswertes Hosting und eine große Auswahl an Komponenten und fertigen Applikationen. PHP ist nicht die am schönsten designte Sprache, jedoch enorm schnell, wenn es darum geht, wichtige neue Webtrends direkt in den Sprachkern aufzunehmen.

t3n: Welches PHP-basierte Open-Source-Projekt finden Sie derzeit besonders spannend?

Lukas: Ich verfolge die Entwicklung von Symfony2 und den diversen Components mit großem Interesse. Mit Doctrine2, den eigenen Symfony Components und auch einigen Zend-Framework-Klassen wird hier die Open-Source-Entwicklung in Reinkultur praktiziert.

In about 7 weeks, the OAuthocalypse is coming. Twitter will turn off Basic Authentication for accessing their API and you have to use OAuth. I assume, there are a lot of php scripts out there, which still use the Basic Authentication approach and not OAuth. The Basic Authentication way also breaks the password anti-pattern and is therefore very bad practice. You have been warned now :)

Here I present my little rss2twi.php script, which posts new entries from an RSS feed to Twitter with OAuth. It is certainly far from perfect, but may be a starting point for some. It is also part of my Real Time Web With A Real World Example Setup.

It has the following features:

* It uses OAuth for authentication. You need the pecl/oauth extension for that. Maybe someone could port it to use PHP Userland OAuth

* It can handle more than one RSS feed
* Configuration is via a YAML file (uses sfYaml)
* It can push updates to Prowl
* It has a customizable URL Shortener (defaults to liip.to)
* Also has a very basic Pubsubhubbub Subscriber for immediate updates.

* It is Open Source (Apache Software Licensed) and available at Github

I won't go into details how it's all done. The main class is only 245 lines of code, sample URL shorteners are also included in the code.

To use it, you first have to get the oAuth Tokens. For that you call the getOAuthToken.php file from a command line and do what it tells you to do (basically to approve the App with Twitter). You add those lines to conf/feeds.yml and then just call "rss2twitter.php" to send your RSS items to Twitter. There are more options for feeds.yml, they are described in feeds.yml-dist. You can also define the defaults in conf/defaults.yml and add more than one feed.

If you want to act as a Pubsubhubbub Subscriber you need to adjust and use "subscribe.php" and "callback.php".

The whole script certainly could be extended and some things polished, but it works for me since quite some time and pushes all updates of this blog, Planet PHP and my private blog quite flawlessly to Twitter (and if all goes well even in real time). Feel free to use it, fork it, extend it and do whatever you want. And I hope, it helps spreading OAuth instead of the bad "asking for passwords" pattern a little bit more.

Last week Lukas, Pierre and myself attended the Symfony Live 2010 conference since we have a growing interest in it lately, including a pilot project at Liip that uses symfony 1.4.

Overall the conference was quite a success I would say, they had a few organizational issues but considering it was the first international event of that scale they hosted it was really good. Most talks revolved around symfony and also doctrine since it's a major part of the symfony ecosystem, but a few sessions were held on more general topics, such as PHP performance, deploying apps to the cloud or Zend Framework. Lukas and I spoke about Liip's framework Okapi since it's second iteration, that is still in the works, is using a few symfony components. As we mentioned there we decided to use them to alleviate some of the maintenance work and benefit from the dependency injection feature, which offers amazing opportunities for customization and testing. Our slides can be found on slideshare and the Okapi2 source on our svn repository.

The highlight of the conference was obviously the announcement and release of the Symfony 2 codebase. Symfony 2, just like Okapi 2, is based on the Symfony dependency injection container component, which means it will have the same flexibility. Given the community it has, it is definitely an interesting development for us and we will follow its development closely. If it happens to fit our requirements as well as Okapi does for fully custom high performance websites we might adopt it but it is too early to say since it won't be stable until the end of the year.

Yesterday, Facebook released HipHop for PHP, a compiler which makes native code from PHP sources (via C++ and gcc). They promise twice the speed with that approach, something which helps you a lot if you have 1000s of servers like Facebook. I won't bore you with the technical details and what it can and can not do as Ilia already summed that up pretty nicely.

We at Liip will certainly look into this technology once it's available and evaluate if it would makes sense for some of our customers and project. As HipHop is not a drop-in replacement for your existing setup this is something which has to be planned and tested carefully. You have to change the way your servers are setup and the way you develop and deploy. It even needs a server restart, if you change some code, so badmouths can now claim that PHP is even more becoming like Java :)

A high amount of Switzerland's most visited sites run on PHP (and some of them are even done by or with us) and maybe HipHop could be useful for some of them, after all the usual optimizations (like APC, memcached, etc) didn't help enough. But for most websites (and more so for small ones), PHP's speed isn't really the bottleneck, it's the database or some other external dependency. And for these cases HipHop isn't the holy grail. So before you change all your setup to fit into HipHop's requirements, do some decent evaluations, what really is the bottleneck in your application and decide accordingly.

For a larger project here at Liip Zurich, we are looking for an experienced Magento PHP Developer. Start as soon as possible. Freelance is ok, but you have to be able to work with the team here in Zurich onsite (most of the time). This also means: No interest in outsourcing, off- and nearshoring or however you call it. We just need a person fully integrated into our already existing Magento team. If you're interested get in contact with Nadja Perroulaz (contact@liip.ch) or me.

Furthermore we're (always) looking for an experienced PHP/Javascript Web Developer. If you're interested in a job where you can learn and develop new stuff almost every day and work on exiting projects and if you have a lot of experiences in the PHP/JavaScript/Web World, do not hesitate to send us your application. The ad linked above is in German, if you don't understand it and want to know more, I'm here to help. Knowing German is no prerequisite for a Job at Liip :)

Last week I gave 2 techtalks (one in Fribourg and one in Zurich) about how we use some of the "Real Time Web" technologies in Flux CMS and its related services. The graph for this looks like that:

And the details to this can be seen in the slides.

I also plan to write some blogposts about some of the technologies used as the blue ones in the graph are written by us and Open Source (see the links at the end of the slides). And I really hope it will end up in more than just a plan :)

If you want to know something in particular before I write those posts, just ask in the comments.

The idea behind sending emails on every svn commit is basically to help developers review their peer's code, which improves both awareness of the global codebase and overall code quality since errors might be picked up and go or bad coding practices can be identified and discussed between the developers.

While we have had svn post-commit hooks set up for a while at Liip for some of our bigger projects, the case of smaller ones that occupy very small teams remained problematic. The main issue being that configuring the classic post-commit hooks requires intervention on the svn server, to which not everybody has access, and since small projects might change often this can be annoying both for the sysadmins and for the developers that have to depend on someone every time they want a new svn directory to be watched.

The idea we came up with is to use svn properties as the configuration for the post-commit hook, so that anyone with commit access can set it up, and it can easily be applied to any number of directories.

The script looks up through the parent directories of every modified/added files in a commit, with a svn propget liip:review call, and once one is found, the value of the property is used as the email "To:" address(es). This means you can easily have an entire project being watched by someone, but some particular sub-directory of it will only be watched by some other persons, for example if you want to extract the xsl commits out of a php okapi project you would just add another liip:review property on the themes directory.

It is fully configurable if you want to try it for yourself, you basically need to give it the svn server url and the property to check for, then some other extras are available. To allow easy filtering in mail clients the mails sent contain a X-SVN-Commit-Mail header with the value being the name of the repository. Also the subject contains the committer name, revision and watched-paths affected.

You can get the script off of our public svn repository at https://svn.liip.ch/repos/public/misc/svnwatch/ - the dispatch-emails.php script uses the svn binary and the dispatch-emails-ext.php uses pecl's svn extension for php if you prefer.