Compare and convert HEIF, JPEG and WebP images with rokka

TL;DR

Go to https://compare.rokka.io/_compare/ and compare the output of the HEIF, JPEG and WebP formats. Even upload your own pictures. All done with rokka.

Long version

Apple produced quite some hype with their support for the HEIF image format in the upcoming iOS 11 and macOS High Sierra. HEIF (High Efficiency Image File Format) is a new image file format, which supports many use cases and uses HEVC for the compression part, also known as H.265. Apple is using HEIF on their latest devices as a replacement for storing pictures and claims up to 50% saves on storage.

Even though no browser does support HEIF yet, also not Safari in the current betas, we nevertheless thought it would be cool to add HEIF support to rokka – our image storage and delivery service. And so we did.

Unfortunately there’s no out-of-the-box solution to create HEIF files currently. But Ben Gotow‘s site jpgtoheif.com inspired us. He published instructions how to create HEIF files with the help of ffmpeg, x265 and Nokia’s heif writerapp. But due to the uncommercial-only license of that Nokia code, we use GPAC to create the HEIF container, which is published under the LGPL license.

Looking at and comparing HEIF compressed images

What’s the fun, when almost no one can look at the result? So we built a little site, where you can compare the output of rokka’s HEIF, JPEG and WebP (the later is only supported on Chrome) and even upload your own pictures. Just head to

https://compare.rokka.io/_compare/

and enjoy it. The uploaded images will be deleted from time to time.

The site uses Nokia’s HEIF Reader JavaScript Implementation, which decodes a HEIF image in JavaScript to a canvas element. This way, everyone can look at HEIF images and compare them to JPEG and WebP output.

The site also allows you to play with different quality settings. All formats support a setting from 1 to 100. 1 is the lowest and 100 the highest (also means lossless for WebP). The different quality settings for the different formats don’t really correspondent to each other. Just play around with them and compare the sizes of the images with different settings.

We use pretty much the default settings of ffmpeg, maybe some stuff could be improved on that side. And we also don’t know what kind of encoder Apple is using for generating HEIF images. So don’t really compare the compression we produce for HEIF images with what maybe other encoders can do.

Also be aware, that we asynchronously compress JPEG images in the background with mozjpeg (see the rokka docs for details), so the first render output is not the maximized compression we can get for JPEG images. Just hit the render button 10 seconds later to get the final compression (the site will inform you, when it’s not done yet with that compression step).

Tags: , , ,

rokka got some new features

Rokka – our image storage and delivery service, got quite some new features lately, thanks also to feedback from our users. Here’s a list of the visible stuff we did.

Brandnew this week

  • autoformat. A new stack option. If set, rokka will return WebP, if the client supports it and a PNG or JPEG was requested, resulting in smaller images.
  • dpr. Device Pixel Ratio. Another new stack option. If set, returns higher resolution images, useful for <img srcset="" /> for example.
  • Width or height are finally optional in the resize operation.
  • The WordPress plugin got some improvements.
  • Work for a Drupal 8 module and a Kirby plugin are under way.
  • General improvements in performance, failover cases and small bugfixes.

    Continue reading about rokka got some new features

Tags: , , ,

Going Crazy with Caching – HTTP Caching and Logged in Users

HTTP caching is an efficient way to make your application scalable and achieve great response times under heavy load. The basic assumption of HTTP caching is that, at least for some time, the same web request will lead to an identical response. As long as “same” means simply the same domain name and path, you will get many cache hits. When users are logged in, we have the opposite situation, where potentially everybody will see different content. Lets take a closer look to see where we can still find safe uses for HTTP caching even with logged in users.

Controlling the HTTP Cache Behaviour

A HTTP request is not only the URL, but also the headers. Some are only for statistics or not relevant for your application. But for some web applications, there are relevant headers. The Accept-Language header can be used to decide on the content language, or when building an API, the Accept header can be used to choose whether to encode the answer in JSON or XML.

HTTP responses can use the header Vary to declare what headers lead to distinct responses on the same URL. A HTTP cache uses the Vary to keep the variants of the same URL apart. This works well when there are few variants – you will still get frequent cache hits. However, if every request comes with a different header, caching on the server side makes no sense anymore. There is no benefit in storing results in the cache that will rarely be reused. Even worse, this is a waste of resources that should be used for caching relevant data.

For this reason, caching proxies like Varnish will by default not attempt any caching as soon as there is a Authorization or Cookie header present in the request. Cookies are commonly used to track a session in the application, meaning the user might see a personalized page that can not be shared with any other user. If you force caching with cookies and have your application send a Vary: Cookie header, you will have the situation described above, where you get no value out of your cache.

The rest of this article will dig into various aspects of what you can do to still do some HTTP caching:

  • Avoid Session Cookie, remove when no longer needed
  • Delegate to the frontend: “Cheating” with Javascript
  • Different cache rules for different parts
  • User Context: Cache by permission group

Continue reading about Going Crazy with Caching – HTTP Caching and Logged in Users

Tags: ,

The Data Stack – Download the most complete overview of the data centric landscape.

This blog post offers an overview and PDF download of the data stack, thus all tools that might be needed for data collection, processing, storage, analysis and finally integrated business intelligence solutions.

(Web)-Developers are used to stacks, most prominent among them probably the LAMP Stack or more current the MEAN stack. On the other hand, I have not heard too many data scientists talking about so much about data stacks – may it because we think, that in a lot of cases all you need is some python a CSV, pandas, and scikit-learn to do the job.

But when we sat down recently with our team, I realized that we indeed use a myriad of different tools, frameworks, and SaaS solutions. I thought it would be useful to organize them in a meaningful data stack. I have not only included the tools we are using, but I sat down and started researching. It turned out into an extensive list aka. the data stack PDF. This poster will:

  • provide an overview of solutions available in the 5 layers (Sources, Processing, Storage, Analysis, Visualization)
  • offer you a way to discover new tools and
  • offer orientation in a very densely populated area

Continue reading about The Data Stack – Download the most complete overview of the data centric landscape.

Tags: , , , , , ,

How to reduce your development and maintenance costs with APIs?

An API-based solution has many advantages. The biggest one is the significant spare on development and maintenance costs, thanks to a modular infrastructure. With the example of a recent work for a watch manufacturer, this blog post explains you in 4 points what added value an API can bring to your IT environment.

Context: a manufacturer’s production line without a central server

We were recently involved in the digital transformation of a manufacturer’s production line. The main issue of this manufacturer was control-desktops that were not centrally managed. In consequence, for each code change, an update was required on each control-desktop. It was an expensive and time-consuming process.

Production line - A

This image represents a production line with standalone control-desktops, and their costly maintenance routines.

Continue reading about How to reduce your development and maintenance costs with APIs?

Tags: , , , , , ,

APIs for the public sector

I recently gave a presentation (in German) at the Beschaffungskonferenz. This is a conference for the public sector to exchange round procurement of IT. There were several tracks some focusing more on legal aspects, different procurement processes and agile development while I presented in the tech track. In my talk I presented some of the more established new development paradigms of the past years. But the key message was that APIs need to become a key aspect of how IT projects are planned for the public sector. Specifically I named transport.opendata.ch as a shining example of how providing existing data via a public API can lead to an entirely new economy of use cases on top of it. The idea is really: “Built it and they will come”.

Update: Another good example of a well documented API in the public sector is api3.geo.admin.ch which we have used for various projects here at Liip in the past already.

Update 2: An article which provides an additional perspective: API First at data.gov.uk

Continue reading about APIs for the public sector

Hacking with Particle server and spark firmware

The particle server

In my previous blog post, I wrote about the concept of my project using particle. Now I will explain what I had to do to increase the data rate transfer of my modules (remember, my goal is to get data  with the closest data transfer of 1 [ms] ).

First, I installed the local Api server (https://github.com/spark/spark-server).

Then I had to register all of my photon’s public key on my server and the server public key on my photons.

Using this command :

Then, I launched the server to see if my photons were responding with something like this :

So from here all was working fine but what I also needed to use there is JS library to get data from OAuth. The thing is that you have to do a lot of configurations if you want to make it works but in this project it was not the goal. I had to test as quickly as possible. So I did what you usually do not have to do with a library installed via npm.

Continue reading about Hacking with Particle server and spark firmware

Tags: , , , , , , , ,

Experimenting with React Create App and Google Sheets API

Since the opening of our Lausanne office, a person who shall remain anonymous collected the most epic — from weirdest to funniest — statements made in the open space to fulfill a constantly growing database of quotes. From post-its to Google Docs to Google Spreadsheets, it definitely deserved a better interface to read, filter and… vote!

Quotes React app

Setting up the development environment

After having done some projects with it, it was clear React would be a good choice. But unlike the previous occasions, I did not want to waste time setting everything up; experiments are about coding right? There’s plenty of React boilerplates out there and whereas some are great, most usually include too many tools and dependencies.

Continue reading about Experimenting with React Create App and Google Sheets API

Tags: , , , , , ,

A recommender system for Slack with Pandas & Flask

Recommender systems have been a pet peeve of me for a long time, and recently I thought why not use these things to make my life easier at liip. We have a great community within the company, where most of our communication takes place on Slack. To the people born before 1990: Slack is something like irc channels only that you use it for your company and try to replace Email communication with it. (It is a quite debated topic if it is a good idea to replace Email with Slack)

So at liip we have a slack channel for everything, for #machine-learning (for topics related to machine learning), for #zh-staff (where Zürich staff announcments are made), for #lambda (my team slack channel) and so on. Everybody can create a Slack channel, invite people, and discuss interactively there. What I always found a little bit hard was «How do I know which channels to join?», since we have over 700 of those nowadays.

Bildschirmfoto 2016-06-16 um 11.34.12

Wouldn’t it be cool if I had a tool that tells me, well if you like machine-learning why don’t you join our #bi (Business Intelligence) channel? Since Slack does not have this built in, I thought lets build it and show you guys how to integrate the Slack-API, Pandas (a multipurpose data scientist tool), Flask (a tiny python web server) and Heroku (a place to host your apps).

Continue reading about A recommender system for Slack with Pandas & Flask

Tags: , , , , ,

Symfony: A Tool to Convert NelmioApiDocBundle to Swagger PHP

We have an API built with Symfony that outputs its specification in the Swagger format. We needed to upgrade from version 1 to 2. As we switched the library to generate the specification while upgrading, we had to convert the configuration. In our case that configuration was so extensive that we decided to build a script to convert the configuration.

Swagger is a standard to document REST APIs. Using a JSON file, an application can document its API. Swagger specifies the path for each resource and allowed HTTP methods, as well as input parameters and the returned data. On top of this specification, tools like Swagger UI can automatically provide an API client in a browser. This is an excellent way to explore the documentation and also very helpful when investigating data issues.

We have been using NelmioApiDocBundle with our application for a while now. This bundle reads annotations on the controllers and combines them with the Symfony routing informations to produce an API documentation in the Swagger 1 format. Support for Swagger version 2 however was not available in NelmioApiDocBundle at the time of this blog post. We would have stayed with NelmioApiDocBundle, as it worked well for us, but we did not want to invest the time to refactor that bundle to Swagger 2.

Continue reading about Symfony: A Tool to Convert NelmioApiDocBundle to Swagger PHP

Tags: ,