Android ORM Performance 2016

With greenDAO 3 released, it’s time again to look at the Android ORM performance landscape and do some benchmarks. This time, we also tested newer ORMs for SQLite like DbFlow, requery, SQLDelight, and SquiDB. Also, we had an extensive look at benchmarks done by others. Let’s get started with our results:

As you can see greenDAO comes in first in for all operations while there is no distinct second place. While this would be easy to claim, we are trying hard to present straight facts here. Objective benchmarks are really difficult to do and we did our best to make this comparison fair. All our benchmarks are open source, so you can verify the procedure for yourselves. Let us know if we can improve.

Other ORM performance benchmarks

Critical minds may still have doubts, which is perfectly understandable because we still might favor a specific product. Granted, so let’s have a look at other popular benchmark projects by Raizlabs and Kevin Galligan.

Benchmarks by Raizlabs

Raizlabs is an interesting case. In early 2015, Raizlabs claimed DbFlow to be the fastest Android ORM. Initially, they did not mention greenDAO at all. Soon, readers left various comments asking about greenDAO. Thus Raizlabs added greenDAO in their benchmark suite, and commented that DbFlow was faster for inserts while greenDAO may have advantages when it comes to loading. Because concrete results still lacked, we ran Raizlab’s benchmarking app with greenDAO included (smaller is better):
Seems like greenDAO comes out fastest right away. A quick look at the code suggests that greenDAO’s time for “Save” might further improve by exchanging the method “insertOrReplace” by just “insert”. But OK, let’s look at the other scenario called “Complex trial”:

Fixing Raizlabs’s “complex trial” benchmark

In this setup, DbFlow is the clear winner, right? But wait a second, why is greenDAO slower than OrmLite? And why is DbFlow loading that fast? It’s time to have a closer look at their code. As it turned out, the “write” test made greenDAO use 101 transactions while the others used just a single one. That makes a big difference, so we fixed it. The results from “load” test are a little harder to understand, but in short, DbFlow already had all entities in memory after insert, so it did not have to load any entity from the database. It’s easy to outperform others by leaps when you are the only one with a nicely filled cache in place. That’s why we filed an issue because we think the test setup does not make sense: either you test caching or loading, but never a mixture. Anyway, it was easy to make greenDAO use its caching mechanisms in their setup. Here are the revised results:

Yep, fixing those seemingly small flaws in the benchmark made a huge difference. In the revised benchmark, greenDAO clearly outperforms DbFlow for saving complex data. And for loading, greenDAO can also reuse previously inserted and thus “cached” entities in memory leading to close-to-zero “load” times.

Benchmarks by Kevin Galligan

Kevin Galligan is a pioneer in the Android ORM world: he’s the developer of the Android Adapter for OrmLite and also made an annotation processing extension for it. He gave a presentation at droidcon UK in late 2015 on Android ORMs accompanied by an blog post diving deeper also on ORM performance. His benchmarks gave a different picture than ours, especially when looking at greenDAO. Let’s look a bit behind the scenes to understand what’s going on.

The benchmarking code came initially from another developer and penalized OrmLite drastically. So Kevin fixed it. Unfortunately, the original code also had a couple of problems regarding greenDAO, which were left unnoticed. We pushed a couple of pull requests to Kevin’s GitHub repository to fix those too:

• For batch inserts, the method insertInTx should be used. Doing 1) manual iterations and 2) using the method insertOrReplace instead of just insert is less efficient. (see pull request #2)
• Like all other ORMs being tested, greenDAO should also use primitive types in the entity. Wrapper types like Long are somewhat expensive. (see pull request #3)
• greenDAO’s optional identity scope introduces a little overhead, which should be switched off in some cases depending on what you compare it with.  (see pull request #4)

After those modifications, we got the following results:

In Kevin’s revised benchmark, greenDAO and DbFlow are about the same speed for writing, while greenDAO seems to read data around 20% faster than DbFlow. While still being similar, it does not exactly match our findings. A possible reason could be that we used DBFlow 3.0 while Kevin used 2.0, but we did not take the time to investigate that. So, that is only a speculation.

About our ORM benchmark

Some remarks and additional info about our 2016 edition of our Android ORM performance benchmark:

• Benchmarked versions: greenDAO 3.0.1, OrmLite 4.48, DbFlow 3.0.1, SQLDelight 0.4.2, SquiDB 3.0.0, requery 1.0.0-beta23.
• The tests were run on a Nexus 5 with Android 6.0.1 (June 2016 patch level).
• The values are the average of 8 separate runs
• The benchmark code is open source
• Last year’s comparison benchmarked greenDAO, ORMLite, and ActiveAndroid. We dropped ActiveAndroid this year, because we think it’s not a good alternative anymore. Its development seems to have stopped and also its results were inferior.

Side note: We used the same hardware like last year. The values for greenDAO and OrmLite remained the same hinting that there were no performance improvements from Android 5.1 (2016) to Android 6.0.1 (2016) done in the internal SQLite version shipping with Android.

Summary

We cannot stress enough how hard it is to make objective performance benchmarks. Even if the benchmarking code is open source (which is great), it may not have been critically reviewed. We did our part and fixed a couple of issues in two other benchmark suites and invite you to do the same. After these revisions, the results are very similar to each other. This similarity should validate each of the (revised) benchmark suites: it is much less likely that three benchmark suites get it wrong.