Leaving Adobe Lightroom; or how to migrate from Lightroom to open source alternatives

This article describes my migration from Lightroom to open source alternatives, such as Darktable, using xmp sidecar files.

I will explain the required migration steps, my photography workflow as well as a bit about the technical background.

Preamble

I used Adobe Lightroom for the past ten years. Although I am in no way a professional photographer, I fell in love with this tool. Lightroom 5 was one of the few programs, I actually bought as a student – and felt happy with.

Unfortunately, with the introduction of a subscription based payment model by Adobe, I started to worry if spending 120€ per year on Lightroom is reasonable for my use-case.

Although Lightroom Classic is an option, it is not really usable on a Linux ecosystem. It is actually the last tool which ties me to a Windows machine.

I am currently using Lightroom 5.7.1 in my hardware accelerated Windows 10 Virtual Machine (VM), which I have initially created for gaming purpose only (get enlightenment here). It is working, but not as smooth as it could be. As I had to create a connection between the Windows VM – and the Linux file system, in order to access the pictures from both systems. This makes it cumbersome to use.

Okay I leave – from Lightroom to open source

In order to find the new perfect match, to fill the empty raw-file-developing-shaped hole, in my software heart, I had to understand what Lightroom was doing for me.

I came up with this list:

  1. Lightroom develops and processes (RAW-) image files – non-destructive ~when I make changes to an image, the original is still unchanged.

  2. Lightroom acts as digital asset manager (DAM). ~it allows me to have a folder based photo library structure.

  3. Lightroom reads and writes metadata. ~I can apply tags, color labels and image ratings during my development workflow.

  4. All information (development steps, sorting, meta data etc.) are stored in the Lightroom catalog file.

Additionally requirement for the new software is that it should run on a Linux platform, and is preferably open-source.

During my search for alternatives, some programs keep poping up: Darktable, digiKam and Raw Therapee

~Very long story short we will do the following:

Darktable will make most of the heavy lifting as it ticks all the boxes. Unfortunately, the DAM part of Darktable is not as advanced as I was used to by Lightroom. Thus, I will use digiKam as support for the digital asset management tasks (see upcoming article section).

The destination is set, what do we need?

I guess before we can continue a little technical background is required.

Technical background

Metadata

Metadata in general are additional information about files. These information can be directly embedded in the file.

When it comes to photography typical metadata information are:

  • Capture date and time
  • GPS coordinates
  • Author
  • Copyright information
  • Color labels
  • Picked / reject flags
  • Ratings

These metadata can be used to organize your library, search for pictures or can support your workflow.

Extensible Metadata Platform (XMP)

Directly from Wikipedia:

The Extensible Metadata Platform (XMP) is an ISO standard, originally created by Adobe Systems Inc., for the creation, processing and interchange of standardized and custom metadata for digital documents and data sets.

XMP Sidecar files

Some file formats (e.g. jpeg, png, dng, tiff) directly have some xmp fields embedded in their file types. Others, especially proprietary RAW formats not (or Lightroom/Darktable is not able to write them correctly)

Lightroom

Per default Lightroom stores every image related processing step, every metadata change (e.g. color label, image flag, etc.) in its catalog file (in order to keep the non-destructive image processing promise).

Additional information:

  • Version used in this guide: Adobe Photoshop Lightroom Classic (5.7.1)
  • Sidecar files in Lightroom are optional
  • The naming pattern of the xmp sidecar files is <filename>.xmp
  • Sidecar files are created when metadata for an image is saved
  • Lightroom allows only one color labels per image
  • “Picked” and “Rejected” flags are supported
  • Lightroom supports “Star ratings”

Darktable

The thing is Darktable works a little different. The image information, processing steps etc. are stored in so called xmp sidecar files. Each image file has its own xmp file, which is stored right next to the image in the folder, and created when Darktable.

Additional information:

  • Version used in this guide: Darktable (2.6.0)
  • Sidecar files in Darktable are necessary.
  • The naming pattern of the xmp sidecar files is <filename>.<original extension>.xmp
  • Sidecar files are created during the image import process
  • Darktable allows more than one color label per image
  • “Picked” flag is not support
  • “Rejected” flag is supported
  • Darktable supports “Star ratings”

My workflow in Lightroom

The following part might not be necessary, but I think it can help to show where we are heading to – *drum roll*

My photography workflow


Okay, let’s start migrating

So what’s the game plan? Well, easy – get the information stored in the Lightroom catalog and transfer it to Darktable. Unfortunately, Darktable is not able to directly read Lightroom’s catalog file, but Lightroom is able to additional create sidecar files, and these can be read by Darktable.

Lightroom settings

Make sure the setting “Automatically write changes into XMP” is enabled. This creates xmp sidecar files when metadata is saved in Lightroom.

Open Lightroom settings via:

“Edit” ➡️ “Catalog Settings”

My catalog settings in Adobe Lightroom 5.7.1
Image 1: My Adobe Lightroom catalog settings.

Make sure the color labels are to have their default values.

This is important!

“Metadata” ➡️ “Color Label Set” ➡️ “Edit”… (or simply choose “Lightroom default” if available)

Metadata default settings in Adobe Lightroom 5.7.1
Image 2: Metadata default settings in Adobe Lightroom 5.7.1

Darktable settings

The following setting will allow Darktable to check for metadata/sidecar file changes during each startup – instead of just once during the image import process.

Open the Darktable global preferences by clicking on the gear symbol in the “lighttable” view.

Darktable settings gear in lighttable view

Image 3: Darktable settings gear in lighttable view

Go to the “Core Options” register and enable the option “look for updated xmp files on startup.”

Metadata transfer overview

Which metadata one wants to transfer, is heavily workflow related. There are also settings one should not (want to) transfer between Lightroom and Darktable (see ❎️ marks in transfer table). As Darktable and Lightroom differ in the used “RAW processing engine”, settings such as, exposure values would lead to different results in the two applications.

The following table show the results of different types of metadata written in Lightroom, followed by an Darktable import.

✅ = successfully imported

❎️ = successfully not imported

😥 = not imported

  JPEG PNG TIFF DNG
(Adobe RAW)

CR2
(Canon RAW)

ARW
(Sony RAW)
Color Label ✅ ✅ ✅ ✅ ✅ ✅
Star rating ✅ ✅ ✅ ✅ ✅ ✅
Tagging ✅ ✅ ✅ ✅ ✅ ✅
Flagged as “picked” 😥 😥 😥 😥 😥 😥
Applied image crop ❎️ ❎️ ❎️ ❎️ ❎️ ❎️
Applied auto-exposure ❎️ ❎️ ❎️ ❎️ ❎️ ❎️
Applied B&W preset ❎️ ❎️ ❎️ ❎️ ❎️ ❎️
Added Author, Title, Caption, Copyright and sublocation ✅* ✅* ✅* ✅* ✅* ✅*

*sublocation is not supported by Darktable

The migration from Lightroom to Darktable step-by-step

Preconditions:

  1. Darktable is not running.
  2. Make sure no Darktable xmp files exist (i.e. files named <Filename>.<extension>.xmp) for the images you want to import.
  3. The Darktable library file has no record of the images you want to import.
    This can be done either by right clicking a folder and choosing remove in the “collect images” section in the “lighttable” view while it is sorted by folder. Or one deletes the complete Darktable library file (default location ~/.config/darktable/darktable.lib). In case the image used to be in the library make sure to double-check step 1 and 2.

Step-by-step:

  1. Identify the metadata attributes you want to transfer, and make sure every image you want to transfer has these metadata attributes set in Lightroom.
  2. Now save the metadata into the xmp sidecar files:
    In Lightrooms “Library” view, select all images you want to transfer to Darktable and press “CTRL + S” (or use “Metadata” ➡️ “Save metadata to image” in the menu bar.). This step will create/update xmp sidecar files.
  3. When everything is done – close Lightroom
  4. *optional step
    In case you want to use a different root album folder for Darktable, now is the time to copy the folder to its new location.
  5. Open Darktable and choose “folder” from the import section in the “lighttable” view (see image 4, red marked area).
  6. Select desired the folder and press the “open” button. In case the folder contains sub-folder with images, make sure to check the option “import directories recursively” under “import options”.
  7. Verify the imported images result. All images known to Darktable can be found in the “collect images” section in the “lighttable” view (see image 4, blue marked area). In order to feel a bit more familiar with the structure, select “folders” from the drop-down menu in the section (see image 4, green marked).
Darktable import and collection section
Image 4: Darktable import and collection section

To be continued …

 

This article is the first part of my leaving Lightroom series:

16 comments on “Leaving Adobe Lightroom; or how to migrate from Lightroom to open source alternatives”

  1. Pascal

    Hi,
    Do the post-treatement values written by Lr fit correctly in Dartakble ? In other words, this means Darktable translates correctly Lr values in is own referential of value ?

    Reply
  2. Graham Byrnes

    A liitle side discovery: if you use a camera that writes directly in DNG (eg Pentax or Leica), LR will not write sidecars. It writes the information inside the DNG file…

    To get it out, you can use exiftool to batch-process your DNG files and write xmp sidecars… in principle. I had a few issues due to some security features in W10 when I tried running big batch jobs, but the solution likely lies in that direction.

    Reply
    1. Iván

      Hello, I’m thinking to migrate but I have a lightroom folder structure (year/month/day) and the most of my files are DNG. How could make a batch to get the xmp sidecar files?

      Thank in advance.

      Reply
      1. Mathias Hueber

        I don’t see why this shouldn’t work. If Lightroom creates the sidecar files Darktable will import them. The folder structure should work. Make a test run with only a test picture compilation.

        Reply
    2. Carl-Erik

      Good tip! I was thinking of using a RawTherapee + DigiKam setup, and I use Pentax, so that means writing a small batch script just like that. Thanks!

      Reply
  3. Tony Hamilton

    For me, this is a most valuable document in what is going to be a major challenge. I’m not going to start this migration (which is surely one way only ?) until I am a confident in my use of Darktable as I am in my use of Lightroom. This will take a few months yet. I am concerned that things change so rapidly on the web that I will not be able to find this page again when I need it. Could you possibly make it available as a downloadable PDF, or similar? Or is there some other way I can keep a local copy ?

    Reply
    1. Carl-Erik

      Every browser out there can make any web page into a PDF. You do not need anyone else to do it for you. On Windows you usually have a “Microsoft PDF Printer” in the list of printers, if you just try to print a page using Ctrl-P. Mac already has this as well. Google Chrome on all platforms has a “Save as PDF” option in its printing dialogue as well.

      Try! If you don’t manage, just post a question on SuperUser or check Youtube.

      Reply
  4. 8BitPapa

    What about the edits to the photos themselves? Color corrections, exposure adjustments, healing brush areas, over and underexposed areas etc. Can all those be migrated to darkroom?

    Reply
    1. Mathias Hueber

      No, none of those. Also, you don’t want it to be transferred, as Darktable simply works different. My workflow solves this problem for me, as I export the final edited image to png/jpeg and import those side by side with the (then untouched) original.

      Reply
  5. Tim

    Hi and thanks for this very useful article !
    Like you, I am switching from Lightroom to Darktable x Digikam. But I have still one question unanswered : can you have a double way to organise your pictures like in Lightroom with folders and collections ? Is it possible to do that in Darktable (I am not sure) or in Digikam ?
    Thanks for your help !

    Reply
  6. Carl-Erik

    > My new workflow with Darktable and digiKam *upcoming
    Is this still upcoming? It’s been a year since this was published 😉 Really looking forward to it.

    Reply
    1. Mathias Hueber

      Ahh, you got me – my second daughter got in the way 😉
      It is still on the todo-list for 2020 – promise!

      One “problem” is also that open-source projects really got a push during corona. With little time on hands, it is pretty hard to keep up with the new changes and create helpful content from it.

      I haven’t had time to look in detail on the 3.2 Darktable changes, but I am tempted to go completely with Darktable and ditch digiKam. Although, digiKam received some new features for custom RAW editors and meta-data updates.

      Reply
  7. Brandon L Woodard.

    I, too, am anxiously awaiting your new workflow article (Nov. 2020)! I’m currently in the process of planning the migration of my files and realized that 90% of everything from 2014 until early 2020 are all DNG-converted files — and no embedded RAW files!

    I took a test file, made adjustments to it, and used ExifTool w/ GUI on Windows to extract the XMP. Importing it w/ XMP into darktable allowed me to get most of the adjustments (really just did exposure and tone curve for testing if it worked or not).

    The end result was a success — I was able to get the XMP data for the current adjustments to the DNG file. However, it looked hideous in darktable. The “settings” applied in darktable were a bit different than how LR interprets them — the tone curve was not as smooth, but faintly resembled an S-curve.

    So, all my photos from the last 5-6 years? 90% of them I will lose the post-processing because I converted to DNG and did not embed nor keep the original RAW files. However, most of those were family photos like parades for holidays, etc. and the oldest ones were simply beginner photos, learning and experimenting.

    In the end, this will be painful for me…but I’ll better learn to cull and let go from here out — and never delete or convert the original RAWs again!

    Reply
  8. HolosPhoto

    Great job Mathias!
    In IT since 1983… Finally leaving from MS Windows to embrace fully Open Source ecosystem software.
    I have a Lightroom archive of about 190K IMAGES converted in DNG with embedded RAW… But half of the archive still with partial editing not exported or finalized yet to final JPG.
    As I understood with big pain, is not possible to transfers to DK or DT settings of CROP, EXPOSURE, ROTATION, and all edits made from Lightroom DEVELOP panel.
    So the only way towards a migration to Linux Open Source software is to finish all edits in Lightroom and export a final JPG, right?
    There are advanced DNG managers for Linux?
    Very important to UPDATE analysis of migration process with LAST VERSIONS OF LIGHTROOM, DIGIKAM 7.1, DARKTABLE AND RAW THERAPEE… Does exists from November 2020 or someone has this plan for next days?
    Thank to all for sharing experiences. FHB @holos_photo

    Reply
  9. Heiko Sieger

    Hi Mathias,

    Thanks for sharing your suggestions and findings on migrating from LR to open source. I’ve been contemplating this myself for a long time. However, every time I tried open source alternatives to the Adobe set of photography tools I found that several things were either missing or not up to the same quality. Over time things got better, but there is still one thing holding me back: about 150K edited RAW images that would have to be completely redone.

    It would be nice if there was a kind of a “filter” that you could run the LR XMP files through to get translated into the corresponding Darktable settings. Do you think that such a “translation” tool is feasible?

    Another thing I noticed: You wrote about metadata but did not mention keywords or keyword hierarchy, both supported in LR. This is crucial to me as most of my photos use keywords and sometimes keyword hierarchies. I rely heavily on these to search and/or identify photos. Other important metadata that I use all the time are title, caption, label, headline as well as the sublocation. When I export images to jpg for posting on the web, the gallery app can automatically fill in fields using the metadata. Any thoughts on that?

    Finally the raw engine: what are your thoughts on the quality of Darktable or other RAW converters such as RAWtherapee? In the past I felt that the open source tools – though improving – are not on par with LR (and that isn’t perfect either). I spent a little fortune in cameras, lenses, tripods and what not to get good quality and would not want to compromise on that.

    Thanks again for sharing your experiences, which are very helpful.

    Heiko

    Reply
    1. Mathias Hueber

      Hello Heiko,
      it seems we share more than one hobby 🙂

      In the beginning I was looking for a tanslation tool as you describe it myself, and even thought about building one myself. The problem with a translation would be the different raw converter engines. One can not simply translate the values one by one. I don’t think it is feasible. I do not have the problem as my workflow ends in “finished” images while I keep only the raw backups for the promising images. I describe it a bit in the outdated “synchronization between darktable and digikam” article.

      The only thing that makes sense to translate are meta data as rating, flags and tags. I have never used hierarchies in tags. I have to try that out.

      A big difference between DT and lr is that while in lr each settings module is meant to work on a certain aspect of the image, in DT there are overlapping functionalities between modules. With the more recent DT releases the LAB workflow has been introduced as alternative to the RGB version. So you can now choose between even more modules now (pixels.us has a very detailed piece on that topic)

      My biggest problem during the migration has always been that I want DT to act and behave exactly the same as lr does. This never worked and got me frustrated. In the end, instead of looking up how I can make DT behave like lr, I does DT work and why. This is tedious but pays of in the long run.

      I am currently preparing a complete rework of this whole lightroom open source migration series. Let’s hope I am done soon 😉

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *