Early in the year, I wanted to do my accounting as usual. However this time, my usual script threw an error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 11397: invalid continuation byte. Something in python from the version I used in the the past to 3.14.5 has changed with regards to encoding/decoding. The error message is cryptic enough, that I rapidly gave up trying to fix it.

This upset me for two reasons: firstly, why does code has to rot so fast? Why is every version bump of a software such a huge risk? I run a very heterogeneous infrastructure, where keeping up with the changes across all technologies is simply not feasible. Secondly, why does this keep happening, even now, where I run Guix as a package manager, whose selling point is reproducibility? This second question I can answer of course: for guix to be reproducible, one needs to declare a manifest and a description of the git commit of all relevant channels. I had created a manifest including all the tools needed to run my accounting but had left out python, which was always available in the main profile. This is why running guix shell with --pure or --container is important, so that the main profile cannot leak into your shell. I had not done so. In the future I need to be more careful with setting up reproducibility from the start to ensure I can run such tools well into the future.

Right now, I could spend time fixing the python script or move to a new tool and I chose the fun route. I discovered csv2ledger. It builds on csv-mode to step through each csv entry and convert to the format used by ledger, passing through a few functions that support automatic matching of payees. In the tradition of Lisp, the design of the software uses composable elements. I felt right at home in the code (which is in total 562 lines). I did spend some amount of time to customize it to match my expected behaviour. I now have two banks, with different CSV formats. In particular, one bank has a column for the payee, while the other includes that information in the description. To replicate the behaviour I was used from icsv2ledger, that would also include the relevant payee when matching on description, I had to extend the current format to a third field. I will try to upstream this if the author is interested.

Next, I walk you through a few customizations. The README in the csv2ledger repository already provides many examples. I also found it a very elegant design that differences in parameter between banks is stored as local variables using Emacs’ Directory Local system (via .dir-local.el).

I use the following, with a small deviation from the csv2ledger README

  (defun samd/c2l-convert-amount (amount)
    "Convert AMOUNT from the format \"-3.150,20\" to \"EUR -3150.20\".
This also handles cases such as \"300\" and \"8,7\"."
    (string-match "\\(-\\)?\\([[:digit:].]+\\(?:,[[:digit:]]+\\)?\\)" amount)
    (let ((sign (or (match-string 1 amount) ""))
          (amount (thread-last (match-string 2 amount)
                               (string-replace "." "")
                               (string-replace "," ".")
                               string-to-number)))
      (format "EUR %s%.2f" sign amount)))

It turns out you can get a list of all payees from ledger by running ledger -f your-ledger.ldg --no-pager payees. This can then be used to complete a previous payee.

(defvar ledger-binary "ledger")
(defvar payee-list nil)
(defvar main-ledger "/path/to/your-ledger.ldg")
(defun samd/ask-payee (transaction)
  (if (alist-get 'payee transaction nil) transaction
    (progn (unless payee-list
               (setq payee-list (process-lines ledger-binary "-f" main-ledger "--no-pager" "payees" c2l-base-account)))
             (let ((payee (completing-read (format "(%s «%.75s») Payee: "
                                                   (alist-get 'amount transaction "0.00")
                                                   (alist-get 'description transaction "?"))
                                           payee-list)))
                                        ;TODO if payee not part of payee-list, push onto it
               (push (cons 'payee payee) transaction)
               transaction))))

I my workflow I have a main.ldg file with include statements for the ledger files that get auto-completed from bank CSV files. These are usually named by replacing .csv with .ldg into a sibling folder journal. So I wrote the following script to name the result buffer containing the converted transactions and include it in the main-ledger

;;needs to be called from the CSV file on which you started the conversion
(defun samd/finish-accounting ()
  (interactive)
  (let* ((csv-name (file-name-base (buffer-file-name))) ;TODO: ensure it ends in csv?
           (txn-name (concat "../journal/" csv-name ".ldg")))
    (pop-to-buffer "*Csv2Ledger Results*");go to buffer with the results
    (ledger-sort-buffer) ; sort the entries in that buffer, which is a ledger-mode buffer
    (write-file txn-name) ;write the buffer to a file
    (append-to-file (concat "include " txn-name "\n") nil main-ledger))); import the file in the main ledger

So now, after having called c2l-convert-buffer in the csv buffer I call samd/finish-accounting and be done!

Here is the csv2ledger configuration snippet that ties it together:

(setopt c2l-target-match-fields '(description)
          c2l-fallback-account "Expenses:Unknown"
          c2l-alignment-column 78
          c2l-auto-cleared t
          c2l-field-modify-functions '((date . c2l-convert-little-endian-to-iso8601-date)
                                       (posted . c2l-convert-little-endian-to-iso8601-date)
                                       (amount . samd/c2l-convert-amount))
          c2l-transaction-modify-functions '(samd/ask-or-match-account samd/ask-payee c2l-create-title)
          c2l-entry-function #'samd/remove-empty-txn)

The different .dir-local.el files configure c2l-accounts-file, c2l-account-matches-file, c2l-base-account, c2l-csv-columns and csv-separators. My own fork including some changes yet to be upstreamed (do not erase the result buffer so that converting can be interrupted) and those that probably will not (changes to the format of the account matchers file) is found here: https://codeberg.org/sam-d/csv2ledger