TIL git filemode and interactive rebase

I just migrated to a new operating system (windows -> macOS) and have restored the system from my backups. I noticed that I kept losing the executable bits on files in my git repository. As it turns out the repository had filemode set to false (which is the default on Windows, an OS that does not recognize executable bits). Since I did not clone the repo that configuration was kept. So I had to manually set it to ‘true’.

Getting Data Into The Datalake

Here I assume you have some kind of storage set up, let’s say a NAS and have personal data scattered across many hard drives, USB thumb drives and other data sources. I wrote the following script to ingest this data into the personal datalake. It will first create an image of the whole disk and then copy all files into the datalake, preserving attributes such as timestamps. In a last pass the data is checksummed to make sure the transfer did not introduce any errors.

Personal Datalake: Concepts and Goals

In many industries there is a need to aggregate all business data and analyse it to drive business decisions. In the 90’s and early 2000’s the concept of data warehouses was popular. Large databases were aggregating all data in a unified schema. More recently the concept of a datalake was defined. In a datalake many different sources feed into the lake, dumping raw data in their own schema without worrying about unification.

NAS Setup Redux

I lost access to my encrypted drives. Here is how it happened. After setting up my NAS using full-disk encryption on both disks, I had the nagging feeling of not remembering the passphrase I chose or the keyfile I used. At this point I should have just read my previous blog post because the main reason I am writing these things down is to document them. It would have told me that I used a keyfile and not a passphrase.

Setting Up Nas

I finally picked up my helios NAS from customs and started setting it up. Building it was straightforward using the instructions published on their wiki. I am using the Armbian image provided by the creators, with the intent of using OpenMediaVault to administrate the NAS. Due to OMV 5 not being recommended for production use just yet, I used the Armbian version based on Debian 9 instead of the recently released Debian 10.

Testing New Drives

To set up my network attached storage (NAS), I recently ordered two 4 TB WD Red drives. After some research (here and here) I came up with the protocol below. Note: run all commands as root and replace /dev/disk with the appropriate device name for your setup: read out S.M.A.R.T attributes: smartctl -a /dev/disk > baseline perform a conveyance test to check for damages during transport: smartctl -t conveyance /dev/disk and compare to baseline perform a short test: smartctl -t short /dev/disk and compare to baseline run badblocks on complete disk: badblocks -wsv -b 4096 -t random -o badblocks.

Personal Finances Update

I described my setup to track my expenses using ledger-cli and some homegrown scripts in a previous post. I have been using this setup for a couple of months now. Previously I regretted not being able to automatically import my bank statements via HCBI. While having such a function would be very comfortable, logging into my bank (I only have one) and downloading the statements as CSV files, takes little time.

TIL GNU Stow

I stumbled upon GNU stow and a way to use it as a dotfile manager. I decided to implement this and upload my dotfiles to a private git repository. Incidentally I had already a similar system in place on an old PC that I wrote myself. Using GNU stow will help streamline this and ensure the software will always be available (at least on GNU/Linux platforms).

One Year Owning a Smartphone

I got my first smartphone in December 2017. Before that I was using an old blackberry phone that I got from my dad. I would use it only for phone calls and texts. My main apprehension against a smartphone was the fear of being perpetually reachable via email and other messaging application. What I did realise, was that a smartphone would be especially helpful during travels and in unfamiliar surroundings.

About 2018

At the end of 2017, I was feeling burnt out. I had been working two jobs, as well as working long days and 6 days each week. So I took a series of new year resolutions that I want to re-evaluate in this post. First, I decided to go back to a 5 day work week. Second I put two alarms on my phone, one for 5:30 pm as a reminder to start wrapping up and another at 6 pm with an awful alarm sound as a hard deadline to stop working.