blogng/blog/2019-01-23-experiments-in-owning-data:-software.markdown

---
layout: post
title: "Experiments In Owning Data: Part 2"
date: 2019-01-23
comments: true
tags: freebsd, nextcloud, data
---

_[Checkout part 1](/blog/2019/01/20/experiments-in-owning-data/index.html)_

I wanted the setup to have as much less friction as possible, more time
maintaining systems and uploading data is less time for things that I actually
enjoy. Luckily a lot of open source tooling exists to help in this space, some
mature, some not so much.

I stumbled upon these two apps during this experiment and they have become my
daily drivers of sort -

### Nextcloud

[Nextcloud](https://nextcloud.com/) is your one-stop cloud storage app. The core
of nextcloud is a file storage server with a web interface, easy enough right?
True to the open source nature it is heavily [extendable with apps (more like
plugins)](https://apps.nextcloud.com/) - and there is an app for everything!
hosting calendars, editing office docs, taking markdown notes and even video
calling.

Nextcloud comes with companion mobile apps for Android and iOS. The apps let you
access and upload data to the nextcloud server. They can also automatically
backup new photos to nextcloud! Loosing a device never means loosing photos.

The server also exposes a [webdav](https://en.wikipedia.org/wiki/WebDAV)
interface, so one can just mount it as a folder with
[davfs](https://savannah.nongnu.org/projects/davfs2) over fuse. Webdav is not
known for being very performant, but I like the simplicity of it and works well for my day today needs where I just need to work on a couple of small files.

Nextcloud has replaced Google Drive/Photos for me. The whole process is not
without pains. Documentation exists and is helpful, but sometimes falls short.
I am yet to figure out how to setup
[Collabora](https://nextcloud.com/collaboraonline/) (online office suite).

### Tiddlywiki

[Tiddlywiki](https://tiddlywiki.com/) claims itself to be a non-linear personal
notebook. That's quite vague, because it depends entierly on the user. I use it
to store tiny bits of notes, lists, code etc. I have managed to move everything
from evernote to tiddlywiki with the exporter plugin. Formatting of exported
notes is not great, but the content is fine - and that's all I care.


## Current and Future
I have been running this for almost an year now, other than nextcloud and
tiddlywiki, the server also runs an email server (not my primary one yet), code
repo and a CI setup.

Where to now? I am obviously missing out on features, and I had to spent more
time setting it up than signing up for a service. But the setup is tailored to
my use and so far I am happy with it.

That is not to say I think its perfect - I have a wishlist!

### 1. Better photos interface and search
I miss the fast and accurate photos search on google photos, and the current
nextcloud photos interface is slow and only has file system hierarchy as a
grouping mechanism. Maybe I can [inception-v3](https://arxiv.org/abs/1512.00567)
or any other similiar image classification models and build an auto tagging
system?

### 2. Disaster proof backups
Current backup system is quite rudimentary, it copys data every day to another
server in the same datacenter (also on `raid1`). Its not great if the DC itself goes
under - I would like something offsite, and also to backup server configs.

Well, that's about it for now. :-) I do have a question to the reader: do you
know any applications in this space that I should know about? please let me know over
[twitter](https://twitter.com/notmycommit) or [email](mailto:mail@dbalan.in).
Experiments in owning data, part 2 2019-01-23 21:00:25 +00:00			`---`
			`layout: post`
			`title: "Experiments In Owning Data: Part 2"`
			`date: 2019-01-23`
			`comments: true`
Convert from old categories to tags import sys import yaml with open(sys.argv[1]) as fp: data = fp.read() if not data.find("---") == 0: # no head print("NO YAML HEAD FOUND") sys.exit(-1) data = data[3:] head_end = data.find("---") head = data[0:head_end] data = data[head_end+3:] metadata = yaml.safe_load(head) cats = metadata.pop('categories', None) if cats != None: if type(cats) == list: tags = cats elif type(cats) == str: tags = cats.split() tags = list(map(lambda t: t.lower(), tags)) metadata["tags"] = ", ".join(tags) new_data = f"---\n{yaml.dump(metadata, default_flow_style=False)}---{data}" # write it print(f"coverted: categories to tags: {tags} - {sys.argv[1]}") with open(sys.argv[1], "w") as fp: fp.write(new_data) sys.exit(0) if not metadata.get("tags", None): metadata["tags"] = "untagged" new_data = f"---\n{yaml.dump(metadata, default_flow_style=False)}---{data}" print(f"untagged: {sys.argv[1]}") # write it with open(sys.argv[1], "w") as fp: fp.write(new_data) sys.exit(0) print("No changes needed") 2019-01-28 22:16:27 +00:00			`tags: freebsd, nextcloud, data`
Experiments in owning data, part 2 2019-01-23 21:00:25 +00:00			`---`

			`_[Checkout part 1](/blog/2019/01/20/experiments-in-owning-data/index.html)_`

			`I wanted the setup to have as much less friction as possible, more time`
			`maintaining systems and uploading data is less time for things that I actually`
			`enjoy. Luckily a lot of open source tooling exists to help in this space, some`
			`mature, some not so much.`

			`I stumbled upon these two apps during this experiment and they have become my`
			`daily drivers of sort -`

			`### Nextcloud`

			`[Nextcloud](https://nextcloud.com/) is your one-stop cloud storage app. The core`
			`of nextcloud is a file storage server with a web interface, easy enough right?`
			`True to the open source nature it is heavily [extendable with apps (more like`
			`plugins)](https://apps.nextcloud.com/) - and there is an app for everything!`
			`hosting calendars, editing office docs, taking markdown notes and even video`
			`calling.`

			`Nextcloud comes with companion mobile apps for Android and iOS. The apps let you`
			`access and upload data to the nextcloud server. They can also automatically`
			`backup new photos to nextcloud! Loosing a device never means loosing photos.`

			`The server also exposes a [webdav](https://en.wikipedia.org/wiki/WebDAV)`
			`interface, so one can just mount it as a folder with`
			`[davfs](https://savannah.nongnu.org/projects/davfs2) over fuse. Webdav is not`
			`known for being very performant, but I like the simplicity of it and works well for my day today needs where I just need to work on a couple of small files.`

			`Nextcloud has replaced Google Drive/Photos for me. The whole process is not`
			`without pains. Documentation exists and is helpful, but sometimes falls short.`
			`I am yet to figure out how to setup`
			`[Collabora](https://nextcloud.com/collaboraonline/) (online office suite).`

			`### Tiddlywiki`

			`[Tiddlywiki](https://tiddlywiki.com/) claims itself to be a non-linear personal`
			`notebook. That's quite vague, because it depends entierly on the user. I use it`
			`to store tiny bits of notes, lists, code etc. I have managed to move everything`
			`from evernote to tiddlywiki with the exporter plugin. Formatting of exported`
			`notes is not great, but the content is fine - and that's all I care.`


			`## Current and Future`
			`I have been running this for almost an year now, other than nextcloud and`
			`tiddlywiki, the server also runs an email server (not my primary one yet), code`
			`repo and a CI setup.`

			`Where to now? I am obviously missing out on features, and I had to spent more`
			`time setting it up than signing up for a service. But the setup is tailored to`
			`my use and so far I am happy with it.`

			`That is not to say I think its perfect - I have a wishlist!`

			`### 1. Better photos interface and search`
			`I miss the fast and accurate photos search on google photos, and the current`
			`nextcloud photos interface is slow and only has file system hierarchy as a`
			`grouping mechanism. Maybe I can [inception-v3](https://arxiv.org/abs/1512.00567)`
			`or any other similiar image classification models and build an auto tagging`
			`system?`

			`### 2. Disaster proof backups`
			`Current backup system is quite rudimentary, it copys data every day to another`
			server in the same datacenter (also on `raid1`). Its not great if the DC itself goes
			`under - I would like something offsite, and also to backup server configs.`

			`Well, that's about it for now. :-) I do have a question to the reader: do you`
			`know any applications in this space that I should know about? please let me know over`
			`[twitter](https://twitter.com/notmycommit) or [email](mailto:mail@dbalan.in).`