the itjerk

my adventures with technology

Category Archives: Linux

backup

The data axiom is “always have at least two copies of anything you want to keep!”

Now that I’ve ripped my entire (well, almost entire) CD collection, I have to back it up. A RAID 1 drive is good protection from drive failures, but it doesn’t protect at all for accidental erasure, file corruption, etc. I’m going old school and bought a new 3TB disc, the same size as my RAID, and plugged it into a hard disc enclosure, the same model I have for my another backup drive; I only need to have the same wall-wart and USB cable handy. I formatted the disk with ext4, the same as the source drive, which prevents file-naming errors during backup. However, if you format your disk for use with Windows, you’ll need to install exfat-utils and exfat-fuse in Ubuntu. (I also recommend doing the initial format on a Windows machine.)

I am using Grsync software to make the backup, which is a graphical front-end for the rsync utility. I marked the –update and –delete options, as I want to make an identical copy of the source on the destination: copy what’s not there, replace (based on checksum) what’s changed and delete what was removed from the source. You can perform a dry-run first; be sure to empty the trash and skip the lost+found folder before you sync (the latter may give errors). Viola, backing up FTW!

When deciding on a backup method, it’s important to always remember what you’re backing up and why you’re backing it up – and what risk you can afford.

In this case, these are music files, most of which I have a CD copy of but would never want to put in the months of work in to rip again. The rest are downloads, paid or otherwise, some I may never have access to again. Now, I could probably use something different method (cloud, internal disk), something automated, something better, but this method works for me because I can assume the risk.

I have an initial backup (which I have tested) and a proven backup method, so it’s up to me to keep up the work.

clean install mania

onedesktop.png
New hardware assembled in no time, and yes it’s a perfect match (well, except I need to rob a mounting bracket and four-pin fan (mATX) from the old machine). For the money, I’ve done well, and I hope it lasts as long as the previous.

Clean install of Ubuntu 18.04LTS was fine, except first time I must have miss-typed my password because I couldn’t login. Second time I got it right, but also decided to do a “minimal install” with full disk encryption. The former, because, the latter also because, but I will say there are potential pitfalls when rebooting because you must type the password to mount the drive.

I installed a whole lot of apps (Audex too), LAMP server with two websites (one requiring php5.x from here), two music streaming servers, an openvpn server, and a whole lot more. Most things were easy, most things didn’t require magic or luck, and it’s liberating in a way to leave things behind, and also to see how things work on a very clean system.

The big takeaways are this: document, document, document what you’ve done. Whether in a blog (like this), using screenshots, sending yourself email, referring to bash_history files, or whatever, if you did something once, you may have to do it again, so tuck it away where you can find it. As we all know, IT professionals are just very good at google searches; but they’re not always that efficient, and after a while the mania sets in:

In my drive for perfection, I did f&ck things up by deleting a directory (or more) in /var/. Punch drunk on the keyboard? Three hours of sleep? I certainly wasn’t thinking straight! Anyway, very luck to recover, as I was almost to the point where I needed to redo the entire clean install again!

So it’s all good, all systems go. Yet I still haven’t migrated any user data (other than my music library, and websites), not even any bookmarks. Yet.

ubuntu 18.04 lts

Desktop upgrade time. The latest version of Ubuntu, 18.04 lts “Bionic Beaver,” was released last week, so I decided to upgrade my desktop computer in situ from 16.04LTS. There’s lots of changes between LTS versions, but the big change here was the switch from Compiz/Unity display manager and desktop to Xorg/Gnome. The reason why I upgrade is that the LTS version is supported until 2023, though I have to admit that having a new UI was enticing, especially with Gnome Shell extensions.
sudo update-manager -cd
After the above command to make the upgrade available to Software Updater, I had errors. Nonetheless, Bionic Beaver installed, and I rebooted. The first error was with ca-certificates during upgrade, which is a known Bug #1767453. The second was a broken intramfs, which I solved by updating it for the current kernel, sudo update-initramfs -c -k 4.15.0-20-generic.

Bigger issue I had was with Xorg/Gnome. When I’d go to log in, I’d get an empty screen, though intermittently between reboots it would work. Ugh. So I reinstalled Xorg/Gnome, by doing this:
sudo tasksel install ubuntu-desktop
then uninstalled Compiz/Unity by this:
sudo apt-get purge compiz compiz-plugins-main-default libcompizconfig0

It ends up the issue boiled down to one of the Display Managers, lightdm or gdm3. I decided to purge lightdm and use gdm3, which after the following thorough reinstallation, seems to be working:
apt-get update
sudo apt-get -d install --reinstall gdm3
sudo apt-get remove --purge gdm3
sudo apt-get install gdm3

I also installed gnome-tweak-tool to move the min/max buttons to the left, and the new theme, Communitheme, because after 8 years of Ambiance we all need a new Ubunutu theme! I also found some useful Gnome Shell extensions, which I installed via the “chrome” plugin in Firefox (go figure!). Oh, and this:
gsettings set org.gnome.shell.extensions.dash-to-dock show-apps-at-top true

After the perfunctory sudo apt-get update/dist-upgrade/autoremove, I went through many things, like local copies of websites, Openvpn, etc. and found they worked. MiniDLNA was also running but Logitech Media Server needed to be reinstalled (with a new version: 7.9.1 – 1522157629 @ Fri Mar 30 12:25:29 CEST 2018).

Mostly good, and a nice change of desktop scenery!

dnscrypt2

Spurred on by some recent articles, I decided to switch to dnscrypt2. It’s an improved version, supports a whole slew of things and more resovers.

This was a bit of work, because, stupidly, I disabled dns! Anyway, long story short, I followed the instructions here, and everything worked out okay. I did to issue the following on resolv.conf to get it to ‘stick’:
chattr +i /etc/resolv.conf

desktop:~$ nslookup -type=txt debug.opendns.com
Server: 127.0.0.1
Address: 127.0.0.1#53

Non-authoritative answer:
debug.opendns.com text = "server m33.chi"
debug.opendns.com text = "flags 20 0 70 7950800000000000000"
debug.opendns.com text = "originid 0"
debug.opendns.com text = "actype 0"
debug.opendns.com text = "source 23.122.56.207:33649"
debug.opendns.com text = "dnscrypt enabled (714E7A696D657555)"

One the web:
DNScrypt

the big upgrade

Production website upgrades (do-release-upgrade from 14.04.5 to 16.04.4) are the most nerve-wracking ever. Being a “one-man show” means it’s all or nothing, no team to lean on, just my wits and google. Sure, I had a back-up if everything went south; yes, I had an idea everything would work; and yes, I ran into problems.

The local copy of my website needed an earlier version of php to work. That fix was easy enough, I just installed ondrej ppa for it. But I ran into a problem with the production site because I have iRedMail installed; so yes, the local copy isn’t completely like the production site. Here the problem was auth_mysql, as evidenced by sudo apache2ctl configtest. Two mods I have – awstats and cluebringer – were calling auth_mysql and preventing apache2 to load. The fix was fairly easy, apt-get install libaprutil1-dbd-mysql, and then clearing out the offending *.conf files. Thanks to iRedMail for information.

Seeing that website, email, etc live again was a good, good feeling.

A few weeks later, there were a few more things I had to tweak to get oh-so-right, but most of that was email related, and mostly non-OS related.

phpbb – new website

I have a web property, progressiverock.com, that’s been dormant ever since I rebranded my prog rock site after my book, strawberrybricks.com, a few years ago. One would think that the former URL is worth some money, but to date I have been unsuccessful in finding a buyer. So rather than just serving as a redirect to the latter URL, I decided to install phpbb and relaunch the site as a bulletin board to discuss all things prog rock and generate some brand recognition.

I use Digital Ocean for virtual hosting because it’s cheap (starting at $5/mo) and easy. The process to create a new virtual host “droplet” is simple enough: pick your choice of options (size, memory), hosting location, operating system (you can even get it pre-loaded with LAMP) and then setup the dns records. Within minutes, it was up and running as progressiverock.com.

Immediately after an apt-get update/dist-upgrade, I added some basics to the core system, like openssh server, ufw, postfix/logwatch and apticron. Most were straight forward installs, but for postfix, be sure to setup your A, MX and TX records before you start, and check your logs/errors for what to tweak; I had to add postconf compatibility and manually create the virtual alias map to clear errors I found in mail.log. I also setup sender_canonical because I just have a “no-reply” email system (for now). Also, don’t forget to set your timezone.

After configuring mysql and apache2, I added my rss feed, which needed the php-xml module installed to work. Let’s Encrypt was next, because why not — everyone should be using SSL. I also added awstats, which needed user www-data added to the adm group to correct the errors I generated by cron. The bulletin board software phpbb was quite simple to install; fortunately I remembered some basic mySQL commands to get the database setup beforehand. I then added American English as a language, and found feedpostbot, an extension that uses rss feeds to create topics — perfect for the “Album of the Day.” Forum hierarchy took a little thought, and I’m sure I’ll change it again before it all goes live. My next task is to get a new style for the site, but that my require some outside help. More later.

None of this was complicated, and most steps took but a few minutes to do. My big take away here is that log files and error messages are your friend: listen to them as they tell you exactly what to correct with your installation.

And if you want the domain progressiverock.com, make me an offer that I can’t refuse!

Update: I found the easiest way to prevent spammers from creating accounts is to use Q&A for the Captcha. 100% reduction in bogus accounts.

On the web:
phpBB • Free and Open Source Forum Software

dnscrypt

Domain Name Service (DNS) is the mechanism by where numeric IP addresses become readable domain names; it’s far easier for me to tell you to visit strawberrybricks.com than a bunch of numbers. When you browse the internet, then, the addresses you type or click on go through a DNS search. Typically, your ISP provides this service, or whomever you get your network connection from – however there is an implicit level of trust involved. Who’s to say that yahoo.com for example, is really yahoo.com? What is the DNS server spoofed the reply? Further, any DNS server can collect a wealth of information by recording your DNS requests. Finally, the speed of your browsing is dependent on how quickly these requests are filled.

Both Google (8.8.8.8) and OpenDNS (208.67.222.222) provide free DNS services that are fast and secure, and supposedly do not track your requests. A third service, Quad9 (9.9.9.9) was very recently launched. Your ISP has a lot of information about you. Switching your DNS to one of these providers is simple (just type them in your router, or network connection), and gives some degree of privacy. Every little bit helps?

DNSCrypt goes one further by encrypting all your DNS requests. It’s an easy enough program to install, available for PC, Mac and Linux, and for routers using DD-WRT. On my Ubuntu box, I needed to install libsodium-dev first, and then was most successful installing DNSCrypt-proxy from source by using the old “configure, make, make install” method with version 1.9.5. Then, you can run it with systemd automatically.

On the web:
DNSCrypt

minidlna

When talking about digital music servers other than Squeezebox Server, I feel like a cheater. It’s been my reliable go-to method for serving up my ripped and downloaded music for over a decade now. But not every piece of hardware speaks to it; Beep appeared a while back and saw me install miniDLNA on my linux box, where all my music files reside.

The Digital Living Network Alliance is a trade group that certifies compliance to a standard for delivering digital media. MiniDLNA is an implementation for Ubuntu, and mini it is! No interface (save a bare bones web page at port 8200), it is configured by editing /etc/minidlna.conf.

Set the path to your music; I’m only looking for audio files, so I mark the directory with an A.
#media_dir=/var/lib/minidlna
media_dir=A,/mnt/data/music

Set the database cache directory (important!) and enable logging:
db_dir=/var/cache/minidlna
log_dir=/var/log

Tell it to look for new files or not:
inotify=yes

Set the name of the server presented to clients. This provides a simple way to check if you’re connecting to you server.
friendly_name=My-MiniDLNA

That’s it! Restart the service after you make changes to the configuration,
sudo service minidlna restart

or rebuild the database if you’ve changed or added music.
sudo service minidlna force-reload

There’s a ton more it can do, including serving videos, pictures, etc, and it also offers per-user configuration as well; but for my purpose my newly acquired Oppo BVD-103 can now stream all the music on my computer.

EDIT: Also including a link for the bubblesoft add-on server. I use this with the Bubblesoft app to access MiniDLNA on my Android phone. Uses java and requires port 58050 to be open.

On the web:
MiniDLNA Ubuntu
ReadyMedia
bubblesoft

ssl grade a

Editing my /etc/apache2/mods-available/ssl.conf to use the following SSLCipherSuite changed my grade from SSLabs from B to A!

SSLProtocol ALL -SSLv2 -SSLv3 -TLSv1
SSLHonorCipherOrder on
SSLCipherSuite ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS

Check it out:
https://www.ssllabs.com/ssltest/analyze.html?d=strawberrybricks.com

ssl 24/7

While I’ve had ssl on my website for sometime (for anything login related), I had never enabled it by default. First, I had to install the patch the Video Filter module to work with https connections to Youtube. Then, using the developers tools built into Chrome, I found I had a http link to a Facebook logo (I have no idea why it isn’t local). That had to be fixed in the site’s theme. Finally, I found I had the remnants of ShareThis in a block. Although I deleted the module eons ago, I forgot about the block (which is how it appears on a page). Thankfully, those developer tools in Chrome made it plain as day. Now that all that was fixed, I edited the .htaccess file for the site, and entered the following to force https connections. (Remember to restart Apache after you edit .htaccess.)

RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://mywebsite.com/$1 [R,L]

With a free certificate from Let’s Encrypt, why not enable ssl. Oddly enough, only Chrome, Firefox and Microsoft browsers make it obvious when your connection to a website is secure. What’s up with that Apple?