the itjerk

my adventures with technology

Tag Archives: backup

my data and the cloud

Working from home gives me a lot of time in front of my computer. Being who I am – an itjerk – I’ve decided to streamline all my cloud services, and clean up my data wherever and everywhere it may be.

First, let’s talk a little about data. Data is ubiquitous. Folders, files, drives, cloud, it just piles up. Keeping it organized though is the key to keeping it safe. Specific categories of data require specific solutions. Here’s mine:

  1. Documents. These are almost exclusively on my PC’s home folder, or in Google Drive. I’ll use OneDrive to keep them backed up, until they are archived (see below).
  2. Photos. Camera photos are in Google Photos (Android Phone user) and then eventually backed up to my RAID1 and external drives, while other photos are in Pictures folders, or shared with Cloud services.
  3. Music. This resides on a RAID1 on my Linux computer, and is backed up to an external drive.
  4. Archived Data. IMPORTANT! Every year I create a folder on my Desktop for all the digital ephemera I collect. It gets backed up to my RAID1 and external drives. I also clean/clear out my home folders, or at least I endeavor to.
  5. Backups. Yep, that’s a class of data. I have a backup of my home folders on my RAID1 and external drives.
  6. The zillion other random photos and files. They are everywhere! In the cloud, in my Downloads folder, on flash drives and backup disks; this is the thing to organize and clean up!

Cloud services are great, but it’s pretty easy for things to get out of hand if you have too many. Here’s a quick overview of my Cloud Services:

Google Backup & Sync. Google let me down when they removed Google Photos from their sync services. I had my photos syncing down to a D: drive on my PC, which was easy enough to backup to external drives. Maybe too easy? No more – it’s a manual download process. Fuckers. Anyway it can do exactly what it says – backup your computer to the cloud, and the cloud down to your computer – but I’m moving on. Google knows enough about me.

Google Drive. This does have exceptional value to me, especially when I was writing my book, as a “work-in-progress” repository for documents. However, like all cloud services, it’s also a wasteland for random bits of data – photos, saved files, wip documents that never finish, transfers – all of which needs cleaning up much more so than backing up. I’ll use it for working in the cloud, but not as a repository for data.

Mega. Anyone remember Kim Dotcom? Whatever happened to him? Anyway, I use Mega to backup my book files. It’s a task-specific solution that serves my need, and is free.

OneDrive. I’m a newly converted fan. Why? Well, I use it extensively at work, but also because of my Surface Go: I used my Microsoft account (Live? 365?) to initially create my user account, and since the Go has only a 64GB hard drive, OneDrive’s is a must (but to be honest, I don’t store any data on the Go). But here’s another essential feature of OneDrive – it can automatically backup your Desktop, Documents and/or Pictures folder. The free plan comes with 5GB of storage, which is good for two of those folders, and 100GB plan only costs $2/month. It does have a Personal Vault (password protected folder), but it only holds three (3) files! LOL! Well integrated into File Explorer, and works with a Mac.

Dropbox. The original cloud, I think, it just ended up being a whole lot of meaningless files for me. I’ll keep the account but only for sharing with others.

Adobe Creative Cloud. I used CC apps for my book, but as this is provided by my work, it’s not a personal solution for me.

iCloud. I’m not an Apple user, but if I were, I would probably use this instead of OneDrive.

Box. Another service I use at work, but redundant with OneDrive.

Amazon Drive. If you have a Prime account, it offers unlimited photo storage. That’s tempting, but Amazon knows enough about me already. Plus, I want my precious digital photos in my physical possession.

So what’s the plan? Use Google Drive as a work space, and OneDrive as my primary cloud. As for the rest of my data, I’ve got a spare D: drive on my PC that I’m using as a staging point as I clean out my various folders and drives. Time I have, OCD I’m great at, so let’s execute the plan!

backup

The data axiom is “always have at least two copies of anything you want to keep!”

Now that I’ve ripped my entire (well, almost entire) CD collection, I have to back it up. A RAID 1 drive is good protection from drive failures, but it doesn’t protect at all for accidental erasure, file corruption, etc. I’m going old school and bought a new 3TB disc, the same size as my RAID, and plugged it into a hard disc enclosure, the same model I have for my another backup drive; I only need to have the same wall-wart and USB cable handy. I formatted the disk with ext4, the same as the source drive, which prevents file-naming errors during backup. However, if you format your disk for use with Windows, you’ll need to install exfat-utils and exfat-fuse in Ubuntu. (I also recommend doing the initial format on a Windows machine.)

I am using Grsync software to make the backup, which is a graphical front-end for the rsync utility. I marked the –update and –delete options, as I want to make an identical copy of the source on the destination: copy what’s not there, replace (based on checksum) what’s changed and delete what was removed from the source. You can perform a dry-run first; be sure to empty the trash and skip the lost+found folder before you sync (the latter may give errors). Viola, backing up FTW!

When deciding on a backup method, it’s important to always remember what you’re backing up and why you’re backing it up – and what risk you can afford.

In this case, these are music files, most of which I have a CD copy of but would never want to put in the months of work in to rip again. The rest are downloads, paid or otherwise, some I may never have access to again. Now, I could probably use something different method (cloud, internal disk), something automated, something better, but this method works for me because I can assume the risk.

I have an initial backup (which I have tested) and a proven backup method, so it’s up to me to keep up the work.

raid, again, and backup

I bought two 3TB drives to replace my 1TB RAID. Easy enough, because with hardware RAID 1 the disks are identical: you can pull one drive out, plug it into a USB enclosure, and provided your computer is compatible with the drive’s format — ext4 in my case — you will have two functioning backup drives to copy over to the new RAID (and eventually erase/shred).
sudo shred -v -nX -z /dev/sdX

A thing about buying a hard drive. I notice that the marketing has now shifted to “intended use” of the drive – Desktop, NAS, Surveillance, etc. Guess what, I call b.s. — they’re all the same drives, probably just the more expensive ones were tested more (to justify price/warranty). Please, let me know otherwise if you think I am wrong.

Maybe I should have got 4TB discs? I don’t know. Going from 70% used to 70% available is a jump. I do want to rip more of my CD library to my computer, because digital music is here to stay (and when I say “more” of my library, I mean a “curated” more of my library). I rip to FLAC, which is all the quality I require (considering the source is 16bit/44khz), so how much will I need? Figuring FLAC at level 5 is about 300-350 MB per CD, 2TB will hold a LOT of CDs.

2000000 MB / 330 MB = 6060 CDs

I should probably also mention that RAID 1 is not a backup. It’s a safety copy in case one drive fails. You want backup? You gotta have two of everything. One here, and one over there. More later.

The bigger question then is what do we do with all the data we have. If I really think about it, I need to back up my photos, SOME of my music, my data (book, website, etc), and what else? Do I really need all the crap on my computer? All the files on those backup drives and old hard disks? Probably not, because I don’t even know what I have most of the time and … p0rn should never be downloaded! 😉

My next project will be to organize all my “digitalia,” and what a project that will be!

nas-ty of backups

My Seagate NAS 110 took a dive this past week. After I couldn’t print, I then noticed the NAS kept on rebooting itself. Fortunately the disk was fine and I had all my data, the problem laying somewhere in the hardware of the device. Trucked down to Microcenter where I bought a new one, this time a Buffalo product.

All of this brings up backups in general, and the power of two. When you think of all the things you need to backup, there really isn’t that many. Photos and Videos are the most obvious, because the digital era allows us to take a gazillion of them, and who prints them out anymore? What else needs to be backed up? I wrote a book, so there’s a far amount of digital goodness with that. I’m sure there’s other things that I want backed up, in fact, I usually just put them on a backup drive and forget what I even have there!

Here’s where the power of two comes in: At a minimum, you need two of everything. Your backup will fail, just as my NAS did. Your hard drive may fail, as I’ve had happen before. Those flash drives? Fail. Hell, I even accidentally deleted some folders once. Recently. Ouch.

I like a NAS drive because all my computers can connect to it, so it centralizes the location of my data. That’s good and bad, good when it works, bad when it fails. I also use Dropbox for some things (current writings), DVDs for completed projects, and a portable HD for things I don’t want others to see.

Just remember. Have at a minimum of two copies of everything. One will vanish, when you least expect it.

raspberry pi and sd cards

As much as I'm enjoying my Raspberry Pi, I've come across one big issue: the SD card has failed twice now. Even though I'm using a "verified" card from this list, it keeps getting corrupt to the point where my RPi won't boot. Why is this? Just a bad card? Voltage on the board checks out okay, I use "sudo halt" to shut down cleanly, I don't manhandle the cards… so maybe it's from overclocking? Fortunately cards are cheap, and I'm not adding a whole lot to the standard Wheezy distro.

Here's how to backup and clone a SD card with Ubuntu. First, mount the SD card, then find out its device name in /dev/ by using ls. (You can also use "sudo fdisk -l").

ls -al /dev/sd*
... 
brw-rw---- 1 root disk 8, 32 Jan 15 18:50 /dev/sdc
brw-rw---- 1 root disk 8, 33 Jan 15 18:50 /dev/sdc1
brw-rw---- 1 root disk 8, 34 Jan 15 18:50 /dev/sdc2

That known (and correct), unmount the disk and use the dd "convert and copy" command to clone an image of the card to a bin file. NB: This isn't the fastest operation (we're talking SD cards) and dd doesn't display a progress bar. It will however give you a summary of the transfer when completed.

sudo umount /dev/sdc*
sudo dd if=/dev/sdc of=~/sd-card.bin bs=1M

Finally, put in a new SD card, verify the device name (again), then issue the same dd command, but in reverse. Note your SD cards must be the or larger than your image. Be patient!

sudo dd if=~/sd-card.bin of=/dev/sdc

Another hint I found is to go into /etc/fstab and add "sync" to allow immediate writes.

/dev/mmcblk0p2  /               ext4    defaults,[B]sync[/B],noatime  0       1

Finally, I picked up a "class 10" SDHC card to give a try (much faster writes).

On the web:
Copying an image to the SD card in Linux (command line)

Other option:
dcfldd
dd if=/dev/sdx | gzip > /path/to/image.gz
gzip -dc /path/to/image.gz | dd of=/dev/sdx

note to self

the good and nasty of nas


Here's the nasty – my Buffalo Linkstation crapped out after a couple of years of use. Hard to believe it, but a small, inconsequential, and incredibly cheap fan brought down a device costing a few hundred dollars. At any rate, it died and that's just not what backup should do.

Everyone should have a backup strategy that should involve what to back up, how to back it up and where to store it. For me, data is what needs to be backed up and that includes photos, music, documents and web. OS restores and application re-installations are done from original discs. While some things can be copied over to a USB flash drive, others, like mp3 files, have greater hardware requirements. How to back up can be as simple as printing out pictures or documents, or using something like Apple's Time Machine with an external hard drive. Most important is that that "the how" works, and works often. Finally, there is where to backup all that data. Trusting is to a single drive is waiting for failure. At a minimum, consider two backup places: 1) an external hard drive and 2) some sort of media, like DVD or CD. A third option that's becoming more popular is "cloud" storage. While this great works for documents or a zipped up website, it's not quite there yet for 100GB of music or movies. But using something like Shutterfly or Flickr, or Ubuntu One or Apple's Mobile Me, are increasing great backup solutions, and ones that are only going to get better in the coming years.

Now the good – a Network Attached Storage device provides a convenient way for several computers to access data storage. As a network device, the NAS is available to all the computers on my local subnet at all times. I can mount a share to my linux box at boot, which makes things like cron-ing a rsync job quite easy. Windows machines can simply map it as a network drive, while the Mac can use Bonjour and Finder to access it (note that TimeMachine is picky about network drives). A NAS also provides an easy solution for accessing a single printer across many machines as most (all?) have a USB port that shares a printer via SMB protocol.

In the meantime, I wanted to get my home solution sorted. A trip to Microcenter – armed with a printout from TigerDirect – landed me with a new Seagate NAS, the BlackArmor 110. To be honest, there was no way I was going to buy another Buffalo product, and at 1TB this doubled my capacity of the previous drive. The Seagate drive had two things going for it: inexpensive price and CIFS support. The drive also uses SMART status and carries a three (3) year warranty. And at $134 with price match, it was an auto-buy. Setup was wizard like from my PC, yet I did two things immediately: First, prior to installation, I added the NAS device's MAC address to my router so it would offer only a fixed IP address. Second, right after installation, I flashed the NAS's firmware to the latest version. The former helps with mapping printers, while the later is generally a good idea, especially before filling it up with data!

The Seagate NAS is intuitive to configure and use. Through its web interface, you can create shares, add users, manage security and set a host of options. With its gigabit interface, performance was much faster than I expected – so, yes, it was quite fast! I needed only slight modifications to my linux box's /etc/fstab and .smbcredentials to get the automount mounting, and that was just to adjust for the new share name and user. Windows machines immediately saw the drive in the network browser. The included software from Acronis is just okay; Microsoft's SyncToy is free and is a great tool for just backing up directories.

On the web:
Seagate BlackArmor NAS 110