hello friends! new(ish)!

Backups: Difference between revisions

From InstallGentoo Wiki v2
Jump to navigation Jump to search
>Mrsnooze
(Created page with "There are two types of data: Backed up data, and data you're prepared to lose. The process of backing up is dead simple. Deciding what to backup and getting into the habit of...")
 
>GNUadeen
m (Hacking is playful cleverness.)
 
(30 intermediate revisions by 10 users not shown)
Line 1: Line 1:
There are two types of data: Backed up data, and data you're prepared to lose.
There are two types of data: Backed up data, and data you're prepared to lose. The process of backing up is dead simple. Deciding what to backup and getting into the habit of backing up is the trickier part.


The process of backing up is dead simple. Deciding what to backup and getting into the habit of backing up is the trickier part.
== Threats to your data ==
Keep these threats in mind when planning a backup strategy:
* Bitrot and file corruption
* Burglary and theft
* Fires
* Hard drive failure
* Software failure
* Human error
** Accidental deletion or modification
** Misplacement of data storage
* Intentional sabotage (by someone you may know personally)
* [[Malware]], [[Malware#Viruses | viruses]], [[Malware#Viruses | ransomware]], etc.
* Natural disasters
** Floods
** Tornadoes and storms
** Coronal mass ejection
* Power irregularities
** Dips and brownouts
** Lightning strikes
** Surges and spikes


== What to Backup ==
A reasonable defense against all of these requires a minimum of three independent copies of the data. One or more of those copies must be powered down (offline) and at a different physical location (offsite). Remember, [https://en.wikipedia.org/wiki/RAID RAID] is no substitute for a backup. Using a modern file system like BTRFS or ZFS with ECC RAM can help protect against bitrot. Surge protectors and UPS are essential protection from power irregularities. However, no protection system can guarantee absolute safety from a lightning strike, so always unplug during thunderstorms. If using RAID, a controller with built in battery backup helps prevent write failures during unexpected power outages.
 
== What to backup ==


First, decide on a place to keep important data. A directory named "data" or "stuff", or a second hard drive. We'll call this our "big data" directory. Most of your important things can be placed in subdirectories within this, such as:
First, decide on a place to keep important data. A directory named "data" or "stuff", or a second hard drive. We'll call this our "big data" directory. Most of your important things can be placed in subdirectories within this, such as:
* Applications with addons, extensions and plugins
* [[Books |Books/Audiobooks]]
* Documents
** Email
** School files
** Work files
* Drivers
* [[Games]]
* [[Music]]
* [[Movies|Movies and TV]]
* Offline code repositories
* [[:Category:Operating systems | Operating systems]] (see [[Backups#Install kits | install kits]], below)
* [[Security#Password_manager | Password databases]]
* Photos
* Photos
* Documents
* [[VirtualBox |Virtual machines]]
* Music
* Movies
* TV
* Books/Audiobooks
* Applications
* Operating Systems (see Install Kits below)
* Chinese Cartoons
* Games
* Porn
* School files
* Work files
* Password databases
* Wallpapers
* Wallpapers
* Offline code repositories
* Android roms


You may have your files in an unsorted mess, such as a catch-all "Downloads" directory. No problem! Make another subdirectory under big data:
You may have your files in an unsorted mess, such as a catch-all "Downloads" directory. No problem! Make another subdirectory under big data:
Line 28: Line 48:


Some files can't be moved to your big data directory because they either live in a specific place, or don't exist until you manually create them:
Some files can't be moved to your big data directory because they either live in a specific place, or don't exist until you manually create them:
* Application and Operating System configurations
* Application and operating system configurations
* Browser bookmarks
**Adblock custom filters
**NoScript whitelist
**RequestPolicy whitelist
**Saved [[Passwords | passwords]] and logins
* Calendars
* Chat logs
* Contacts
* [[Databases | Database]]/FTP/RDP/SSH/TeamViewer server bookmarks
* Offline email
* Offline email
* Video game save files
* Things in "My Documents" and "AppData"
* Things in "My Documents"
* [[Games | Video game]] save files
* VOIP/Teamspeak server bookmarks and identity keys


To have a copy of these things in your big data directory, you may have to manually "export" or "backup" from the applications they're used with. Some of this you will be able to automate with a script. Don't forget to keep the script in your big data directory too!
To have a copy of these things in your big data directory, you may have to manually "export" or "backup" from the applications they're used with. Alternatively you may need to backup a copy of the corresponding AppData or .config folder. Some of this you will be able to automate with a script. Don't forget to keep the script in your big data directory too!


You may want to include a datestamp in the filesnames of these things to keep multiple copies. If your password database becomes corrupt you'll want to restore an earlier version, rather than a perfect backup of the same corruption!
You may want to include a datestamp in the file names of these things to keep multiple copies. If your password database becomes corrupt you'll want to restore an earlier version, rather than a perfect backup of the same corruption!


=== Install Kits ===
=== Install kits ===
You should aim to be able to go from an unformatted drive to a complete working system, with all applications fully installed and configured, WITHOUT network access. If you need network access to download or activate anything then you don't really own it and don't have it backed up.
You should aim to be able to go from an unformatted drive to a complete working system, with all applications fully installed and configured, WITHOUT [[:Category:Networking | network]] access. If you need network access to download or activate anything then you don't really own it and don't have it backed up.


To do this, you will need copies of a few things:
To do this, you will need copies of a few things:
* Your Operating System(s) isos
* Your [[:Category:Operating systems | Operating System(s)]] isos
** Tools to put the iso onto a thumbdrive such as [http://rufus.akeo.ie/ Rufus]
** Tools to put the iso onto a thumbdrive such as [http://rufus.akeo.ie/ Rufus]
* Drivers for all of your hardware
* Drivers for all of your hardware
** Wired and Wireless NICs
** Wired and Wireless NICs
** GPUs
** [[GPU]]s
** USB3
** USB3
** Soundcard
** Soundcard
Line 51: Line 81:
** Printers
** Printers
* Firmware for your motherboard/peripherals/devices
* Firmware for your motherboard/peripherals/devices
* Applications you use
* [[Recommended software | Applications]] you use
** Configuration files for them, including window managers and ricing configs
** Configuration files for them, including window managers and [[Ricing | ricing]] configs
** Plugins (Web Browsers, etc)
** Plugins ([[Web Browsers]], etc)
*** Configuration files for the plugins (adblocklists etc)
*** Configuration files for the plugins (adblocklists etc)
* Windows Specific stuff
* [[Windows]] Specific stuff
** Activation tools such as DAZ Loader.
** Activation tools such as DAZ Loader and/or [https://github.com/massgravel/Microsoft-Activation-Scripts Microsoft-Activation-Scripts]
** dotNet
** dotNet
** DirectX
** DirectX
** msxml
** msxml
** vcredist 2005, 2008, 2010, 2012, 2013
** vcredist 2005, 2008, 2010, 2012, 2013, 2015
** xnafx
** xnafx
** Games for Windows Live
** [[Video Games |Games]] for Windows Live
** Java
** Java
** Flash


Make sure all of the installers you have are full installers and not web setups. These can be tricky to find, like the [https://www.adobe.com/products/flashplayer/distribution3.html Offline Flash Player Installer].
Make sure all of the installers you have are full installers and not web setups.


Alongside your main backup, have a USB thumbdrive with your operating system installer on it. It's handy to have all of your drivers, applications and configuration on this thumbdrive too.
Alongside your main backup, have a USB thumbdrive with your operating system installer on it. It's handy to have all of your drivers, applications and configuration on this thumbdrive too.
Line 73: Line 102:


=== Android ===
=== Android ===
If you have a custom bootloader applied such as [https://www.clockworkmod.com/ ClockworkMod], you can likely create a full backup of your device including data and configuration to an external storage card.
If you have a custom bootloader applied such as [https://www.clockworkmod.com/ ClockworkMod], or [https://twrp.me/ TWRP], you can likely create a full backup of your device including data and configuration to an external storage card. You can also use free-libre apps like [https://f-droid.org/en/packages/com.machiav3lli.backup/ OAndBackupX] (requires root) found on f-droid to backup apps and data.


Once complete, copy the backup from your storage card to your "big data" directory.
Once complete, copy the backup from your storage card to your "big data" directory.


== How to Backup ==
== How to backup ==
With all your data in a central location, you're ready to back it up.
With all your data in a central location, you're ready to back it up.


Get yourself an external hard drive of sufficient capacity for you data. Encrypt it if you want.
Depending on your needs, investing in a file server, NAS or SAN may be worthwhile. A simple option is to get yourself an external hard drive of sufficient capacity for you data. [[Encryption | Encrypt]] it if you want.


Once it's mounted:
We use [https://rsync.samba.org/ rsync] to backup data.
* $ rsync -av --delete /path/to/mydata/ /path/to/externaldrive/
 
** -v being verbose
Once the external drive is mounted:
** --delete [optional] delete files on your /externaldrive/ which no longer exist in /mydata/
  $ rsync -av --delete /path/to/mydata/ /path/to/externaldrive/
** -a being "archive mode, which includes:
* -v being verbose
*** -r recursive mode
* --delete [optional] delete files on your /externaldrive/ which no longer exist in /mydata/
*** -p preserve permissions
* -a being "archive mode, which includes:
*** -t preserve times
** -r recursive mode
*** -g preserve groups
** -p preserve permissions
*** -o preserve owner
** -t preserve times
*** -l copy symlinks as symlinks
** -g preserve groups
** -o preserve owner
** -l copy symlinks as symlinks
 
Note: if you are backing up your full system, use the --exclude on your backup folder directory so that the process doesn't loop. --dry-run is also helpful.


rsync will update any modified files and create any new ones.
rsync will update any modified files and create any new ones.
There are many other software options which work in a similar way to rsync. See [https://www.cis.upenn.edu/~bcpierce/unison/ Unison (all platforms)] or [https://www.microsoft.com/en-us/download/details.aspx?id=15155 SyncToy (Windows)]. There is also a GUI front end for rsync for newbs: [https://www.opbyte.it/grsync/ Grsync].


Keep the external drive unplugged and stored away when not in use. You don't want a power surge taking out your computer and your backup in one hit.
Keep the external drive unplugged and stored away when not in use. You don't want a power surge taking out your computer and your backup in one hit.


=== Cloud Storage ===
=== Cloud storage ===
Cloud storage is a bad idea because if you:
Cloud storage is a bad idea because if you:
* don't have network access
* don't have network access
Line 104: Line 139:
* have a data limited connection which costs extra for excessive data
* have a data limited connection which costs extra for excessive data
* lose your password
* lose your password
* get phished/hacked
* get phished/cracked
* get banned from the service
* get banned from the service
then you don't have a backup and have lost your data.
then you don't have a backup and have lost your data.


Other considerations:
Other considerations:
* Unencryped backups can be read the cloud provider
* Unencrypted backups can be read by the cloud provider
* Cloud provider could go offline/shutdown without warning
* Cloud provider could go offline/shutdown without warning
* Cloud provider may be in a different legal jurisdiction to you
* Cloud provider may be in a different legal jurisdiction to you
* Cloud provider can raise the cost of subscription at any time
* Data could be leaked by cloud providers in a hack and be read if unencrypted


== When to Backup ==
== When to backup ==
This depends on how often your data changes. Once a week is a good. Once a month at least.
This depends on how often your data changes. Once a week is a good. Once a month at least.


You may want to run an ad hoc backup when something important changes such as encryption keys or password databases.
You may want to run an ad-hoc backup when something important changes such as encryption keys or password databases.
 
== Testing the backup ==
Most of your backup (except drivers) can be tested in an isolated [[VirtualBox | Virtual Machine]] to see if you've missed anything or if some of your install files require internet access.
 
If you're fairly confident in your backup, [[Data Destruction | wipe]] your computer and test it. It's better to find out today that something was lost than finding out when it really matters. If this seems too scary, recheck your backed up data until it gives you confidence.
 
==External Links==
[https://wiki.archiveteam.org/index.php/Backup_Tips Archive Team] - Backup Tips.


== Testing the Backup ==
== See also ==
[[Home server]]


Most of your backup (except drivers) can be tested in an isolated virtual machine to see if you've missed anything or if some of your install files require internet access.
[[Storage devices]]


If you're fairly confident in your backup, wipe your computer and test it. It's better to find out today that something was lost than finding out when it really matters. If this seems too scary, recheck your backed up data until it gives you confidence.
[[Category:HowTo]]
[[Category:Tutorials]]

Latest revision as of 07:04, 9 May 2022

There are two types of data: Backed up data, and data you're prepared to lose. The process of backing up is dead simple. Deciding what to backup and getting into the habit of backing up is the trickier part.

Threats to your data

Keep these threats in mind when planning a backup strategy:

  • Bitrot and file corruption
  • Burglary and theft
  • Fires
  • Hard drive failure
  • Software failure
  • Human error
    • Accidental deletion or modification
    • Misplacement of data storage
  • Intentional sabotage (by someone you may know personally)
  • Malware, viruses, ransomware, etc.
  • Natural disasters
    • Floods
    • Tornadoes and storms
    • Coronal mass ejection
  • Power irregularities
    • Dips and brownouts
    • Lightning strikes
    • Surges and spikes

A reasonable defense against all of these requires a minimum of three independent copies of the data. One or more of those copies must be powered down (offline) and at a different physical location (offsite). Remember, RAID is no substitute for a backup. Using a modern file system like BTRFS or ZFS with ECC RAM can help protect against bitrot. Surge protectors and UPS are essential protection from power irregularities. However, no protection system can guarantee absolute safety from a lightning strike, so always unplug during thunderstorms. If using RAID, a controller with built in battery backup helps prevent write failures during unexpected power outages.

What to backup

First, decide on a place to keep important data. A directory named "data" or "stuff", or a second hard drive. We'll call this our "big data" directory. Most of your important things can be placed in subdirectories within this, such as:

You may have your files in an unsorted mess, such as a catch-all "Downloads" directory. No problem! Make another subdirectory under big data:

  • Unsorted

Some files can't be moved to your big data directory because they either live in a specific place, or don't exist until you manually create them:

  • Application and operating system configurations
  • Browser bookmarks
    • Adblock custom filters
    • NoScript whitelist
    • RequestPolicy whitelist
    • Saved passwords and logins
  • Calendars
  • Chat logs
  • Contacts
  • Database/FTP/RDP/SSH/TeamViewer server bookmarks
  • Offline email
  • Things in "My Documents" and "AppData"
  • Video game save files
  • VOIP/Teamspeak server bookmarks and identity keys

To have a copy of these things in your big data directory, you may have to manually "export" or "backup" from the applications they're used with. Alternatively you may need to backup a copy of the corresponding AppData or .config folder. Some of this you will be able to automate with a script. Don't forget to keep the script in your big data directory too!

You may want to include a datestamp in the file names of these things to keep multiple copies. If your password database becomes corrupt you'll want to restore an earlier version, rather than a perfect backup of the same corruption!

Install kits

You should aim to be able to go from an unformatted drive to a complete working system, with all applications fully installed and configured, WITHOUT network access. If you need network access to download or activate anything then you don't really own it and don't have it backed up.

To do this, you will need copies of a few things:

  • Your Operating System(s) isos
    • Tools to put the iso onto a thumbdrive such as Rufus
  • Drivers for all of your hardware
    • Wired and Wireless NICs
    • GPUs
    • USB3
    • Soundcard
    • Bluetooth
    • Printers
  • Firmware for your motherboard/peripherals/devices
  • Applications you use
    • Configuration files for them, including window managers and ricing configs
    • Plugins (Web Browsers, etc)
      • Configuration files for the plugins (adblocklists etc)
  • Windows Specific stuff
    • Activation tools such as DAZ Loader and/or Microsoft-Activation-Scripts
    • dotNet
    • DirectX
    • msxml
    • vcredist 2005, 2008, 2010, 2012, 2013, 2015
    • xnafx
    • Games for Windows Live
    • Java

Make sure all of the installers you have are full installers and not web setups.

Alongside your main backup, have a USB thumbdrive with your operating system installer on it. It's handy to have all of your drivers, applications and configuration on this thumbdrive too.

Once a month or so, check on these files to see if new versions are available. Perhaps when you run your regular backup?

Android

If you have a custom bootloader applied such as ClockworkMod, or TWRP, you can likely create a full backup of your device including data and configuration to an external storage card. You can also use free-libre apps like OAndBackupX (requires root) found on f-droid to backup apps and data.

Once complete, copy the backup from your storage card to your "big data" directory.

How to backup

With all your data in a central location, you're ready to back it up.

Depending on your needs, investing in a file server, NAS or SAN may be worthwhile. A simple option is to get yourself an external hard drive of sufficient capacity for you data. Encrypt it if you want.

We use rsync to backup data.

Once the external drive is mounted:

 $ rsync -av --delete /path/to/mydata/ /path/to/externaldrive/
  • -v being verbose
  • --delete [optional] delete files on your /externaldrive/ which no longer exist in /mydata/
  • -a being "archive mode, which includes:
    • -r recursive mode
    • -p preserve permissions
    • -t preserve times
    • -g preserve groups
    • -o preserve owner
    • -l copy symlinks as symlinks

Note: if you are backing up your full system, use the --exclude on your backup folder directory so that the process doesn't loop. --dry-run is also helpful.

rsync will update any modified files and create any new ones.

There are many other software options which work in a similar way to rsync. See Unison (all platforms) or SyncToy (Windows). There is also a GUI front end for rsync for newbs: Grsync.

Keep the external drive unplugged and stored away when not in use. You don't want a power surge taking out your computer and your backup in one hit.

Cloud storage

Cloud storage is a bad idea because if you:

  • don't have network access
  • have a slow connection
  • have a data limited connection which costs extra for excessive data
  • lose your password
  • get phished/cracked
  • get banned from the service

then you don't have a backup and have lost your data.

Other considerations:

  • Unencrypted backups can be read by the cloud provider
  • Cloud provider could go offline/shutdown without warning
  • Cloud provider may be in a different legal jurisdiction to you
  • Cloud provider can raise the cost of subscription at any time
  • Data could be leaked by cloud providers in a hack and be read if unencrypted

When to backup

This depends on how often your data changes. Once a week is a good. Once a month at least.

You may want to run an ad-hoc backup when something important changes such as encryption keys or password databases.

Testing the backup

Most of your backup (except drivers) can be tested in an isolated Virtual Machine to see if you've missed anything or if some of your install files require internet access.

If you're fairly confident in your backup, wipe your computer and test it. It's better to find out today that something was lost than finding out when it really matters. If this seems too scary, recheck your backed up data until it gives you confidence.

External Links

Archive Team - Backup Tips.

See also

Home server

Storage devices