root@Trooper:/home/allan#zpool status -v pool: media state: ONLINE scan: scrub repaired 0 in 2h3m with 0 errors on Thu Dec 13 00:20:18 2012 config: NAME STATE READ WRITE CKSUM media ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 ONLINE 0 0 0 errors: No known data errors root@Trooper:/home/allan#zpool list -v NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT media 10.9T 4.90T 5.98T 45% 1.00x ONLINE - raidz1 10.9T 4.90T 5.98T - ada0 - - - - ada1 - - - - ada2 - - - - ada3 - - - - root@Trooper:/home/allan#zfs list NAME USED AVAIL REFER MOUNTPOINT media 3.56T 4.22T 395K /media media/Apps 4.03G 4.22T 4.03G /media/Apps media/Games 107G 4.22T 107G /media/Games media/Movies 1.18T 4.22T 1.18T /media/Movies media/OS_Images 24.9G 4.22T 24.9G /media/OS_Images media/ROOT 2.64G 4.22T 386M legacy media/ROOT/9.0_router 11.6K 4.22T 386M legacy media/ROOT/9.1_freebsd 370M 4.22T 783M legacy media/ROOT/9.1_pcbsd 2.28G 4.22T 1.86G legacy media/TV 1.89T 4.22T 349K /media/TV media/TV/Comedy 242G 4.22T 94.2G /media/TV/Comedy media/TV/Comedy/Top_Gear 148G 4.22T 148G /media/TV/Comedy/Top_Gear media/TV/Counter_Terrorism 110G 4.22T 110G /media/TV/Counter_Terrorism media/TV/Crime 72.4G 4.22T 72.4G /media/TV/Crime media/TV/Documentaries 124G 4.22T 124G /media/TV/Documentaries media/TV/Drama 63.0G 4.22T 63.0G /media/TV/Drama media/TV/Educational 144G 4.22T 3.65G /media/TV/Educational media/TV/Educational/Holmes_on_Homes 33.3G 4.22T 33.3G /media/TV/Educational/Holmes_on_Homes media/TV/Educational/Modern_Marvels 45.4G 4.22T 45.4G /media/TV/Educational/Modern_Marvels media/TV/Educational/Myth_Busters 62.1G 4.22T 62.1G /media/TV/Educational/Myth_Busters media/TV/Historic 189G 4.22T 189G /media/TV/Historic media/TV/Legal 61.4G 4.22T 61.4G /media/TV/Legal media/TV/Medical 251G 4.22T 251G /media/TV/Medical media/TV/Military 175G 4.22T 175G /media/TV/Military media/TV/Police 256G 4.22T 256G /media/TV/Police media/TV/SciFi 251G 4.22T 251G /media/TV/SciFi media/backups 243G 4.22T 243G /media/backups media/oldserver 94.3G 4.22T 94.3G /media/oldserver media/oldsys 5.93G 4.22T 5.93G /media/oldsys media/scaleengine 209K 4.22T 209K /media/scaleengine media/system 14.6G 4.22T 209K / media/system/tmp 721K 4.22T 407K /tmp media/system/usr 13.1G 4.22T 371M /usr media/system/usr/home 1.81G 4.22T 257M /usr/home media/system/usr/home/allan.jude 1.43G 4.22T 174M /usr/home/allan.jude media/system/usr/home/allan.jude/docs 1.26G 4.22T 1.26G /usr/home/allan.jude/docs media/system/usr/jails 221K 4.22T 209K /usr/jails media/system/usr/local 8.40G 4.22T 6.45G /usr/local media/system/usr/local/pkg 1.53G 4.22T 1.53G /usr/local/pkg media/system/usr/obj 326K 4.22T 209K /usr/obj media/system/usr/ports 1.21G 4.22T 1.21G /usr/ports media/system/usr/ports/distfiles 1.94M 4.22T 1.81M /usr/ports/distfiles media/system/usr/ports/packages 209K 4.22T 209K /usr/ports/packages media/system/usr/src 997M 4.22T 636M /usr/src media/system/var 1.52G 4.22T 146M /var media/system/var/crash 355K 4.22T 215K /var/crash media/system/var/db 1.36G 4.22T 1.27G /var/db media/system/var/db/pkg 83.6M 4.22T 80.8M /var/db/pkg media/system/var/empty 209K 4.22T 209K /var/empty media/system/var/log 3.27M 4.22T 2.55M /var/log media/system/var/mail 1.77M 4.22T 1.48M /var/mail media/system/var/run 1.13M 4.22T 715K /var/run media/system/var/tmp 7.82M 4.22T 7.56M /var/tmp media/test 326K 4.22T 209K /media/test root@Trooper:/home/allan#gpart create -s gpt ada5 ada5 created root@Trooper:/home/allan#gpart add -t freebsd-boot -l bootfs4 -s 512K -a 4k ada5 ada5p1 added root@Trooper:/home/allan#gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada5 bootcode written to ada5 root@Trooper:/home/allan#gpart add -t freebsd-swap -l swap4 -s 2G -a 1m ada5 ada5p2 added root@Trooper:/home/allan#gpart add -t freebsd-zfs -l data4 -a 1m ada5 ada5p3 added root@Trooper:/home/allan#gpart show -l => 34 5860533101 ada5 GPT (2.7T) 34 6 - free - (3.0k) 40 1024 1 bootfs4 (512k) 1064 984 - free - (492k) 2048 4194304 2 swap4 (2.0G) 4196352 5856335872 3 data4 (2.7T) 5860532224 911 - free - (455k) root@Trooper:/home/allan#gpart backup ada5 | gpart restore ada6 root@Trooper:/home/allan#gpart show -l => 34 5860533101 ada5 GPT (2.7T) 34 6 - free - (3.0k) 40 1024 1 bootfs4 (512k) 1064 984 - free - (492k) 2048 4194304 2 swap4 (2.0G) 4196352 5856335872 3 data4 (2.7T) 5860532224 911 - free - (455k) => 34 5860533101 ada6 GPT (2.7T) 34 6 - free - (3.0k) 40 1024 1 (null) (512k) 1064 984 - free - (492k) 2048 4194304 2 (null) (2.0G) 4196352 5856335872 3 (null) (2.7T) 5860532224 911 - free - (455k) root@Trooper:/home/allan#gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada6 bootcode written to ada6 root@Trooper:/home/allan#gpart modify -l bootfs5 -i 1 ada6 ada6p1 modified root@Trooper:/home/allan#gpart modify -l swap5 -i 2 ada6 ada6p2 modified root@Trooper:/home/allan#gpart modify -l data5 -i 3 ada6 ada6p3 modified root@Trooper:/home/allan#gpart show -l => 34 5860533101 ada5 GPT (2.7T) 34 6 - free - (3.0k) 40 1024 1 bootfs4 (512k) 1064 984 - free - (492k) 2048 4194304 2 swap4 (2.0G) 4196352 5856335872 3 data4 (2.7T) 5860532224 911 - free - (455k) => 34 5860533101 ada6 GPT (2.7T) 34 6 - free - (3.0k) 40 1024 1 bootfs5 (512k) 1064 984 - free - (492k) 2048 4194304 2 swap5 (2.0G) 4196352 5856335872 3 data5 (2.7T) 5860532224 911 - free - (455k) root@Trooper:/home/allan#gnop create -S 4096 ada5p3 ada6p3 root@Trooper:/home/allan#zpool create tempa ada5p3.nop root@Trooper:/home/allan#zpool create tempb ada6p3.nop root@Trooper:/home/allan#zpool status -v pool: media state: ONLINE scan: scrub repaired 0 in 2h3m with 0 errors on Thu Dec 13 00:20:18 2012 config: NAME STATE READ WRITE CKSUM media ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 ONLINE 0 0 0 ada4 ONLINE 0 0 0 errors: No known data errors pool: tempa state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tempa ONLINE 0 0 0 ada5p3.nop ONLINE 0 0 0 errors: No known data errors pool: tempb state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tempb ONLINE 0 0 0 ada6p3.nop ONLINE 0 0 0 errors: No known data errors root@Trooper:/home/allan#zfs snapshot -r media@bigmove root@Trooper:/home/allan#zfs send -nPv -R media/TV@bigmove full media/TV@bigmove 23040 full media/TV/Crime@bigmove 77826170464 full media/TV/Drama@bigmove 67730446952 full media/TV/Military@bigmove 188202709392 full media/TV/Legal@bigmove 65998449056 full media/TV/Police@bigmove 275160323512 full media/TV/Counter_Terrorism@bigmove 117877130344 full media/TV/Educational@bigmove 3927335520 full media/TV/Educational/Myth_Busters@bigmove 66774109216 full media/TV/Educational/Modern_Marvels@bigmove 48783737008 full media/TV/Educational/Holmes_on_Homes@bigmove 35796701744 full media/TV/Medical@bigmove 269531975728 full media/TV/Historic@bigmove 202818774616 full media/TV/SciFi@bigmove 270169447696 full media/TV/Documentaries@bigmove 133407609904 full media/TV/Comedy@bigmove 101219004952 full media/TV/Comedy/Top_Gear@bigmove 158905423288 size 2084129372432 root@Trooper:/home/allan#zfs send -R media/TV@bigmove | dd obs=1m | dd bs=1m | bar -s 2084129372432 -bl 1m -bs 512m | zfs receive -F tempa/TV root@Trooper:/home/allan#zfs send -R media/Games@bigmove | dd obs=1m | dd bs=1m | bar -bl 1m -bs 512m | zfs receive -F tempb/Games etc... root@Trooper:/home/allan#gpart list ada1 Geom name: ada1 modified: false state: OK fwheads: 16 fwsectors: 63 last: 5860533134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada1p1 Mediasize: 524288 (512k) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r0w0e0 rawuuid: 992a6594-f716-11e2-b299-001b211a22a8 rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f label: bootfs0 length: 524288 offset: 20480 type: freebsd-boot index: 1 end: 1063 start: 40 2. Name: ada1p2 Mediasize: 2147483648 (2.0G) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r0w0e0 rawuuid: 995113ed-f716-11e2-b299-001b211a22a8 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: swap0 length: 2147483648 offset: 1048576 type: freebsd-swap index: 2 end: 4196351 start: 2048 3. Name: ada1p3 Mediasize: 2998443966464 (2.7T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e1 rawuuid: 9961e6e5-f716-11e2-b299-001b211a22a8 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: data0 length: 2998443966464 offset: 2148532224 type: freebsd-zfs index: 3 end: 5860532223 start: 4196352 Consumers: 1. Name: ada1 Mediasize: 3000592982016 (2.7T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e2 root@Trooper:/home/allan#truncate -s 2998443966464 /tmp/fakedisk1 root@Trooper:/home/allan#truncate -s 2998443966464 /tmp/fakedisk2 root@Trooper:/home/allan#mdconfig -a -t vnode -S 4096 -f /tmp/fakedisk1 md0 root@Trooper:/home/allan#mdconfig -a -t vnode -S 4096 -f /tmp/fakedisk2 md1 root@Trooper:/home/allan#zpool create media raidz2 ada1p3.nop ada2p3.nop ada3p3.nop ada4p3.nop md0 md1 root@Trooper:/home/allan#zpool status pool: media state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM media ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 ada1p3.nop ONLINE 0 0 0 ada2p3.nop ONLINE 0 0 0 ada3p3.nop ONLINE 0 0 0 ada4p3.nop ONLINE 0 0 0 md0 ONLINE 0 0 0 md1 ONLINE 0 0 0 root@Trooper:/home/allan#zpool offline media md0 root@Trooper:/home/allan#zpool offline media md1 root@Trooper:/home/allan#mdconfig -d -u md0 root@Trooper:/home/allan#mdconfig -d -u md1 root@Trooper:/home/allan#zpool status pool: media state: DEGRADED status: One or more devices has been taken offline by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scan: none requested config: NAME STATE READ WRITE CKSUM media DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 ada1p3.nop ONLINE 0 0 0 ada2p3.nop ONLINE 0 0 0 ada3p3.nop ONLINE 0 0 0 ada4p3.nop ONLINE 0 0 0 6051069297020915093 OFFLINE 0 0 0 was /dev/md0 1640323803909328146 OFFLINE 0 0 0 was /dev/md1 ... zfs send files back ... root@Trooper:/home/allan#zpool destroy tempa root@Trooper:/home/allan#zpool destroy tempb root@Trooper:/home/allan#zpool replace media 6051069297020915093 /dev/ada5p3 Make sure to wait until resilver is done before rebooting. If you boot from pool 'media', you may need to update boot code on newly attached disk '/dev/ada5p3'. Assuming you use GPT partitioning and 'da0' is your new boot disk you may use the following command: gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0 root@Trooper:/home/allan#zpool status pool: media state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Sun Jul 28 00:45:13 2013 138G scanned out of 5.35T at 520M/s, 2h55m to go 45.8G resilvered, 2.51% done config: NAME STATE READ WRITE CKSUM media DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 ada1p3 ONLINE 0 0 0 ada2p3 ONLINE 0 0 0 ada3p3 ONLINE 0 0 0 ada4p3 ONLINE 0 0 0 replacing-4 OFFLINE 0 0 0 6051069297020915093 OFFLINE 0 0 0 was /dev/md0 ada5p3 ONLINE 0 0 0 (resilvering) replacing-5 OFFLINE 0 0 0 1640323803909328146 OFFLINE 0 0 0 was /dev/md1 ada6p3 ONLINE 0 0 0 (resilvering) errors: No known data errors root@Trooper:/media#zpool status pool: media state: ONLINE scan: resilvered 1.78T in 3h2m with 0 errors on Sun Jul 28 03:47:40 2013 config: NAME STATE READ WRITE CKSUM media ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 ada1p3 ONLINE 0 0 0 ada2p3 ONLINE 0 0 0 ada3p3 ONLINE 0 0 0 ada4p3 ONLINE 0 0 0 ada5p3 ONLINE 0 0 0 ada6p3 ONLINE 0 0 0 errors: No known data errors