I back up my volumes daily, using incremental backups to minimize
network traffic and storage consumption. I want to periodically remove
old backups, and during this pruning operation, avoid entering a state
where a volume has no recent backups. Ceph RBD appears to support this
workflow, but unfortunately, Cinder does not. I can only delete the
*latest* backup of a given volume, and this precludes any reasonable
way to prune backups. Here, I'll show you.
Let's make three backups of the same volume:
```
openstack volume backup create --name backup-1 --force volume-foo
openstack volume backup create --name backup-2 --force volume-foo
openstack volume backup create --name backup-3 --force volume-foo
```
Cinder reports the following via `volume backup show`:
- backup-1 is not an incremental backup, but backup-2 and backup-3 are
(`is_incremental`).
- All but the latest backup have dependent backups (`has_dependent_backups`).
We take a backup every day, and after a week we're on backup-7. We
want to start deleting older backups so that we don't keep
accumulating backups forever! What happens when we try?
```
# openstack volume backup delete backup-1
Failed to delete backup with name or ID 'backup-1': Invalid backup:
Incremental backups exist for this backup. (HTTP 400)
```
We can't delete backup-1 because Cinder considers it a "base" backup
which `has_dependent_backups`. What about backup-2? Same story. Adding
the `--force` flag just gives a slightly different error message. The
*only* backup that Cinder will delete is backup-7 -- the very latest
one. This means that if we want to remove the oldest backups of a
volume, *we must first remove all newer backups of the same volume*,
i.e. delete literally all of our backups.
Also, we cannot force creation of another *full* (non-incrmental)
backup in order to free all of the earlier backups for removal.
(Omitting the `--incremental` flag has no effect; you still get an
incremental backup.)
Can we hope for better? Let's reach behind Cinder to the Ceph backend.
Volume backups are represented as a "base" RBD image with a snapshot
for each incremental backup:
```
# rbd snap ls volume-e742c4e2-e331-4297-a7df-c25e729fdd83.backup.base
SNAPID NAME
SIZE TIMESTAMP
577 backup.e3c1bcff-c1a4-450f-a2a5-a5061c8e3733.snap.1535046973.43
10240 MB Thu Aug 23 10:57:48 2018
578 backup.93fbd83b-f34d-45bc-a378-18268c8c0a25.snap.1535047520.44
10240 MB Thu Aug 23 11:05:43 2018
579 backup.b6bed35a-45e7-4df1-bc09-257aa01efe9b.snap.1535047564.46
10240 MB Thu Aug 23 11:06:47 2018
580 backup.10128aba-0e18-40f1-acfb-11d7bb6cb487.snap.1535048513.71
10240 MB Thu Aug 23 11:22:23 2018
581 backup.8cd035b9-63bf-4920-a8ec-c07ba370fb94.snap.1535048538.72
10240 MB Thu Aug 23 11:22:47 2018
582 backup.cb7b6920-a79e-408e-b84f-5269d80235b2.snap.1535048559.82
10240 MB Thu Aug 23 11:23:04 2018
583 backup.a7871768-1863-435f-be9d-b50af47c905a.snap.1535048588.26
10240 MB Thu Aug 23 11:23:31 2018
584 backup.b18522e4-d237-4ee5-8786-78eac3d590de.snap.1535052729.52
10240 MB Thu Aug 23 12:32:43 2018
```
It seems that each snapshot stands alone and doesn't depend on others.
Ceph lets me delete the older snapshots.
```
# rbd snap rm volume-e742c4e2-e331-4297-a7df-c25e729fdd83.backup.base@backup.e3c1bcff-c1a4-450f-a2a5-a5061c8e3733.snap.1535046973.43
Removing snap: 100% complete...done.
# rbd snap rm volume-e742c4e2-e331-4297-a7df-c25e729fdd83.backup.base@backup.10128aba-0e18-40f1-acfb-11d7bb6cb487.snap.1535048513.71
Removing snap: 100% complete...done.
```
Now that we nuked backup-1 and backup-4, can we still restore from
backup-7 and launch an instance with it?
```
openstack volume create --size 10 --bootable volume-foo-restored
openstack volume backup restore backup-7 volume-foo-restored
openstack server create --volume volume-foo-restored --flavor medium1
instance-restored-from-backup-7
```
Yes! We can SSH to the instance and it appears intact.
Perhaps each snapshot in Ceph stores a complete diff from the base RBD
image (rather than each successive snapshot depending on the last). If
this is true, then Cinder is unnecessarily protective of older
backups. Cinder represents these as "with dependents" and doesn't let
us touch them, even though Ceph will let us delete older RBD
snapshots, apparently without disrupting newer snapshots of the same
volume. If we could remove this limitation, Cinder backups would be
significantly more useful for us. We mostly host servers with
non-cloud-native workloads (IaaS for research scientists). For these,
full-disk backups at the infrastructure level are an important
supplement to file-level or application-level backups.
It would be great if someone else could confirm or disprove what I'm
seeing here. I'd also love to hear from anyone else using Cinder
backups this way.
Regards,
Chris Martin at CyVerse
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
network traffic and storage consumption. I want to periodically remove
old backups, and during this pruning operation, avoid entering a state
where a volume has no recent backups. Ceph RBD appears to support this
workflow, but unfortunately, Cinder does not. I can only delete the
*latest* backup of a given volume, and this precludes any reasonable
way to prune backups. Here, I'll show you.
Let's make three backups of the same volume:
```
openstack volume backup create --name backup-1 --force volume-foo
openstack volume backup create --name backup-2 --force volume-foo
openstack volume backup create --name backup-3 --force volume-foo
```
Cinder reports the following via `volume backup show`:
- backup-1 is not an incremental backup, but backup-2 and backup-3 are
(`is_incremental`).
- All but the latest backup have dependent backups (`has_dependent_backups`).
We take a backup every day, and after a week we're on backup-7. We
want to start deleting older backups so that we don't keep
accumulating backups forever! What happens when we try?
```
# openstack volume backup delete backup-1
Failed to delete backup with name or ID 'backup-1': Invalid backup:
Incremental backups exist for this backup. (HTTP 400)
```
We can't delete backup-1 because Cinder considers it a "base" backup
which `has_dependent_backups`. What about backup-2? Same story. Adding
the `--force` flag just gives a slightly different error message. The
*only* backup that Cinder will delete is backup-7 -- the very latest
one. This means that if we want to remove the oldest backups of a
volume, *we must first remove all newer backups of the same volume*,
i.e. delete literally all of our backups.
Also, we cannot force creation of another *full* (non-incrmental)
backup in order to free all of the earlier backups for removal.
(Omitting the `--incremental` flag has no effect; you still get an
incremental backup.)
Can we hope for better? Let's reach behind Cinder to the Ceph backend.
Volume backups are represented as a "base" RBD image with a snapshot
for each incremental backup:
```
# rbd snap ls volume-e742c4e2-e331-4297-a7df-c25e729fdd83.backup.base
SNAPID NAME
SIZE TIMESTAMP
577 backup.e3c1bcff-c1a4-450f-a2a5-a5061c8e3733.snap.1535046973.43
10240 MB Thu Aug 23 10:57:48 2018
578 backup.93fbd83b-f34d-45bc-a378-18268c8c0a25.snap.1535047520.44
10240 MB Thu Aug 23 11:05:43 2018
579 backup.b6bed35a-45e7-4df1-bc09-257aa01efe9b.snap.1535047564.46
10240 MB Thu Aug 23 11:06:47 2018
580 backup.10128aba-0e18-40f1-acfb-11d7bb6cb487.snap.1535048513.71
10240 MB Thu Aug 23 11:22:23 2018
581 backup.8cd035b9-63bf-4920-a8ec-c07ba370fb94.snap.1535048538.72
10240 MB Thu Aug 23 11:22:47 2018
582 backup.cb7b6920-a79e-408e-b84f-5269d80235b2.snap.1535048559.82
10240 MB Thu Aug 23 11:23:04 2018
583 backup.a7871768-1863-435f-be9d-b50af47c905a.snap.1535048588.26
10240 MB Thu Aug 23 11:23:31 2018
584 backup.b18522e4-d237-4ee5-8786-78eac3d590de.snap.1535052729.52
10240 MB Thu Aug 23 12:32:43 2018
```
It seems that each snapshot stands alone and doesn't depend on others.
Ceph lets me delete the older snapshots.
```
# rbd snap rm volume-e742c4e2-e331-4297-a7df-c25e729fdd83.backup.base@backup.e3c1bcff-c1a4-450f-a2a5-a5061c8e3733.snap.1535046973.43
Removing snap: 100% complete...done.
# rbd snap rm volume-e742c4e2-e331-4297-a7df-c25e729fdd83.backup.base@backup.10128aba-0e18-40f1-acfb-11d7bb6cb487.snap.1535048513.71
Removing snap: 100% complete...done.
```
Now that we nuked backup-1 and backup-4, can we still restore from
backup-7 and launch an instance with it?
```
openstack volume create --size 10 --bootable volume-foo-restored
openstack volume backup restore backup-7 volume-foo-restored
openstack server create --volume volume-foo-restored --flavor medium1
instance-restored-from-backup-7
```
Yes! We can SSH to the instance and it appears intact.
Perhaps each snapshot in Ceph stores a complete diff from the base RBD
image (rather than each successive snapshot depending on the last). If
this is true, then Cinder is unnecessarily protective of older
backups. Cinder represents these as "with dependents" and doesn't let
us touch them, even though Ceph will let us delete older RBD
snapshots, apparently without disrupting newer snapshots of the same
volume. If we could remove this limitation, Cinder backups would be
significantly more useful for us. We mostly host servers with
non-cloud-native workloads (IaaS for research scientists). For these,
full-disk backups at the infrastructure level are an important
supplement to file-level or application-level backups.
It would be great if someone else could confirm or disprove what I'm
seeing here. I'd also love to hear from anyone else using Cinder
backups this way.
Regards,
Chris Martin at CyVerse
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack