cancel
Showing results for 
Search instead for 
Did you mean: 

MaxDB Backup de-duplication ability

lbreddemann
Active Contributor
0 Kudos

Hi all,

because thread is marked as answered I open this one, so that everybody can post their test results with the backup appliances that provide "de-duplication".

One question is also open for me: why not use incremental backups instead?

The impact to MTTR can only be a doubling of the restore time for the worst case scenario (that is: each and every single page in the database had been changed since the last complete data backup - not too likely really).

And the de-duplication server will have to recreate the complete file as well !

The advantages on the other hand are:

- MaxDB does the "de-duplication" by just saving changed pages (so that comes for free)

- less I/O impact to the system during the backup

- less data to send to the backup appliance (network)

- just one additional recovery step (instead of recover data + log, now data + pages + log need to get recovered)

So to me a incremental data backup from MaxDB seems to be the even better choice.

I mean you also won't select data from the database and sort it yourself, would you?

KR Lars

Accepted Solutions (0)

Answers (5)

Answers (5)

werner_thesing
Explorer
0 Kudos

Hi,

Data Domain as introduced the DD Compression Type (10) in DD OS v4.6. This solves the problem. It has been tested in different environments.

Please contact Data Domain in case of further questions.

Best regards,

Werner

werner_thesing
Explorer
0 Kudos

Hi,

some customers complained about bad de-duplication rates for MaxDB backups.

This is under investigations. We work together with datadomain.

I met a consultant from datadomain a few days ago. We tested backups of different MaxDB versions. The de-duplicaton rate is insufficient even if two backups have been created in series without changes to the database by the application in between.

MaxDB It doesn't read the data pages in the order of the converter during the backup. It does sequential reads from the data volumes and writes in 64 chunks (standard, backup template option) to the pipe/backup devices. It reads from all data volumes in parallel. The 8 KB blocks in the backups are at different positions, even if nearly 100% of the blocks are the same.

Obviously the de-duplicaton algorithm is not able to identify the 8 KB duplicates if the duplicates are at different file positions - theoretically it should be. This is under investigation by datadomain. We have to wait for an answer.

Best regards,

Werner Thesing

Former Member
0 Kudos

Hi Werner,

Small world. It is because of our engagement with Data Domain that you have ended up looking into this. We were about to start a proof-of-concept trial with Data Domain, when late last week/early this week that mentioned that they had been working with SAP, and were only seeing a maximum de-dup rate of 4:1.

They have informed me that they believe it will be possibly several months before they have a viable answer to this issue.

Cheers

Shaun

markus_doehr2
Active Contributor
0 Kudos

Test environment:

Linux X86_64

SuSE SLES 10 SP1

MaxDB 7.6.04.10

Database size: 40 GB

Backup Software: Legato Networker 7.4 SP1

Deduplication Software: Exagrid EX1000

Test run: Three full backups (data migration) in three days were done.

Result: deduplication factor: 0, compression is impressive: 6:1

Markus

lbreddemann
Active Contributor
0 Kudos

Hi Markus,

thanks for the data.

What kind of compression is done here? Do you know any details like dictionary/block size?

Which compression algorithm is used?

How much does the time for the backup change?

Did you already try out recovery?

Thanks and regards,

Lars

markus_doehr2
Active Contributor
0 Kudos

Hi Lars,

> What kind of compression is done here? Do you know any details like dictionary/block size?

> Which compression algorithm is used?

I must admit, I have no idea. The guy who set this up also didn't knew (was a partner of Exagrid).

> How much does the time for the backup change?

Almost same speed. We use the same physical network to transfer from the database to the backup server as from backup server to the NFS mounted directory.

> Did you already try out recovery?

yes

srvlibrh:/exbackup/SAPDBData # find . -type f | xargs ls -lh | cut -f 5,6,10 -d " "
  12:17
 40G ./04/46/a76bad41-00000006-7e31576b-4831576b-0f840000-c0a88501
 172 ./04/46/notes/a76bad41-00000006-7e31576b-4831576b-0f840000-c0a88501
 32K ./16/10/004535e0-00000006-7f315734-48315734-0f830000-c0a88501
 276 ./16/10/notes/004535e0-00000006-7f315734-48315734-0f830000-c0a88501
1.1K May
 216 ./19/15/notes/103186d3-00000006-7d315a7b-48315a7b-0f850000-c0a88501
 20G ./20/33/0b2ea481-00000006-2f323ebc-48323ebc-10d30000-c0a88501
 172 ./20/33/notes/0b2ea481-00000006-2f323ebc-48323ebc-10d30000-c0a88501
 34K ./20/94/eb00d2d0-00000006-7c315a7b-48315a7b-0f860000-c0a88501
 276 ./20/94/notes/eb00d2d0-00000006-7c315a7b-48315a7b-0f860000-c0a88501
 40G ./21/51/38e71b76-00000006-7b316f9b-48316f9b-0f870000-c0a88501
 172 ./21/51/notes/38e71b76-00000006-7b316f9b-48316f9b-0f870000-c0a88501
1.1K May
 216 ./22/07/notes/2988b432-00000006-80315734-48315734-0f820000-c0a88501
 20G ./22/90/80770a6d-00000006-ce339057-48339057-12340000-c0a88501
 172 ./22/90/notes/80770a6d-00000006-ce339057-48339057-12340000-c0a88501
 31K ./28/08/b297541d-00000006-8131571d-4831571d-0f810000-c0a88501
 276 ./28/08/notes/b297541d-00000006-8131571d-4831571d-0f810000-c0a88501
 20G ./35/17/5db9c4d2-00000006-d233904c-4833904c-12300000-c0a88501
 172 ./35/17/notes/5db9c4d2-00000006-d233904c-4833904c-12300000-c0a88501
 40G ./48/81/48088c0f-00000006-8331540c-4831540c-0f7f0000-c0a88501
 172 ./48/81/notes/48088c0f-00000006-8331540c-4831540c-0f7f0000-c0a88501
 37K ./56/87/aaa640c0-00000006-793172b2-483172b2-0f890000-c0a88501
 276 ./56/87/notes/aaa640c0-00000006-793172b2-483172b2-0f890000-c0a88501
1.1K May
 216 ./65/31/notes/1624b596-00000006-8231571c-4831571c-0f800000-c0a88501
1.1K May
 216 ./73/08/notes/c972cfed-00000006-7a3172b2-483172b2-0f880000-c0a88501
 20G ./79/92/f48cf434-00000006-25323ec6-48323ec6-10dd0000-c0a88501
 172 ./79/92/notes/f48cf434-00000006-25323ec6-48323ec6-10dd0000-c0a88501
  12:17

What you see here are the files created. The 40 GB are the raw data, the others I don't know. It works like this:

- backup is started on the backup server to a VTL (Virtual Tape Libary)

- This VTL is an NFS mount to the Exagrid server

- the backup is written to disk

- after some time the system starts to try deduplicating the files (no idea how it's done)

- half an hour later one can see on the web frontend, that the deduplication finished.

We found this out by doing a fourth backup. The compression/deduplication was first increased to 19,4:1 then slowly, after each refresh it got back again to finally 6,1:1. So no matter how many backups we save on the system, the compression/deduplication is steady at 6,1:1. Deduplication doesn't take place, otherwise this number should increase much more. This 6.1 were there also after the first backup where there's no second to compare.

If you're interested I can send you some screenshots how it looks like.

Markus

Former Member
0 Kudos

Hi Markus,

So MaxDB didn't de-dup at all to your Exagrid storage device?

The only space saving you got was via compression?

And after backing up your 40GB database x times, the used physical (addressable) disk on your Exagrid backup device was what?

Cheers

Shaun

Edited by: Shaun on May 22, 2008 4:04 PM

markus_doehr2
Active Contributor
0 Kudos

I Shaun

yes. As outlined by Lars that was expected.

Markus

Former Member
0 Kudos

I'm still surprised that you got NO deduplication at all.

Dosen't MaxDB perform the backup by sending 8k blocks to the pipe? (or file ... whatever you have configured)

The dedup vendors we are looking at (Data Domain and Quantum) can dedup variable length blocks varying from 4k to 16k block sizes. (we are yet to test these devices)

Although MaxDB backups are performed based on the order of the pages in the converter, the whole idea of de-duplication is to go, hey I already have a block that looks like this one, so I'll just point to the block I have already. Thus regardless of where the page exists in the database, or in the backup file, if its already on the de-dup applicance/device/server its not actually saved again ... just fancy pointer stuff.

Thus I would expect a good de-dup rate on any database. The larger the database the better the de-dup rate, as the only stuff changing is the "recent" data/blocks/pages. Old stuff isn't touched, and should de-dup really well .... i think....

Did that make sense ... or am I just so wrong on this ???

Cheers

Shaun

markus_doehr2
Active Contributor
0 Kudos

I'm still surprised that you got NO deduplication at all.

I´m not surprised - I expected that - I just wanted to make sure

Dosen't MaxDB perform the backup by sending 8k blocks to the pipe? (or file ... whatever you have configured)

Yes.

The dedup vendors we are looking at (Data Domain and Quantum) can dedup variable length blocks varying from 4k to 16k block sizes. (we are yet to test these devices)

Although MaxDB backups are performed based on the order of the pages in the converter, the whole idea of de-duplication is to go, hey I already have a block that looks like this one, so I'll just point to the block I have already. Thus regardless of where the page exists in the database, or in the backup file, if its already on the de-dup applicance/device/server its not actually saved again ... just fancy pointer stuff.

Yes. But imagine the following:

The database is saved and the first five blocks saved look like this:


02 04 06 08 10

Now the database is saved a second time and now the blocks look like


02 03 04 06 08

(so basically one block/page was inserted)

How should the deduplication algorithm detect, that there is just one block inserted? In our case the exagrid system compares two files on an 8kb size. In this example as of the second block everything is different to the previous backup so deduplication can´t take place.

Thus I would expect a good de-dup rate on any database. The larger the database the better the de-dup rate, as the only stuff changing is the "recent" data/blocks/pages. Old stuff isn't touched, and should de-dup really well .... i think....

In theory - yes

What I could imagine where this would work is:

- you stop the database to offline mode

- you save the FILES physically

- you start the database and work normally

- you stop again to offline

- you save the FILES a second time

THAT should give you a deduplication ability.

Markus

Former Member
0 Kudos

The database is saved and the first five blocks saved look like this:


 02 04 06 08 10
 

Now the database is saved a second time and now the blocks look like


02 03 04 06 08

(so basically one block/page was inserted)

How should the deduplication algorithm detect, that there is just one block inserted? In our case the exagrid system compares two files on an 8kb size. In this example as of the second block everything is different to the previous backup so deduplication can´t take place.

Markus

I would have thought this scenario is exactly what de-duplication is all about.

Using your example, you backup the following blocks/pages:


02 04 06 08 10

A new block/page is created in the DB giving us:


02 03 04 06 08 10

When I perform my next backup the dedup device would do this:

02 = already have it - create a pointer for this backup set to the existing unique block

03 = new block - save to the device

04 = already have it - create a pointer for this backup set to the existing unique block

06 = already have it - create a pointer for this backup set to the existing unique block

08 = already have it - create a pointer for this backup set to the existing unique block

10 = already have it - create a pointer for this backup set to the existing unique block

Now the blocks on the device are as follows even though I have two complete backups:

02 03 04 06 08 10

Cheers

Shaun

Edited by: Shaun on May 22, 2008 4:58 PM

markus_doehr2
Active Contributor
0 Kudos

> 02 = already have it - create a pointer for this backup set to the existing unique block

> 03 = new block - save to the device

> 04 = already have it - create a pointer for this backup set to the existing unique block

> 06 = already have it - create a pointer for this backup set to the existing unique block

> 08 = already have it - create a pointer for this backup set to the existing unique block

> 10 = already have it - create a pointer for this backup set to the existing unique block

How should the algorithm distinguish between "new block" and "changed block"? I mean one could have


02 03 04 06 08 10

as "03" was inserted or as "as of 03 everything changed" (because 04 was 06, 06 was 08 etc.)

Markus

Former Member
0 Kudos

Hey Markus,

I understand where your coming from, but I thought dedup worked differently to this. Heres how I thought it worked ...

To distinguish the different blocks the dedup devices create a hash of each incoming block. Eg

8k Block 03 contains "ABCD"

8k Block 04 contains "EFGH"

The hash of block 03 is 555AAA

The hash of block 04 is 777BBB

Regardless of what block movements occurs, basically if the block being sent to the dedup device is hashed, and the hash is either 555AAA or 777BBB (in this scenario) then the dedup device knows it already has that unique block saved already.

My understanding the only time a "new" or "changed" block is saved to the dedup device as a new unique block would be if the contents of the block no longer matches that of an existing block on the dedup device. eg block 05 contains "BDAC" and hases to 999CCC, then this would be identified as a new unique block.

But if block 06 contains "EFGH" thus hashes to 777BBB, then this block is not saved as there is already a unique matching block - just a pointer is used.

Whats that chance that all your 8k blocks in the DB are changed between backup A and backup B. Wouldn't this be the only way that you would get no deduplication?

Atleast I thought this is how the dedup devices worked ...

Cheers

Shaun

markus_doehr2
Active Contributor
0 Kudos

Yes - you might be right. I don´t know how those algorithms work - admittedly.

What I just see is that even if I do an backup in cold/admin mode two times one after the other, the compresison factor doesn´t increase 6.1:1. I see that the space occupied is a result of


Database size / compression factor * number of backups

I tried that by deleting all the backups and created two cold/admin backups which are identical by definition. After some time the compression factor goes down from 150:1 to 6.1:1 and there it stays - no matter how many additional (cold/admin) backups are created.

Markus

0 Kudos

Shaun,

Although MaxDB backups are performed based on the order of the pages in the converter

the pages of the database are actually saved in physical order on the volumes. If you have a single data volume, the pages will always be saved in the same order and de-duplication should work unless pages are inserted, deleted or updated (and moved to a different location). With multiple volumes, the pages are copied in parallel from all attached data volume to the backup file. Each volume is copied in strict physical sequence, but the order of the volumes is determined by the scheduler after a block of pages is written.

first 8 blocks from volume 1

first 8 blocks from volume 2

next 8 blocks from volume 1

first 8 blocks from volume 3

....

So the resulting order of pages in your backup file is de facto random from the de-duplication point of view.

henrik

Former Member
0 Kudos

Thanks Henrick,

But that still dosen't explain why MaxDB is not de-duplicating well (or at all).

Cheers

Shaun

lbreddemann
Active Contributor
0 Kudos

Hi Shaun,

the procedure you describe how the deduplication should work really is a dictionary compression procedure.

For each yet unknown chunk (I'll use "blocks" for the data groups written by MaxDB and "chunks" for the data groups used by the de-duplication system) the de-deplucation system "sees" a unique dictionary entry is created.

To be effective for a MaxDB backup the chunks taken by the de-duplication system have to be the same 8K as the DB blocks are and have to be aligned to the same block boundaries.

Once this is not given and the block order in the backup changes (Henrick has described the circumstances for this) each backup will lead to new chunks as seen through the glasses of the de-duplication system.

For rather stable files (like Word-documents or Oracle datafiles), this size+alignement problem is not that big, since the blocks stay in their place most of the time. So even if a chunk of the de-duplication system overlaps two or more blocks the same overlapping will occur with the following backups and thus the chunks found are already in the dictionary.

So Henrick is right - the relative randomness of the chunks seen by the de-duplication system leads to an ever growing 'dictionary' so that nearly each data backup consists of unique dictionary entries.

One possible way to enable de-duplication for the MaxDB that pops into my mind would be to put all changed pages to the end of each backup (the kernel knows this pages - they are used to make incremental backups). If in addition to that the pages where put into the backup in a fixed order (e.g. dev #1, dev #2, dev #3, dev #1, dev #2, dev #3 ...) the de-duplication should have no problem anymore.

Anyhow, this would make many changes in the DB Kernel necessary and I assume that deeper investigation on how the de-duplication really works (we're just talking about, how we think it works here, aren't we?) is necessary.

Probably the best thing would be to have the vendors of the de-duplication system talk to the MaxDB development...

KR Lars

nelis
Active Contributor
0 Kudos

Hi,

For me our database is not that big(300GB) and we have a good system in place - Legato with EMC SAN and robotic tape library. To backup/restore 300GB only takes just over an hour even when running many backups simultaneously, we have dedicated fibre for backups across our network. I don't think incremental backups would make much of a difference except more administration as already mentioned.

What I did try the other day as I was curious to see results was to backup through two pipes using "parallel medium". It almost halved the time of the backup to about 40 minutes! ..except it creates two identities for the backup which I'm not sure is a safe method for restoring as you would first need to restore the first part then the second if that is even possible with a database ?

Regards,

Nelis

lbreddemann
Active Contributor
0 Kudos

Hi Nelis,

depending on how much your data changes, the incremental backup will do much difference in terms of backup size/time.

Anyhow, if you're happy with the backup speed and size - it's of course simpler to use the complete data backup.

For the parallel backup: you're right, for each parallel device you created one file/pipe/tape is written - of course as this is the type of parallelity you create with parallel mediums.

When you perform a restore from such a parallel medium you will of course need all parts of it again - but you're going to restore them in parallel as well and not sequentially (although this is possible).

Best regards,

Lars

markus_doehr2
Active Contributor
0 Kudos

What I did try the other day as I was curious to see results was to backup through two pipes using "parallel medium". It almost halved the time of the backup to about 40 minutes! ..except it creates two identities for the backup which I'm not sure is a safe method for restoring as you would first need to restore the first part then the second if that is even possible with a database ?

You must configure Legato to not put the two sessions on one tape. Apart from that, it´s safe to do that, we´re doing those kind of parallel backups for a long time already.

Markus

markus_doehr2
Active Contributor
0 Kudos

Hi Lars,

One question is also open for me: why not use incremental backups instead?

The answer is (for me): it will add additional overhead for the restore scenario. If you use deduplication from an external vendor, the backups will appear as full backups to the database and can be restored as such.

Given the fact of using that for system copies, it´s MUCH easier to just tell the database "restore backup XY with EBID Z" than "restore XY, then apply delta Z and then A and B", ESPECIALLY if external backup tools are used and the backup is stored then on a number of different tapes of different days that need to be mounted etc. All that stuff will be done by the third party deduplication mechanism (if integrated of course).

From an administrative point of view that "db_activate recover", "recover_replace" and "recover_ignore" thing may be obvious and easy for you but for me it is not, especially if one has different versions (7.3 vs. 7.5 vs. 7.6) and the syntax changes in each version. The DBMGUI is also not that helpfull in that area because one would need to use the DBMGUI for the 7.3 version but 7.5/7.6 one needs to use Database Studio.

Just my EUR 0.02

Markus

lbreddemann
Active Contributor
0 Kudos

The DBMGUI is also not that helpfull in that area because one would need to use the DBMGUI for the 7.3 version but 7.5/7.6 one needs to use Database Studio.

Hi Markus,

why do you think that? DBMGUI currently support all versions from 7.3 to 7.6 (and it does work with 7.7!).

With the DBMGui it's rather simple to add the additional step to the restore scenario.

The problem I see with de-duplication is the high risk in loosing backup copies - once one block is corrupted in such a backup all follow-up backups will have this corruption (until of course the block is different in the backup again).

Best regards,

Lars

markus_doehr2
Active Contributor
0 Kudos

> why do you think that? DBMGUI currently support all versions from 7.3 to 7.6 (and it does work with 7.7!).

> With the DBMGui it's rather simple to add the additional step to the restore scenario.

What I can tell you: It did never really work completely for me. I have done a LOT of system copies (and restores) like that and what I was doing finally is to start the recover process once, noted down the (EMC Legato) recovery call, created manually two pipes and called the recover process manually as user root (using SSID instead of EBID).

This is most likely all Legato specific and may be specific to our installation but I seriously was never able to use DBMGUI to do system copies. Restores on the same system yes.

Markus