If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

 
Go Back  dBforums > Data Access, Manipulation & Batch Languages > Unix Shell Scripts > Bzip2 size constraints?

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1 (permalink)  
Old 08-27-08, 11:25
brianr8011 brianr8011 is offline
Registered User
 
Join Date: Sep 2007
Posts: 6
Bzip2 size constraints?

Does anyone know if there are any size limitations with bzip2? I couldn't find any online?

Also, are there any size limits on the other compression methods like gzip or compress?

Thanks in advance...
Reply With Quote
  #2 (permalink)  
Old 08-28-08, 07:36
pdreyer pdreyer is offline
Registered User
 
Join Date: May 2005
Location: South Africa
Posts: 1,268
Depends on the version
A few years back most programs sufferd with the 2GB limit
Here are 2 links complaining about bzip2
http://rspsoftware.com.br/rspbzip2.htm
http://bytes.com/forum/thread431897.html

Consider using 7-zip
It beats bzip2 in time and size
Here is a test I've done on a 1GB log file (plain text)
Code:
					 size	  zip%	mm:ss.n	zipped size
AP_send_20080717.log	           1105986047	 0.00%	#N/A	#N/A	
gzip -1				     62949561	94.31%	00:26.6	||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||	Deflate method
bzip2 -1			     31637027	97.14%	05:31.8	|||||||||||||||||||||||||||||||		Burrows-Wheeler block sort-ing algorithm
gzip -9				     29785817	97.31%	01:52.1	|||||||||||||||||||||||||||||		Deflate method
7z -mx=1			     22776486	97.94%	01:07.0	||||||||||||||||||||||		Lempel-Ziv based algorithm
7z -mx=1 -m0=lzma:d=4m:lc=4:fb=128   12638145	98.86%	01:32.6	||||||||||||			Lempel-Ziv based algorithm
bzip2 -9			     11632622	98.95%	08:56.5	|||||||||||			Burrows-Wheeler block sort-ing algorithm
7z -m0=ppmd:o=32:mem=29		      9942841	99.10%	02:50.3	|||||||||			PPM-based algorithm (very good for plain text files)
For database dumps I find this best for me (speed & compression)
7z -mx=1 -m0=lzma:d=4m:lc=4:fb=128

Another site doing a far greater job to compare than me is:
http://www.maximumcompression.com/data/log.php

PS. The vesions used in my tests
gzip 1.3.5
bzip2 1.0.2
7-Zip 4.57

Last edited by pdreyer; 08-28-08 at 07:55.
Reply With Quote
  #3 (permalink)  
Old 08-28-08, 08:42
brianr8011 brianr8011 is offline
Registered User
 
Join Date: Sep 2007
Posts: 6
Thanks for the info and links.

We have bzip2 version 0.9.0c at work and don't have 7z..

And unfortunately we are unlikely to upgrade as we are about to be bought by another company

I'll just have to make due I guess.

Thanks again!
Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On