'ZLIB' COMPRESSED DATA FORMAT SPECIFICATION version 3.2 November 12th, 1995 Copyright (C) 1995 L. Peter Deutsch and Jean-loup Gailly Permission is granted to copy and distribute this document for any purpose and without charge, including translations into other languages and incorporation into compilations, provided that it is copied as a whole (including the copyright notice and this notice) and with no changes. Questions about this specification can be sent by email to Jean-loup Gailly and Mark Adler . 1. Introduction 1.1 Purpose The purpose of this specification is to define a lossless compressed data format that: (a) Is independent of CPU type, operating system, file system, and character set, and hence can be used for interchange; (b) Can be produced or consumed, even for an arbitrarily long sequentially presented input data stream, using only an a priori bounded amount of intermediate storage, and hence can be used in data communications or similar structures such as Unix filters; (c) Can use a number of different compression methods; (d) Can be implemented readily in a manner not covered by patents, and hence can be practiced freely. The data format defined by this specification does not attempt to allow random access to compressed data. 1.2 Intended audience This specification is intended for use by implementors of software to compress data into zlib format and/or decompress data from zlib format. The text of the specification assumes a basic background in programming at the level of bits and other primitive data representations. 1.3 Scope The specification specifies a compressed data format, to be used for in-memory compression of a sequence of arbitrary bytes. 1.4 Compliance Unless otherwise indicated below, a compliant decompressor must be able to accept and decompress any data set that conforms to all the specifications presented here; a compliant compressor must produce data sets that conform to all the specifications presented here. 1.5 Related standards None. 1.6 Other related publications Deutsch, L.P.,"'Gzip' Compressed Data Format Specification". available in ftp.uu.net:/pub/archiving/zip/doc/gzip-*.doc Deutsch, L.P.,"'Deflate' Compressed Data Format Specification". available in ftp.uu.net:/pub/archiving/zip/doc/deflate-*.doc Thomas Boutell, "PNG (Portable Network Graphics) specification". available in ftp://ftp.uu.net/graphics/png/png* Fletcher, J. G., "An Arithmetic Checksum for Serial Transmissions," IEEE Transactions on Communications, Vol. COM-30, No. 1, January 1982, pp. 247-252. ITU-T Recommendation X.224, Annex D, "Checksum Algorithms," November, 1993, pp. 144, 145. (Available from gopher://info.itu.ch). ITU-T X.244 is also the same as ISO 8073. 1.7 Definitions of terms and conventions used byte: 8 bits stored or transmitted as a unit (same as an octet). (For this specification, a byte is exactly 8 bits, even on machines which store a character on a number of bits different from 8.) See section 3.1 below for the numbering of bits within a byte. 2. Detailed specification. 2.1 Overall conventions. In the diagrams below, a box like this: +---+ | | <-- the vertical bars might be missing +---+ represents one byte; a box like this: +==============+ | | +==============+ represents a variable number of bytes. Bytes stored within a computer do not have a 'bit order', since they are always treated as a unit. However, a byte considered as an integer between 0 and 255 does have a most- and least-significant bit, and since we write numbers with the most-significant digit on the left, we also write bytes with the most-significant bit on the left. In the diagrams below, we number the bits of a byte so that bit 0 is the least-significant bit, i.e., the bits are numbered: +--------+ |76543210| +--------+ Within a computer, a number may occupy multiple bytes. All multi-byte numbers in the format described here are stored with the MOST-significant byte first (at the lower memory address). For example, the decimal number 520 is stored as: 0 1 +--------+--------+ |00000010|00001000| +--------+--------+ ^ ^ | | | + less significant byte = 8 + more significant byte = 2 x 256 2.2 Data format. A zlib stream has the following structure: 0 1 +---+---+=====================+---+---+---+---+ |CMF|FLG|...compressed data...| ADLER32 | +---+---+=====================+---+---+---+---+ Any data which may appear after ADLER32 are not part of the zlib stream. CMF (Compression Method and flags) This byte is divided into a 4-bit compression method and a 4-bit information field depending on the compression method. bits 0 to 3 CM Compression method bits 4 to 7 CINFO Compression info CM (Compression method) This identifies the compression method used in the file. CM = 8 denotes the 'deflate' compression method with a window size up to 32K. This is the method used by gzip and PNG (see section 1.6 for the reference documents). CINFO (Compression info) For CM = 8, CINFO is the base-2 logarithm of the LZ77 window size, minus eight (CINFO=7 indicates a 32K window size). Values of CINFO above 7 are not allowed in this version of the specification. CINFO is not defined in this specification for CM not equal to 8. FLG (FLaGs) This flag byte is divided as follows: bits 0 to 4 FCHECK (check bits for CMF and FLG) bit 5 reserved, must be zero bits 6 to 7 FLEVEL (compression level) The FCHECK value must be such that CMF and FLG, when viewed as a 16-bit unsigned integer stored in MSB order (CMF*256 + FLG), is a multiple of 31. FLEVEL (Compression level) These flags are available for use by specific compression methods. The 'deflate' method (CM = 8) sets these flags as follows: 0 - compressor used fastest algorithm 1 - compressor used fast algorithm 2 - compressor used default algorithm 3 - compressor used maximum compression, slowest algorithm The information in FLEVEL is not needed for decompression; it is there to indicate if recompression might be worthwhile. compressed data For compression method 8, the compressed data is stored in the deflate compressed data format as described in the document "'Deflate' Compressed Data Format Specification" by L. Peter Deutsch. (See section 1.6). Other compressed data formats are not specified in this version of the zlib specification. ADLER32 (Adler-32 cheksum) This contains a checksum value of the uncompressed data computed according to Adler-32 algorithm. This algorithm is a 32-bit extension and improvement of the Fletcher algorithm, used in the ITU-T X.224 / ISO 8073 standard. Adler-32 is composed of two sums accumulated per byte: s1 is the sum of all bytes, s2 is the sum of all s1 values. Both sums are done modulo 65521. s1 is initialized to 1, s2 to zero. The Adler-32 checksum is stored as s2*65536 + s1 in most-significant-byte first (network) order. 3. Appendix: Rationale The Adler-32 algorithm is much faster than the CRC32 algorithm yet still provides an extremely low probability of undetected errors. The modulo on unsigned long accumulators can be delayed for 5552 bytes, so the modulo operation time is negligible. If the bytes are a, b, c, the second sum is 3a + 2b + c + 3, and so is position and order sensitive, unlike the first sum, which is just a checksum. That 65521 is prime is important to avoid a possible large class of two-byte errors that leave the check unchanged. (The Fletcher checksum uses 255, which is not prime and which also makes the Fletcher check insensitive to single byte changes 0 <-> 255.) The sum s1 is initialized to 1 instead of zero to make the length of the sequence part of s2, so that the length does not have to be checked separately. (Any sequence of zeroes has a Fletcher checksum of zero.) The following C code computes the Adler-32 checksum of a data buffer: #define BASE 65521 /* largest prime smaller than 65536 */ #define NMAX 5552 /* NMAX is the largest n such that 255n(n+1)/2 + (n+1)(BASE-1) <= 2^32-1 */ unsigned long adler32(unsigned char *b, int n) { unsigned char *p = b; unsigned long s1 = 1; unsigned long s2 = 0; int k; while (n > 0) { k = n < NMAX ? n : NMAX; n -= k; do { /* could unroll loop some for speed */ s1 += *p++; s2 += s1; } while (--k); s1 %= BASE; s2 %= BASE; } return (s2 << 16) | s1; } .