HotXLS, the native Excel component library for Delphi and C++Builder, inflates several XLSX worksheets at the same time out of a single open ZIP package. The mechanism is TZipReadGate, a small class in lxZipArchive.pas that holds the package stream plus one critical section and exposes exactly one method. It serializes the seek-and-read pair. Everything above that pair runs concurrently
The problem that forced this design is one every Delphi developer who has opened a large workbook has met. A 80 MB xlsx is 80 MB of deflated XML, and the worksheet parts inside it expand roughly five to ten times. If your open path extracts each worksheet into a memory stream before parsing it, you pay for the inflated bytes on top of the workbook you are building, and the peak arrives before a single cell has been created. This article is about the package-level concurrency that removes that staging step. The allocator ceiling that sits above it is covered in the article on parallel XLSX parsing and the memory manager, and the read-once, never-materialize API is covered in the streaming direct reader walkthrough
Why the old open path staged every worksheet in RAM
The original parallel open in HotXLS was a three-phase pipeline, and the middle phase was the only one that ran on workers. Phase A walked the sheet list serially, created each worksheet, read its relationship part, and copied the whole inflated worksheet XML into a private TMemoryStream. Phase B fanned ParseWorksheetXml out over the pool. Phase C went back to the archive on the calling thread for the small satellite parts: comments, threaded comments, drawings, charts, tables. That shape was chosen for a stated reason. The header comment on lxParallelParse.pas used to say, in as many words, that the zip archive and its inflate state are not thread-safe, and the internal notes went further: do not bother locking the archive, because once the inflate state machine is serialized per entry the lock buys nothing. Phase A existed to keep every archive touch on one thread. The cost was that a workbook with eight busy sheets held eight fully inflated worksheet XML buffers in memory simultaneously, and those buffers are the largest transient objects in the entire open path
Can two threads inflate from one ZIP stream?
Yes, and the old judgement was wrong in a specific, locatable way: it collapsed two different pieces of state into one sentence. Inflate state is genuinely not shareable. A zlib z_stream carries the sliding window, the Huffman tables and the bit position for one compressed member, and two threads pushing bytes through the same one produces garbage. The underlying byte source is a different question entirely, and the answer there is that a file stream has exactly one piece of mutable shared state worth protecting, its position cursor
The ZIP container makes the separation legal. Each member in a ZIP archive is compressed independently: its own local file header, its own deflate bit stream at its own DataOffset, its own CRC32 and sizes in the central directory. There is no shared dictionary spanning members the way a solid 7z block has, so entry N can be inflated without touching entry M. Give each worker its own z_stream over its own byte range and the only thing they collide on is the seek. That collision is what TZipReadGate removes, and the whole class is short enough to read in one screen
type
TZipReadGate = class
private
FBaseStream: TStream;
FLock: TRTLCriticalSection;
public
constructor Create(ABaseStream: TStream);
destructor Destroy; override;
function ReadAt(AOffset: Int64; var Buffer; Count: Longint): Longint;
end;
function TZipReadGate.ReadAt(AOffset: Int64; var Buffer;
Count: Longint): Longint;
begin
if Count <= 0 then
begin
Result := 0;
Exit;
end;
EnterCriticalSection(FLock);
try
FBaseStream.Position := AOffset;
Result := FBaseStream.Read(Buffer, Count);
finally
LeaveCriticalSection(FLock);
end;
end;What TZipReadGate protects and what it deliberately does not
TZipReadGate.ReadAt guards one indivisible operation, positioning the shared stream and reading from it, and nothing else. TZipArchive.OpenArchive constructs the gate over FInputStream once the central directory has parsed successfully, and TZipArchive.Close frees it. Archives opened for writing never get one. Every read that a worker performs on the package therefore funnels through a single critical section held for the duration of one buffered read
Everything else stays outside the lock because it is already private or already immutable. TZipSubStream keeps its own FPosition, so each worker tracks its own place in its own entry. The TZLibStream that TZipEntry.GetStream builds over that sub-stream is per-entry, created with windowBits of -15 for raw deflate, and never shared. The central directory is fully parsed before any worker starts, including every local header, so GetEntryByName is a read-only hash lookup by the time concurrency begins. The routing itself is three lines in TZipSubStream.Read, and the gateless branch is what keeps every existing single-threaded caller on the old code path
function TZipSubStream.Read(var buffer; Count: longint): longint;
var
rest: Int64;
rc: longint;
begin
rest := FSize - FPosition;
if (Count > rest) then
Count := rest;
if FReadGate <> nil then
rc := FReadGate.ReadAt(FOffset + FPosition, buffer, Count)
else
begin
FBaseStream.Position := FOffset + FPosition;
rc := FBaseStream.Read(buffer, Count);
end;
FPosition := FPosition + rc;
Result := rc;
end;How much does the gate cost under contention?
Less than the phrase "global lock on the archive" suggests, because of the granularity that TZLibStream happens to use. Its input buffer is BufferSize, defined as $4000, so ReadInputBuffer pulls 16 KB of compressed bytes per refill and hands them to zng_inflate. One lock acquisition therefore covers 16 KB of deflate input, which for worksheet XML expands into something on the order of 100 KB of markup that the worker then decodes and parses without holding anything. The lock is held for a positioned read against the operating system cache; the work it gates is measured in milliseconds
The honest boundary is where that ratio inverts. Entries stored rather than deflated read through the gate one-to-one with no inflate work to hide the latency, so a package full of stored members would serialize much harder. A cold file on slow media widens the critical section, because the read inside it is now a real disk transfer rather than a cache hit. And past a handful of workers the gate is not what you hit first anyway: worksheet parsing is allocation-heavy, and the Delphi memory manager serializes allocations across threads well before the read gate becomes the constraint. That is why TXLSXWorkbook.ParallelParseThreads defaults to an automatic cap instead of one thread per core
The worker body, and the drain loop that is easy to forget
With the gate in place, HotXLS deleted Phase A staging outright. The worker now opens its own entry stream and feeds it straight to the parser. Two transient fields carry the inputs: FParZip holds the archive for the duration of the parallel phase, FParSheetPartNames holds the part names, and both are cleared in the finally block so no stale pointer survives a failed open. The stream that comes back from TZipArchive.OpenFile is a TZipVerifiedStream wrapping a TZLibStream wrapping a TZipSubStream, and freeing the outer one frees the chain
procedure TXLSXWorkbook.ParseSheetJob(AIndex: Integer);
var
Stream: TStream;
DrainBuffer: array [0..32767] of Byte;
PartName: WideString;
begin
PartName := WideString(FParSheetPartNames[AIndex]);
Stream := FParZip.OpenFile(PartName);
if Stream = nil then
Exit;
try
ParseWorksheetXml(Stream, FSheets.ByPos[AIndex], FParSst,
FParRels[AIndex], FParFontMap, FParFillMap, FParBorderMap,
FParNumFmtMap, FParAlignMap, FParProtMap, FParDateMap);
// Consume any trailing bytes so the ZIP entry size and CRC are verified.
while Stream.Read(DrainBuffer, SizeOf(DrainBuffer)) > 0 do
;
finally
Stream.Free;
end;
end;The drain loop is the detail that a straight port of the old code would drop, and dropping it silently disables integrity checking. TZipVerifiedStream accumulates a running CRC32 as bytes pass through and calls VerifyComplete only when its position reaches the uncompressed size recorded in the central directory; that is where the size mismatch and the CRC32 mismatch exceptions come from, plus a one-byte probe read that catches an entry longer than declared. An XML reader stops at the closing element and usually leaves a newline or a few bytes of trailing whitespace unread, so without the drain the position never reaches the declared size and the checks never fire. Reading the remainder into a scratch buffer costs nothing and restores them. When the staging streams existed, XlsxCopyStreamAll was doing this by accident
What still runs serially, and the flag that turns it all off
Phase A survives, minus the extraction. It still creates each worksheet and reads its relationships on the calling thread, which is what leaves every shared map immutable once workers start. Phase C still walks the sheets serially afterwards for comments, drawings, charts and tables, and its guard changed from a null check on the old staging array to zip.Exists against the part name. The shared read-only inputs the workers touch, the shared string table and the cellXf maps, are complete before Phase B begins and are never written during it
var
Wb: TXLSXWorkbook;
begin
Wb := TXLSXWorkbook.Create;
try
Wb.ParallelParse := True; // default; False forces one sheet at a time
Wb.ParallelParseThreads := 4; // 0 selects the automatic cap
Wb.Open('quarterly-consolidation.xlsx');
// ... workbook is identical either way ...
finally
Wb.Free;
end;
end;Setting ParallelParse to False before Open dispatches the same job procedure with a thread count of one, and RunParallelJobs degenerates to a plain loop on the calling thread. That is worth knowing for two reasons: it is the one-line answer if a threading concern ever surfaces in the field, and it means the serial and parallel paths share a single body of parsing code rather than diverging. Worker exceptions are captured, the lowest job index wins, and the error is re-raised on the calling thread after every worker joins, so a corrupt worksheet still surfaces as one exception in the expected place. General tuning of the surrounding open path is covered in the guide to large workbook performance in Delphi
The read gate, the parallel open phase and the streaming entry access described here ship as part of the standard HotXLS Excel component for Delphi and C++Builder, with full source; the product page carries the complete TXLSXWorkbook reference including the parallel open properties