Combines the random obtain admission to indexing belief from tarindexer after which mounts the TAR the exhaust of fusepy for easy read-finest obtain admission to accurate like archivemount.
In distinction to libarchive, on which archivemount is predicated completely, random obtain admission to and appropriate seeking out is supported.

Diverse capabilities:

• Highly Parallelized: The utilization of the -P option will set off parallel xz and bzip2 decoders. This will seemingly presumably yield wide speedups on most up-to-date processors.
• Recursive Mounting: Ratarmount will moreover mount TARs internal TARs internal TARs, … recursively into folders of the an identical name, which is functional for the 1.31TB ImageNet knowledge voice.
• Mount Compressed Recordsdata: You may presumably moreover mount files without a doubt one of many supported compression schemes. Even supposing these files make no longer own a TAR, you would leverage ratarmount’s appropriate seeking out capabilities when opening the mounted uncompressed look of this kind of file.
• Read-Only Bind Mounting: Folders may presumably very effectively be mounted read-finest to diversified folders for usecases like merging a backup TAR with more recent variations of these files residing in a usual folder.
• Union Mounting: A pair of TARs, compressed files, and bind mounted folders will also be mounted beneath the an identical mountpoint.

Compressions supported for random obtain admission to:

• BZip2 as provided by indexed_bzip2 as a backend, which is a refactored and prolonged version of bzcat from toybox. Note moreover the reverse engineered specification.
• Gzip as provided by indexed_gzip by Paul McCarthy. Note moreover RFC1952.
• Rar as provided by rarfile by Marko Kreen. Note moreover the RAR 5.0 archive format.
• Xz as provided by python-xz by Rogdham or lzmaffi by Tomer Chachamu. Note moreover The .xz File Format.
• Zip as provided by zipfile, which is dispensed with Python itself. Note moreover the ZIP File Format Specification.
• Zstd as provided by indexed_zstd by Marco Martinelli. Note moreover Zstandard Compression Format.

1. Installation
2. Benchmarks
3. The Effort
4. The Resolution
5. Utilization
   1. Metadata Index Cache
   2. Bind Mounting
   3. Union Mounting
   4. File variations
   5. Compressed non-TAR files
   6. Xz and Zst Recordsdata

Dependencies

Python 3.6+, preferably pip 19.0+, and FUSE are required.
These must silent be preinstalled on most systems.
On Debian-like systems like Ubuntu, you would install/update all dependencies the exhaust of:

sudo trusty install python3 python3-pip fuse

On macOS, you’ll need to install macFUSE with:

While you occur to are installing on a system for which there exists no manylinux wheel, then you definately’ll desire to install dependencies required to design from supply:

sudo trusty install python3 python3-pip fuse design-obligatory tool-properties-traditional zlib1g-dev libzstd-dev liblzma-dev cffi

PIP Kit Installation

Then, you would simply install ratarmount from PyPI:

Or, must you’ll need to take a look at primarily the most up-to-date version:

python3 -m pip install --particular person --force-reinstall git+https://github.com/mxmlnkn/ratarmount.git@cancel#egginfo=ratarmount

If there are troubles with the compression backend dependencies, you would strive the pip --no-deps argument.
Ratarmount will work with out the compression backends.
The exhausting requirements are fusepy and for Python variations older than 3.7.0 dataclasses.

For xz toughen, lzmaffi will seemingly be used if accessible.
Because lzmaffi does no longer provide wheels and the design from supply depends on cffi, which may presumably per chance be lacking, finest python-xz is a dependency of ratarmount.
If there are complications with xz files, please chronicle any encountered components.
However, as a snappy workaround, you would strive to easily switch out the xz decoder backend by installing lzmaffi manually and ratarmount will exhaust that in its place with better precedence:

sudo trusty install liblzma-dev
python3 -m pip install --particular person cffi  # Predominant attributable to lacking pyprojects.toml
python3 -m pip install --particular person lzmaffi

Benchmarks

• No longer shown within the benchmarks, nonetheless ratarmount can mount files with preexisting index sidecar files in beneath a 2d making it vastly more efficient when when put next with archivemount for every and every subsequent mount.
  Also, archivemount has no development indicator making it no longer attainable the particular person will wait hours for the mounting to make.
  Fuse-archive, an iteration on archivemount, has the --asyncprogress technique to present a development indicator the exhaust of the timestamp of a dummy file.
  Show that fuse-archive daemonizes with out prolong nonetheless the mount level may presumably no longer be usable for a truly very lengthy time and every part searching to make exhaust of it must dangle till then when no longer the exhaust of --asyncprogress!
• Getting file contents of a mounted archive is mostly vastly faster than archivemount and fuse-archive and does no longer lift with the archive measurement or file depend resulting within the supreme noticed speedups to be around 5 orders of magnitude!
• Reminiscence consumption of ratarmount is mostly less than archivemount and mostly does no longer develop with the archive measurement.
  No longer shown within the plots, nonetheless the memory utilization will seemingly be necessary smaller when no longer specifying -P 0, i.e., when no longer parallelizing.
  The gzip backend grows linearly with the archive measurement since the tips for seeking out is thousands of cases bigger than the easy two 64-bit offsets required for bzip2.
  The memory utilization of the zstd backend finest appears humongous due to it makes exhaust of mmap to open.
  The memory used by mmap is no longer even counted as used memory when showing the memory utilization with free or htop.
• For empty files, mounting with ratarmount and archivemount does no longer appear be bounded by decompression nor I/O bandwidths nonetheless in its place by the algorithm for rising the internal file index.
  This algorithm scales linearly for ratarmount and fuse-archive nonetheless appears to scale worse than even quadratically for archives containing bigger than 1M files when the exhaust of archivemount.
  Ratarmount 0.10.0 improves upon earlier variations by batching SQLite insertions.
• Mounting bzip2 and xz archives has in actuality change into faster than archivemount and fuse-archive with ratarmount -P 0 on most up-to-date processors due to it in actuality makes exhaust of bigger than one core for decoding these compressions. indexed_bzip2 helps block parallel decoding since version 1.2.0.
• Gzip compressed TAR files are two cases slower than archivemount all the method thru first time mounting.
  It just isn’t completely sure to me why that’s due to streaming the file contents after the archive being mounted is comparably fleet, explore the subsequent benchmarks beneath.
  In verbalize to trust neatly-behaved speeds for every and every of these, I am experimenting with a parallelized gzip decompressor just like the prototype pugz gives for non-binary files finest.
• For the diversified cases, mounting cases change into roughly the an identical when when put next with archivemount for archives with 2M files in an roughly 100GB archive.
• Getting hundreds of metadata for archive contents as demonstrated by calling gain on the mount level is an verbalize of magnitude slower when when put next with archivemount. Since the C-based completely fuse-archive is even slower than ratarmount, the variation is extremely seemingly that archivemount makes exhaust of the low-degree FUSE interface while ratarmount and fuse-archive exhaust the excessive-degree FUSE interface.

• Finding out files from the archive with archivemount are scaling quadratically rather than linearly.
  That is due to archivemount begins studying from the beginning of the archive for every and every requested I/O block.
  The block measurement depends on this technique or operating system and must silent be within the verbalize of 4 kiB.
  That implies, the scaling is O( (sizeOfFileToBeCopiedFromArchive / readChunkSize)^2 ).
  Each and every, ratarmount and fuse-archive take a ways from this habits.
  Which means quadratic scaling, the moderate bandwidth with archivemount appears to be as if it decreases with the file measurement.
• Finding out bz2 and xz are each and every an verbalize of magnitude faster, as tested on my 12/24-core Ryzen 3900X, thanks to parallelization.
• Reminiscence is bounded in these checks for all applications nonetheless ratarmount is rather a lot more lax with memory due to it makes exhaust of a Python stack and due to it wants to take care of caches for a constant amount of blocks for parallel decoding of bzip2 and xz files.
  The zstd backend in ratarmount appears to be unbounded due to it makes exhaust of mmap, whose memory utilization will automatically close and be freed if the memory limit has been reached.
• The height for the xz decoder studying speeds happens due to some blocks will seemingly be cached when loading the index, which is no longer included within the benchmark for technical reasons. The cost for the 1 GiB file measurement is more realistic.

Further benchmarks will also be considered right here.

You downloaded a colossal TAR file from the cyber web, for instance the 1.31TB colossal ImageNet, and you now desire to make exhaust of it nonetheless lack the residing, time, or a file system fleet enough to extract the complete 14.2 million image files.

Partial Alternate ideas

Archivemount

Archivemount appears to trust colossal efficiency components for too many files and colossal archive for every and every mounting and file obtain admission to in version 0.8.7. A more in-depth comparison benchmark will also be came upon right here.

• Mounting the 6.5GB ImageNet Massive-Scale Visual Recognition Anguish 2012 validation knowledge voice, after which attempting out the rate with: time cat mounted/ILSVRC2012_val_00049975.JPEG | wc -c takes 250ms for archivemount and 2ms for ratarmount.
• Attempting to mount the 150GB ILSVRC object localization knowledge voice containing 2 million photos became as soon as given up upon after 2 hours. Ratarmount takes ~15min to assemble a ~150MB index and
• Doesn’t toughen recursive mounting. Though, you may presumably well presumably write a script to stack archivemount on prime of archivemount for all contained TAR files.

Tarindexer

Tarindex is a repeat line to tool written in Python that can presumably well assemble index files after which exhaust the index file to extract single files from the tar fleet. Alternatively, it moreover has some caveats which ratarmount tries to resolve:

• It finest works with single files, that means it’d be obligatory to loop over the extract-call. However this will seemingly seemingly require loading the presumably rather colossal tar index file into memory at any time when. As an illustration for ImageNet, the resulting index file is a complete bunch of MB colossal. Also, extracting directories will seemingly be a effort.
• Or no longer it is sophisticated to integrate tarindexer into diversified production environments. Ratarmount in its place makes exhaust of FUSE to mount the TAR as a folder readable by any diversified applications requiring obtain admission to to the contained knowledge.
• Can no longer take care of TARs recursively. In verbalize to extract files internal a TAR which itself is internal a TAR, the packed TAR first wants to be extracted.

TAR Browser

I did not gain out about TAR Browser sooner than I done the ratarmount script. That’s moreover no doubt one of it is cons:

• No longer easy to search out. I make no longer seem just like the very best one who has effort finding it because it has one star on Github after 7 years when when put next with 45 stars for tarindexer after roughly the an identical length of time.
• Anguish to voice up. Desires compilation and I gave up after I became as soon as urged to voice up a MySQL database for it to make exhaust of. Confusingly, the setup instructions are no longer on its Github nonetheless right here.
• Doesn’t appear to toughen recursive TAR mounting. I did not take a look at it attributable to the MysQL dependency nonetheless the code does no longer appear to trust common sense for recursive mounting.
• Xz compression moreover is finest block or physique based completely, i.e., finest works faster with files created by pixz or pxz.

Execs:

• helps bz2- and xz-compressed TAR archives

The Resolution

Ratarmount creates an index file with file names, possession, permission flags, and offset knowledge.
This sidecar is saved at the TAR file’s quandary or in ~/.ratarmount/.
Ratarmount can load that index file in beneath a 2d if it exists after which gives FUSE mount integration for easy obtain admission to to the files contained within the archive.

The take a look at with the first version (50e8dbb), which used the eliminated quandary backend for serializing the metadata index, for the ImageNet knowledge voice is promising:

• TAR measurement: 1.31TB
• Contains TARs: sure
• Recordsdata in TAR: ~26 000
• Recordsdata in TAR (including recursively in contained TARs): 14.2 million
• Index advent (first mounting): 4 hours
• Index measurement: 1GB
• Index loading (subsequent mounting): 80s
• Finding out a 40kB file: 100ms (first time) and 4ms (subsequent cases)

The studying time for a minute file simply verifies the random obtain admission to by the exhaust of file secret agent to be working. The variation between the first read and subsequent reads is no longer attributable to ratarmount nonetheless attributable to operating system and file system caches.

Right here’s a more most up-to-date take a look at for version 0.2.0 with the brand new default SQLite backend:

• TAR measurement: 124GB
• Contains TARs: sure
• Recordsdata in TAR: 1000
• Recordsdata in TAR (including recursively in contained TARs): 1.26 million
• I